|Home | About | Journals | Submit | Contact Us | Français|
We sought to determine whether high rates of compliance with perioperative process of care measures used for public reporting and pay-for-performance are associated with lower rates of risk-adjusted mortality and complications with high-risk surgery.
Retrospective analysis of Medicare inpatient claims data from beneficiaries undergoing one of six high-risk surgeries in 2,000 hospitals during 2005 and 2006. Hierarchal logistic regression models assessed the relationship between adverse outcomes and hospital compliance with the surgical processes of care reported in Hospital Compare.
30-day post-operative mortality, venous thromboembolism, surgical site infection.
Process compliance ranged from 53.7 percent in low compliance hospitals to 91.4 percent in the highest. Risk-adjusted outcomes did not vary at high compliance hospitals relative to average compliance for mortality (OR = 0.98, 95% CI 0.92, 1.05), surgical site infection (OR = 1.01, 95% CI 0.90, 1.13), or venous thromboembolism (OR = 1.04, 95% CI 0.89, 1.2) or in lowest compliance. Outcomes also did not vary at low-compliance hospitals. Stratified analyses by operation type confirm these trends for the six procedures individually.
Currently available information on Hospital Compare will not help patients identify hospitals with better outcomes for high-risk surgery. CMS needs to identify higher leverage process measures and devote greater attention to profiling hospitals based on outcomes to improve public reporting and pay-for-performance efforts.
As variations in surgical quality are increasingly appreciated, payers are escalating efforts to reduce them.i,ii,iii The Centers for Medicare and Medicaid Services (CMS), the largest public payer, now mandates public reporting of two sets of the Surgical Care Improvement Project (SCIP) measures covering infection and venous thromboembolism. Hospitals are required to submit data quarterly, which is posted on the Hospital Compare website, in order to receive annual Medicare payment updates.iv This reporting is believed to aid consumers and payers in choosing high quality hospitals and to stimulate quality improvement amongst reporting hospitals.v,vi,vii
It is unclear whether these efforts will translate into better outcomes for surgical patients. Although the SCIP measures were selected because of strong evidence linking them to certain outcomes, there is reason to be skeptical that improved compliance will result in significant improvements in the most important outcome, risk-adjusted mortality. Namely, SCIP processes are associated with outcomes that are rare (e.g., deep venous thrombosis and pulmonary embolism) or considered secondary (e.g., superficial surgical site infections).viii,ix It is unknown whether measured processes of care are important determinants of surgical outcomes. If there is a weak link between process compliance and surgical outcomes, CMS public reporting and pay for performance efforts will be unlikely to stimulate important improvements or help patients find the safest hospitals.x,xi
In this context, we sought to determine whether hospital compliance rates for targeted surgical processes of care reported in Hospital Compare are related to risk-adjusted mortality, venous thromboembolism, and surgical site infection. We use national Medicare claims data to focus on patient outcomes following six high-risk surgical procedures.
Hospital Compare reports of medical care process compliance have been described previously.xii Hospitals begin reporting two SCIP measures in 2005: the rate of prophylactic antibiotic receipt within two hours of surgery and the rate of prophylactic antibiotic discontinuation within 24 hours of surgery. Three additional measures are added in 2006 reports: rate of correct antibiotic administration to prevent infection; recommended venous thrombosis prophylaxis ordered; and recommended venous thrombosis prophylaxis ordered within 24 hours of surgery.xiii Hospitals report the number of patients eligible for each process and the percentage receiving each process. SCIP measures are collected across a broad range of procedures including cardiac, orthopedic, vascular, general, and gynecological surgery.xiv
We obtained archived Hospital Compare data covering the period January 2005 – December 2006. Data are posted with a nine-month lag.xv To assess hospital compliance, we calculate an opportunity score based on the number of times a hospital complies with recommended measures for each eligible patient on up to 5 SCIP measures for each year of Hospital Compare data (two infection measures introduced in 2005 and three infection and two venous thromboembolism measures collected in 2006). Hospitals are classified into quintiles based on average level of compliance with recommended measures. In sensitivity analyses, we examined infection and venous thromboembolism scores separately.
Forty percent of hospitals performing 43 percent of operations reported SCIP compliance in 2005. Non-reporting hospitals had significantly lower annual procedure volume (78 versus 90) and were more likely to be government-owned or non-profit. There was no difference in non-surgical composite compliance rates across reporters and non-reporters (97 percent of SCIP non-reporters reported other measures). By 2006, virtually all hospitals performing study procedures report SCIP compliance. Only 15 hospitals performing 142 procedures do not report.
We identified all Medicare Fee-for-Service hospitalizations for six high-risk surgical procedures in the 100% MedPAR dataset from January 2005 – December 2006. Eligible admissions include abdominal aortic aneurism repair, aortic valve repair, coronary artery bypass graft, esophageal resection, mitral valve repair, and pancreatic resection. These hospitalizations are well-suited to our study because they are sufficiently common and high-risk to reveal variation in surgical mortality and complication rates across hospitals.
During the study period, 325,052 elderly Fee-for-Service Medicare beneficiaries underwent one of the included procedures at 2,189 hospitals nationwide. Contemporaneous SCIP compliance data are available for 229,665 admissions at 2,038 hospitals. We identify three focal surgical outcomes in the Medicare data; 30-day mortality, postoperative deep vein thrombosis or pulmonary embolism, and postoperative surgical site infection.
We estimate risk-adjusted surgical outcomes overall and for each procedure using hierarchical linear models including hospital-level indicators of quality known to relate to patient outcomes; procedure volume and indicators for highest SCIP compliance quintile and lowest SCIP compliance quintile, and patient characteristics; age, race, gender, severity of comorbid conditions classified using the Charlson index, patient zipcode median income from the 2000 census, whether the admission was elective or emergent, and year of admission.xvi,xvii Hospital random effects are included to account for clustering of patients in hospitals.
The base analysis examines the relationship between contemporaneous surgical process compliance and mortality. We also examine lagged SCIP compliance and mortality, essentially testing whether the historical information posted on Hospital Compare effectively provides consumers with information about their risk of an adverse surgical event.
We conduct additional analyses using only the 2006 data, when a more comprehensive set of SCIP measures are available. In these data, we examine whether the collected measures provide information about patient risks of experiencing targeted outcomes associated with the SCIP measures, venous thrombosis and postoperative surgical site infection.
Mean surgical compliance rates varied considerably, ranging from 53.7 percent in hospitals in the lowest compliance quintile to 91.4 percent in the highest quintile (Figure 1). Hospitals in the lowest quintile of process compliance were less likely to be accredited or to have an emergency room (Table 1). These hospitals also have lower rates of non-surgical process compliance and lower surgical volume.
Unadjusted thirty-day mortality rates vary from 3.9 percent for abdominal aortic aneurysm repair to 11.3 percent for mitral valve replacement. There is little variation in patient characteristics across levels of SCIP compliance, though hospitals in the lowest quality of compliance consistently serve Medicare beneficiaries from the lower-income zipcodes (Table 2).
We find little evidence of a consistent relationship between hospital compliance with processes of care and operative mortality (Figure 2). In univariate analysis, mortality rates in low compliance hospitals are statistically indistinguishable from those in the highest quintile of compliance for all procedures studied except for aortic valve replacement, where high compliance hospitals had lower mortality rates. Hospitals that do not report SCIP compliance have similar rates of risk-adjusted mortality as those in the highest quintile of SCIP compliance.
In multivariate analysis, relative to average compliance, risk-adjusted mortality did not vary at low SCIP compliance hospitals (OR = 1.06, 95% CI 0.97–1.16) or high compliance hospitals (OR = 0.98, 95% CI 0.92–1.05) (Table 3). Hospital compliance with the SCIP measures accounts for only 3.3% of the hospital variance in mortality. Stratified analyses by operation type also fail to show a significant association between hospital process compliance and mortality. Prior year SCIP compliance quintiles provide similar inference with wider confidence intervals, reflecting greater statistical noise from a lagged measure and the smaller number of hospitals reporting 2005 data (Table 3).
Unadjusted complication rates are lower amongst hospitals with the lowest quintile of compliance with SCIP measures than those in the highest quintile of compliance for DVTPE (low compliance = 0.43 percent, high compliance = 0.59 percent) and INF (low compliance = 1.1 percent, high compliance = 1.9 percent).
These relationships persist in multivariate analysis (Table 4), where we find no significant relationship between quintile of compliance and risk of venous thromboembolism (high compliance OR = 1.04, 95% CI 0.89, 1.20; low compliance OR = 0.93, 95% CI 0.73, 1.20) or infection (high compliance OR = 1.01, 95% CI 0.90, 1.13; low compliance OR = 0.96, 95% CI 0.80, 1.16). While statistically insignificant, estimates suggest that patients face lower risk of adverse surgical outcomes at hospitals with poor compliance with SCIP measures.
We conduct several additional analyses to test the robustness of these findings. Results are unchanged when we replace our SCIP compliance composites with outcome-specific measures; risk of infection does not vary with hospital compliance with SCIP infection compliance nor does risk of venous thromboembolism significantly vary with use of venous thromboembolism prophylaxis. We also eliminate hospitals in the middle 3 quintiles of compliance and directly compare highest compliance hospitals to lowest compliance hospitals. Risk-adjusted mortality is insignificantly lower at high-compliance hospitals (OR = 0.88, 95% CI 0.78, 1.01), while risks of thromboembolism (OR = 1.12, 95% CI 0.85, 1.48) and surgical site infection (OR = 1.04, 95% CI 0.85, 1.28) remain insignificantly greater at high-compliance hospitals.
We also consider extended length of stay, which could result from a number of postoperative complications. Extended hospital stays, in the highest quartile of procedure-specific inpatient days, are less likely at more compliant hospitals. Patients at high-compliance hospitals are 12 percent less likely to experience an extended stay relative to middle compliance hospitals (OR = 0.88, 95% CI 0.81, 0.94), though there is no difference between low and average compliance (OR = 1.05, 95% CI 0.95, 1.17).
The risk of patient death and the higher costs to Medicare associated with adverse surgical events emphasize the importance of providing beneficiaries with information that facilitates choice of a high-performing hospital. There is a clear business case for increased use of high-quality hospitals for surgical patients. Although compliance with surgical process measures varies widely, we find little evidence that SCIP measures reliably correlate with risk-adjusted surgical outcomes. As a result, patients who choose their hospital based on high rates of process compliance will not improve their chance of survival or complications. Patients choosing high compliance hospitals do, however reduce their risk of experiencing an extended stay.
In contrast to our findings for surgery, previous research in the medical literature has found mixed evidence of a relationship between process compliance and mortality. Werner and colleagues have shown that medical process compliance rates reported on Hospital Compare, particularly those for acute myocardial infarction care, are both related to inpatient mortality and also reflect other dimensions of quality.xviii,xix Similar to our own results, however, medical process compliance explains little of the variation in mortality.
Our study has several important limitations primarily related to our use of administrative Medicare data. Hospitals are required to report process measures for all eligible admissions, not just Medicare admissions. Although we have generally found high correlations between surgical mortality in Medicare and all-payer data and use the universe of Medicare hospitalizations during our study years, it is possible that we fail to find a relationship between process compliance and surgical outcomes because of insufficient sample size. Relatively few of the high-risk procedures we examined are performed in the lowest compliance hospitals. Findings are robust to alternative categorizations of compliance and across multiple measures of adverse outcomes.
Our study also faces the well-known limitation of risk-adjustment in administrative data. However, this will only bias our results if unobserved patient acuity is systematically related to hospital process compliance, which seems unlikely, especially as surgical process compliance rates were not generally known prior to their reporting on Hospital Compare during our study period. Furthermore, for rates of process compliance to be meaningful as indicators of hospital quality, their relationship to patient outcomes should not be heavily dependent on complex risk-adjustment techniques which will be inaccessible to most Hospital Compare users.
There are several reasons why public reporting based on SCIP measures may be inadequate to differentiate quality of inpatient surgical care. SCIP measures are low leverage because they relate to secondary and relatively less important outcomes. Even when processes are tied to an important outcome like pulmonary embolism, these events are very rare and offer insufficient variation to differentiate between high and low quality hospitals.
Unlike other surgical quality measures such as procedure volume and inpatient mortality which can be calculated from existing administrative data, reporting process measures imposes additional compliance costs on hospitals. It is important to document whether tracking these measures is important given the cost of collection. If not, CMS and other payers and policymakers targeting surgical compliance measures may want to consider other reporting options. Direct reporting of surgical outcomes would be an alternative. The Leapfrog Group, a large coalition of health care purchasers, encourages use of high-quality hospitals based on a composite measure of mortality and volume for several high-risk surgical procedures.
Despite CMS intentions to provide consumers with information that will facilitate patient choice of high quality hospitals, currently available information on Hospital Compare will not help patients identify hospitals with better outcomes for high-risk surgery. CMS needs to identify higher leverage process measures and devote greater attention to profiling hospitals based on outcomes for improved public reporting and pay-for-performance programs. Future research should ascertain whether process measures become more useful as indicators of surgical quality as public reporting programs mature.
This study was supported by a training grant to Dr. Nicholas from the National Institute on Aging (AG000221-17), a training grant to Dr. Osborne from the Robert Wood Johnson Foundation, a grant to Dr Birkmeyer from the National Cancer Institute (R01 CA098481), and a career development award to Dr. Dimick from the Agency for Healthcare Research and Quality (K08 HS017765). Dr. Nicholas had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Institute for Social Research (ISR), University of Michigan (LN); Michigan Surgical Collaborative for Outcomes Research and Evaluation (M-SCORE), Department of Surgery, University of Michigan (NO, JB, JD).
Lauren Hersch Nicholas, Institute for Social Research, University of Michigan, 426 Thompson Street, Ann Arbor, MI 48106, (734) 764-2562 (p) (734) 998-7473 (f)
Nicholas H. Osborne, Department of Surgery, University of Michigan Medical School, 211 N. Fourth Avenue, Suite 2A, Ann Arbor, MI 48104, (734) 936 5732 (p) (734) 998-7473 (f)
John D. Birkmeyer, Department of Surgery, University of Michigan Medical School, 211 N. Fourth Avenue, Suite 2A, Ann Arbor, MI 48104, (734) 998-7470 (p) (734) 998-7473 (f)
Justin B. Dimick, Department of Surgery, University of Michigan Medical School, 211 N. Fourth Avenue, Suite 2A, Ann Arbor, MI 48104, (734) 997 -7470 (p) (734) 998-7473 (f)