|Home | About | Journals | Submit | Contact Us | Français|
The Centers for Medicare and Medicaid Services require hospitals performing carotid artery stenting (CAS) to recertify the quality of their programs every two years, but currently this involves no explicit comparisons of post-procedure mortality across hospitals. Hence, the current recertification process may fail to identify hospitals that are performing poorly in relation to peer institutions. Our objective was to compare risk-standardized procedural outcomes across U.S. hospitals that performed CAS, and to identify hospitals with statistically high post-procedure mortality rates.
We conducted a retrospective cohort study of Medicare beneficiaries who underwent CAS from July, 2009 to June, 2011 at 927 U.S. hospitals. Thirty-day risk-standardized mortality rates were calculated using the “Hospital Compare” statistical method—a well-validated hierarchical generalized linear model that included both patient-level and hospital-level predictors. Claims were examined from 22,708 patients undergoing CAS, with a crude 30-day mortality rate of 2.0%. Risk-standardized 30-day mortality rates after CAS varied from 1.1% to 5.1% (p<0.001 for the difference). Thirteen hospitals had risk-standardized mortality rates that were statistically (p<0.05) higher than the national mean. Conversely, five hospitals had risk-standardized mortality rates that were statistically (p<0.05) lower than the national mean.
We used administrative claims to identify several CAS hospitals with excessively high 30-day mortality after carotid stenting. When combined with information currently used by Medicare for CAS recertification such as clinical registry data and program reports, clinical outcomes comparisons could enhance Medicare’s ability to identify hospitals that are questionable candidates for recertification.
After the Food and Drug Administration’s (FDA’s) 2005 approval of carotid artery stenting (CAS) systems with cerebral protection1—an alternative, less-invasive treatment for carotid stenosis2–4—the Centers for Medicare and Medicaid Services (CMS) issued an unprecedented national coverage decision that required hospitals to certify their ability to provide high-quality carotid artery stenting before being permitted to receive Medicare reimbursement for the procedure.5 In the absence of outcomes data, the initial hospital certification was accomplished via institutional self-assessments of CAS program quality. Each hospital’s chief executive attested to the presence of various clinical programs, equipment, organizational characteristics, and procedure volume that were assumed to be associated with high-quality CAS.5 Subsequently, Medicare’s requirement for bi-annual recertification of CAS programs compels hospitals to re-affirm the presence of these structural characteristics, and also to submit CAS registry data describing the characteristics of CAS patients treated and the frequency of short-term procedural complications.6
The goal of Medicare’s hospital certification program is ostensibly to ensure that the clinical outcomes of CAS, a complex procedure with considerable rates of morbidity and mortality,2–4, 7, 8 are sufficiently good in all hospitals certified to receive Medicare payments for the procedure. However, other than self-reports of short-term complications, no objective longer-term measures of CAS clinical outcomes have been incorporated into Medicare’s hospital recertification process, raising the possibility that hospitals with systematically poor CAS clinical outcomes will nevertheless be recertified.6 Therefore, our research goal was to apply Medicare’s “Hospital Compare” statistical methodology,9, 10 which has been used previously to compare nationwide hospital outcomes using Medicare administrative claims, to compare risk-standardized CAS outcomes across U.S. hospitals, and determine whether such a comparison would be a useful addition to Medicare’s hospital recertification program.
We used the statistical methods developed for Medicare’s “Hospital Compare” program, which is used for public reporting of hospital outcomes for acute myocardial infarction,10 congestive heart failure,10 and pneumonia,11 to compare risk-standardized mortality rates across hospitals.12 The Hospital Compare model uses hierarchical Bayesian methods to statistically “stabilize” the mortality estimate from each individual hospital so that hospitals with smaller case volumes are neither penalized nor rewarded for the random variation in the adverse event rate that is inherently greater at hospitals with smaller numbers of treated patients.13, 14 In implementing the Hospital Compare model for the analysis of CAS outcomes, we incorporated the recent methodological recommendations of the 2012 Committee of Presidents of Statistical Societies–CMS White Paper Committee on the use of the CMS Hospital Compare Model,14 including the use of both patient-level and hospital-level characteristics as critical components of the risk adjustment model for outcome rates.15
Patients were selected from among all Medicare claims for CAS submitted by all U.S. hospitals certified by Medicare to receive reimbursement for CAS from July, 2009 through June, 2011—the list of certified hospitals and their original certification date is reported by Medicare.16 We used this two-year data window for CAS because it represented the most recent data available, and it is the expressed intention of the Medicare program to require hospital recertification every two years,6 thus two years of data would be the typical extent of information on clinical outcomes available to Medicare when considering a hospital’s candidacy for recertification. While hospitals that have been certified for more than two years hypothetically could be required to submit more years of outcomes data at the time of recertification, under the terms of the current recertification program it is essential to verify that a clinical outcomes comparison model using data from the minimum time frame (i.e., 2 years) would be sufficient to identify low-quality performers.
Inpatient claims indicating the provision of a CAS were identified by the appearance of International Classification of Diseases, 9th Revision—Clinical Modification (ICD-9) procedure codes 00.61 and 00.63 on the inpatient claim. We validated the accuracy of this coding by verifying that claims with such codes were also classified into the appropriate diagnosis-related group (DRG) for carotid stenting (i.e., DRG 34–36). We found that greater than 99% of claims with the procedure codes 00.61 and 00.63 were classified into the carotid stenting DRGs, and that DRG 34–36 had procedure codes 00.61 and 00.63 in 98% of claims, suggesting these procedure codes were highly sensitive and specific indicators of CAS.
For each CAS recipient, all prior inpatient and outpatient Medicare claims were obtained for the 12-month period immediately preceding CAS receipt. Diagnosis codes were extracted from these claims and were used to identify patient comorbidities. Following the practices established by Hospital Compare, the Hierarchical Condition Categories coding scheme17 and the Deyo-Charlson comorbidity score18, 19 were used to classify and quantify each patient’s pre-existing comorbid conditions. The Medicare enrollment database was used to ascertain each patient’s age and sex, and the vital status of each patient as reported 30 days after receipt of CAS was used to establish 30-day mortality. Death dates in this database originate from the Social Security Death Master File and thus are considered highly accurate indicators of Medicare beneficiaries’ deaths.
Structural characteristics of the hospitals were obtained via Medicare’s Hospital Cost Report Information System from data submitted in 2011,20 and/or from linkage to the 2010 American Hospital Association survey. Among these characteristics were the hospital’s geographic location, ownership status (e.g., for-profit, not-for-profit, government-operated), and whether the hospital was a major academic center, defined by membership in the American Association of Medical Colleges’ Council of Teaching Hospitals. Additional information about each hospital’s inpatient population was calculated from the full set of inpatient Medicare claims for each hospital from July, 2009 to June, 2011. From these data we calculated each hospital’s mean annual number of Medicare hospitalizations and the proportion of each hospital’s Medicare patients who were non-white.
Details of the Hospital Compare risk adjustment methods have been described in detail elsewhere.14 Following the methods originally established by Krumholz et al.,10, 21 we identified 30 candidate patient-level predictors of 30-day mortality that were measureable from Medicare claims using the methods described above, as well as six hospital characteristics (U.S. Census region, urban location, academic status, ownership, minority patient population fraction, and CAS procedure volume over the 2-year study period). These 36 predictors were entered into a single-level (i.e., non-hierarchical) multivariable logistic regression model, and stepwise backwards elimination (excluding predictors with p-values exceeding 0.15) was used to identify a parsimonious prediction model (Table 1). Subsequently we estimated a hierarchical generalized linear model (HGLM) with a logit link,22 including as independent variables all of the patient-level and hospital-level predictors retained in the final single-level logistic regression model. The hospital-level “random effects” estimated by the hierarchical model were used to calculate a risk-standardized mortality ratio for each hospital, using the Hospital Compare formula (i.e., the hospital’s “predicted mortality” as calculated from the observed outcome rate with shrinkage applied to account for instability of rate estimates in small samples, divided by the “expected mortality” as derived from the observed characteristics of each hospital and its CAS patients).10 Confidence intervals for each hospital’s risk-standardized mortality estimate were derived using 500 bootstrap replications of the patient-level dataset with random selection at the hospital level, following the methods for confidence interval derivation described by Ash et al.14
Following the methods of Krumholz et al., we then identified hospitals that had 95% confidence intervals for 30-day risk-standardized mortality that excluded the mean national 30-day mortality rate. While it is uncertain that a hospital’s achieving the national mean risk-standardized CAS mortality rate would unequivocally indicate high-quality care, it is likely that the national mean mortality rate is higher than it would be if all hospitals performed CAS with optimal quality, and thus a hospital with a significantly higher mortality rate could be conservatively assumed to have outcomes that were not commensurate with high-quality care.
Because the inclusion of hospital-level predictors in models of this type is controversial, we repeated the analysis using a model that only included patient-level predictors and a random, hospital-level intercept. We compared the list of outlier hospitals generated by this alternative specification to our base-case model to assess the importance of including hospital-level predictors.
All statistical analyses were performed using SAS 9.3 (Cary, NC) or Stata 13.0 (College Station, TX). The hierarchical model was implemented using PROC GLIMMIX in SAS, with hospital characteristics entering the model as fixed effects in the “model” statement, which also included random hospital-level intercepts. The research protocol was approved by the University of Pennsylvania’s Institutional Review Board.
The 927 hospitals in the study ranged in CAS procedure volume from 1 to 246 cases during the two-year study period (Table 2). A total of 22,709 CAS procedures were included. High-volume CAS hospitals were more likely to have larger numbers of Medicare patients in general, and high-volume hospitals were also more likely to be academic medical centers. Approximately 40% of CAS hospitals were in the southern United States, and 52% were located in urban counties. Over half of CAS hospitals had greater than 12% non-white Medicare patients, with 12% approximating the national percentage of all Medicare hospitalizations involving non-white beneficiaries.
The 30-day risk-standardized mortality rate (RSMR) for CAS hospitals ranged from 1.1% to 5.1% (p<0.001 for the difference), with a national mean mortality rate of 2.0% (Figure 1). Among the 639 (69%) hospitals with a RSMR higher than the national mean, 13 hospitals had 95% confidence intervals that excluded the national mean (Figure 2), signifying a statistically significant difference in risk-standardized mortality from the national mean. These 13 hospitals performed a total of 324 CAS procedures among Medicare beneficiaries from July, 2009 – June, 2011, with a mean RSMR of 3.27% (95% confidence interval 3.20% to 3.34%). Grouped by CAS procedure volume, these hospitals included two high-volume (>25 procedures), two medium-volume (8–25 procedures), and nine low-volume hospitals (<8 procedures). Eleven of these hospitals were academic hospitals, and seven of the thirteen hospitals had black inpatient populations that exceeded 12%.
Among all 927 hospitals performing CAS, 288 hospitals (31%) had a RSMR that was lower than the national mean, although this difference was statistically significant for only five hospitals. These five hospitals performed a total of 333 CAS procedures (range: 27 to 115) during the two-year observation period, with a mean RSMR of 1.17% (95% confidence interval 1.15% to 1.18%). All five hospitals were non-academic centers located in the southern United States Census region; four of the five had black patient populations exceeding 12%.
Unless every hospital performing CAS in 2009–2011 had zero avoidable deaths at 30 days, the national crude mean 30-day mortality rate of 2.0% in 2009–2011 was unlikely to represent an optimal outcome rate, and thus it could be argued that the appropriate national benchmark mortality rate for acceptable quality should be lower. Assuming that the true “acceptable quality” mortality rate is lower than 2.0% would classify more hospitals with a RSMR that statistically exceeded the acceptability threshold. For example, if the optimal 30-day RSMR benchmark were lowered to 1.75%, then 51 hospitals (performing 683 CAS procedures over two years) would have significantly exceeded the mortality rate threshold. Further reducing the optimal 30-day RSMR benchmark to 1.5% would have classified 111 hospitals (performing 1,849 CAS procedures over two years) as having excessively high RSMR. In both of these scenarios, low-volume hospitals were disproportionally represented relative to their frequency in the overall population of CAS-performing hospitals (i.e., 33%). Of the 51 hospitals as outliers using a mortality threshold of 1.75%, 36 (71%) were low volume. Of the 111 hospitals classified as outliers using a 1.5% threshold, 61 (55%) were low volume.
Our findings were unchanged when modest modifications to the specification of the multivariable risk-adjustment model were made, including deleting any one of the patient-level predictors from the model or using alternative cut-points for assigning a hospital’s CAS volume status. When we re-analyzed the data using a hierarchical model that included only patient level predictors, only two hospitals (both high-volume centers) were identified as having RSMR’s that were statistically higher than the national mean.
Hospitals certified by Medicare for the provision of CAS varied widely in their clinical outcomes, with risk-standardized 30-day mortality rates varying approximately five-fold between the hospitals with the best and worst outcomes. Of the 927 hospitals in 2009–2011 that were certified to provide CAS and that submitted at least one claim for CAS to Medicare, 14 hospitals had risk-standardized mortality rates that statistically exceeded the national mean mortality rate of 2.0%. An even larger number of hospitals would have been identified as having excessively high risk-standardized mortality if the optimal 30-day CAS risk-standardized mortality rate were assumed to be lower than the observed national mean of 2.0%
Medicare’s CAS coverage policy is among the few national technology coverage decisions explicitly designed to assess (and reassess) the quality of care that hospitals provide, and to restrict coverage of the technology only to hospitals that demonstrate sufficient quality. Ostensibly the goal of this policy is to ensure that hospitals performing CAS are able to provide stenting with sufficiently high quality, while preventing hospitals that do not provide high-quality care from receiving Medicare payments for CAS.23 As even in the highly optimized settings of clinical trials, the mortality and morbidity rates after CAS were non-trivial,2, 3 and because optimal use of CAS requires clinical expertise, institutional experience, and an extensive array of advanced technologies,5 efforts to restrict reimbursements solely to the hospitals that provide high-quality CAS appear justified. While Medicare’s certification program has restricted the number of hospitals that provide CAS to Medicare beneficiaries and thus has potentially prevented some low-quality hospitals from providing CAS,23 we found that there were important, clinically meaningful differences in CAS outcomes among the nation’s 927 Medicare-certified CAS hospitals in 2009–2011. Importantly, a small number of outlier hospitals can be identified that had death rates that were statistically higher than the national mean. These findings could easily and inexpensively be incorporated into Medicare’s recertification process to further refine the selection and certification of hospitals that are providing CAS with sufficient quality.
Prior studies have investigated variation in provider outcomes after carotid stenting. Nallamothu et al used 2005–2007 data from the Medicare program to determine that lower annual CAS procedure volumes and limited clinician experience with CAS were significant predictors of post-procedure mortality,24 with more experienced and high-volume CAS clinicians having markedly lower 30-day mortality rates. Our study was not designed to capture the experience levels of individual clinicians performing CAS, but nevertheless we did find a similar inverse relationship between CAS procedure volume and 30-day mortality at the hospital level—a factor that was included in our risk-standardized prediction model. Our findings suggest that the variation in clinician-level outcomes observed by Nallamothu et al. is similarly observable at the hospital level and could hypothetically be incorporated in Medicare’s quality assessment program for CAS hospitals.
Goodney et al. found substantial small-area geographic variation in the frequency that CAS is used to treat carotid stenosis, with case rates per 1000 Medicare beneficiaries varying ten-fold across U.S. Hospital Referral Regions (n=306) in 2007. Our study showed similar variation in the frequency that hospitals performed CAS relative to the number of Medicare beneficiaries each hospital admitted annually. Nevertheless our findings did not suggest any discernible pattern to the geographic location or urban/rural status of hospitals with risk-standardized mortality rates that were statistically higher than the national mean.
The assessment of health care quality has long been approached comprehensively using the framework of Donabedian,25 who delineated the domains of structure, process, and outcome as the three essential aspects of quality assessment. By necessity, the Medicare program was forced to rely on structural aspects alone to assess hospitals for CAS quality. The patient registry required of all hospitals receiving Medicare payments for CAS provides a window into process measures of quality (e.g., whether patients undergoing CAS meet established clinical criteria for the procedure). Our study indicates that clinical outcomes could provide a valuable third source of information for the quality assessment process. The combination of information from the three classic quality assessment domains is likely to maximize Medicare’s ability to meet its policy goals for CAS institutional certification.
Whether hospital-level factors such as procedure volume should be included in hospital comparison models such as these is a controversial issue.14, 15 Unlike patient-level effects, hospital effects in a hierarchical (Baysian) model influence the “prior” assumption about hospital performance, thus if low-volume hospitals as a class have worse outcomes, the “prior” mortality assumption at any given low-volume hospital will be higher than at high-volume hosptials. Silber et al. have demonstrated that models that ignore volume effects are highly unlikely, by design, to ever identify low-volume centers as outliers—a result that we have duplicated in our findings.15 As prior reports suggest a clinically significant volume-outcome relationship for CAS,24 we chose to include hospital volume in our risk-standardization models.
Our investigation was limited by the lack of detailed clinical information on administrative claims—many of these details are captured in CAS registries and are likely to provide important information on CAS outcomes and quality, including the degree of carotid stenosis, symptomaticity, and impending major (non-carotid) surgery.26 While claims data lack the clinical detail of data sources such as the CAS registry, claims data nevertheless have been previously used successfully to compare risk-standardized outcomes for patients with acute myocardial infarction (AMI) and acute heart failure (AHF). As hospitals have greater ability to select their CAS recipients than to select the patients who are hospitalized with AMI/AHF, it is possible that unobserved confounders may excessively influence a hospital’s RSMR after CAS. It is also possible that up-coding/over-coding of comorbidities by hospitals could artificially deflate a hospital’s RSMR, by making the hospital’s CAS patients appear “sicker” than they actually were. For these reasons an evaluation of clinical outcomes using claims data would appropriately only be a single component, rather than the totality, of the hospital quality assessment process.
Our study examined all-cause mortality among CAS recipients as cause-of-death is not universally identifiable in Medicare claims. It is possible that some deaths in the 30-day follow-up period were not attributable to CAS, and thus some hospitals may have had high RSMR when the post-CAS deaths were caused by unrelated factors. These and other details would likely be easily uncovered in a thorough case review. Hence, it would be inadvisable to base certification decisions solely on analyses such as ours—however, our analysis does permit the identification of a small number of hospitals whose clinical performance invites greater scrutiny, and for which a comprehensive case review could be conducted by CMS.
We used relatively few hospital variables in our prediction model compared to the number of patient variables (Table 1). It is possible that the inclusion of hospital variables more specific to CAS outcomes might provide better discrimination between hospitals. Silber et al. demonstrated the critical importance of including hospital-level predictors in the Hospital Compare method of investigating risk-standardized outcomes, and in fact, our study revealed several hospital factors that were independent predictors of adverse outcomes.15 As hospitals are currently required to report structural characteristics to Medicare in the course of obtaining CAS recertification, this may be an ideal mechanism for collecting data that could be incorporated in a more detailed hospital-level mortality model.
The current study depends on the Social Security Death Master File to identify mortality. Greater accuracy about 30-day mortality might be obtained were Medicare to require hospitals to ascertain 30-day mortality status and capture this information in the CAS patient registries that Medicare requires of all CAS hospitals, however this ascertainment might be difficult to validate.
Risk-standardized hospital outcomes after CAS vary markedly across the selected subset of U.S. hospitals that have been certified by Medicare to be reimbursed for CAS. Comparisons of hospital outcomes can be readily made using Medicare claims combined with hospital characteristics, and these comparisons can identify a small subset of hospitals that may be appropriate targets for increased scrutiny at the time of recertification. Such efforts would be in line with Medicare’s programmatic goal of ensuring that Medicare beneficiaries receive CAS at hospitals that provide this high-technology care with acceptable quality.
This study was supported by grant 1R01HS018403 from the Agency for Healthcare Research and Quality, which had no role in the design and conduct of the study, in the collection, management, analysis, and interpretation of the data, or in the preparation, review, and approval of the manuscript, or the decision to submit the manuscript for publication. Dr. Groeneveld had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. None of the authors have any conflicts of interest relevant to the topic, or work presented, in this manuscript.
The authors have no conflicts of interest (real or perceived) in relation to this study.