|Home | About | Journals | Submit | Contact Us | Français|
The Centers for Medicare and Medicaid Services publicly-reports hospital-specific risk-standardized, 30-day, all-cause, mortality rates (RSMRs) for all hospitalizations among fee-for-service Medicare beneficiaries for acute myocardial infarction (AMI), heart failure (HF), and pneumonia at non-Federal hospitals.
To examine the performance of the statistical models used by CMS among veterans at least 65 years of age hospitalized for AMI, HF, and pneumonia in Veterans Health Administration (VHA) hospitals.
Cross-sectional analysis of VHA administrative claims data between October 1, 2006 and September 30, 2009.
13,046 veterans hospitalized for AMI among 123 VHA hospitals; 26,379 veterans hospitalized for HF among 124 VHA hospitals; and 31,126 veterans hospitalized for pneumonia among 124 VHA hospitals.
Hospital-specific RSMR for AMI, HF, and pneumonia hospitalizations calculated using hierarchical generalized linear models.
Median number of AMI hospitalizations per VHA hospital was 87. Average AMI RSMR was 14.3% (95% Confidence Interval (CI), 13.9%-14.6%) with modest heterogeneity among VHA hospitals (RSMR range: 8.4%-20.3%). The c-statistic for the AMI RSMR statistical model was 0.79. Median number of HF hospitalizations was 188. Average HF RSMR was 10.1% (95% CI, 9.9%-10.4%) with modest heterogeneity (RSMR range: 6.1%-14.9%). The c-statistic for the HF RSMR statistical model was 0.73. Median number of pneumonia hospitalizations was 221.5 Average pneumonia RSMR was 13.0% (95% CI, 12.7%-13.3%) with modest heterogeneity (RSMR range: 9.0%-18.4%). The c-statistic for the pneumonia RSMR statistical model was 0.72.
The statistical models used by CMS to estimate RSMRs for AMI, HF, and pneumonia hospitalizations at non-Federal hospitals demonstrate similar discrimination when applied to VHA hospitals.
In 2007, the Centers for Medicare and Medicaid Services (CMS) began publicly reporting hospital-specific risk-standardized, 30-day, all-cause, mortality rates (RSMRs) for all non-Federal U.S. hospitals for hospitalizations among fee-for-service Medicare beneficiaries for two common and costly medical conditions: acute myocardial infarction (AMI) and heart failure (HF) (1). In 2008, CMS also began publicly reporting hospital-specific RSMRs for pneumonia hospitalizations (1). CMS publicly reports these RSMRs as measures of hospital quality in an effort to promote health care quality improvement by encouraging hospitals to focus on outcomes of care and to allow patients to compare mortality rates among hospitals. Each RSMR is calculated using a statistical model based on CMS administrative claims data for fee-for-service Medicare beneficiaries and was validated using medical record data from a similar population (2-4). Although these measures have been endorsed by the National Quality Forum (NQF) for hospital comparison (5, 6), it is unknown how well they function for different patient populations as a tool for examining hospital quality.
Similarly, the Veterans Health Administration (VHA) monitors the quality of care of veterans hospitalized with AMI, HF, and pneumonia treated within VHA hospitals. To date, however, the VHA has not utilized externally derived risk-adjustment models to compare hospital outcomes. Nevertheless, the VHA has a strong interest in benchmarking care between VHA and the private sector, and one step toward such benchmarking is to validate risk models derived outside the VHA within a veteran cohort. Accordingly, our research objective was to examine the performance of the statistical models used by CMS to calculate hospital-specific RSMRs for AMI, HF, and pneumonia among veterans 65 years of age or older treated in VHA hospitals.
The study population included all veterans 65 years or older hospitalized in VHA hospitals during fiscal years 2007-2009, from October 1, 2006, through September 30, 2009, with a principal discharge diagnosis of AMI, HF, or pneumonia as determined using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic codes. Data were obtained from the VA Patient Treatment File, which contains administrative records for all hospital admissions to VA facilities and is housed at the Austin Automation Center, Austin, Texas. We included patients with 12 months of continuous VA eligibility before and after the index admission in order to obtain complete mortality and co-morbidity information. For each condition, to avoid survival bias, we randomly selected one admission for patients with multiple admissions for the same diagnosis during the study period.
For AMI, we included any admission with an ICD-9-CM code of 410.xx, excluding 410.02. For HF, we included admissions with the following ICD-9-CM codes: 402.01, 402.11, 402.91, 404.01, 404.11, 404.91, 428.x, and 404.03, 404.13, and 404.93. For pneumonia, we included admissions with the following ICD-9-CM codes: 480.3, 480.8, 480.9, 481, 482.x, 483.0, 483.1, 483.8, 485, 486, and 487.0. We used both inpatient and outpatient data to determine patient comorbidities, past medical history, and procedure use during the 12 months before the index admission (the first hospital to which the patient arrived). Patients who were transferred between VHA acute care facilities or to a non-VHA acute care hospital during a single hospitalization were linked into a single episode of care with outcomes attributed to the index VHA hospital. In these cases, only comorbidities identified from the index hospitalization were considered to avoid misclassifying complications as pre-existing conditions. We excluded patients discharged alive within 24 hours not against medical advice because of concerns about the accuracy of the principal diagnosis.
We examined death from any cause within 30 days following index admission date. To calculate hospital-specific RSMRs for AMI, HF, and pneumonia, we used the same measures and approach developed for CMS and endorsed by the NQF. These models are based on administrative data, but produce estimates of RSMRs that are good surrogates for estimates from a medical record model, and include patient-specific information on age, sex, and clinical characteristics; further details have been described previously (2-4, 7, 8). Using hierarchical condition categories (9), the AMI model includes 10 cardiovascular clinical characteristics and 15 other co-morbid conditions; the HF model includes 8 and 14; and the pneumonia model includes 7 and 22.
The hospital-specific RSMR for each condition was calculated from estimates derived from two-level (patient and hospital) hierarchical generalized linear models. Model analyses included hospital-level random intercepts assumed to be normally distributed in order to account for the clustering (non-independence) of patients within the same hospital (10). This permitted separation of within-hospital and between-hospital variation after adjusting for patient characteristics.
Patient mortality was determined using the VA Vital Status File, which includes information from the Beneficiary Identification and Record Locator Subsystem (BIRLS) Death File, Social Security Administration (SSA) Death Master File, and Medicare Vital Status File. Combining these four sources of mortality data has shown comparable accuracy and completeness with the National Death Index (11).
We calculated means and distributions of patient characteristics included in the statistical models for risk-standardization. We then estimated the mean observed hospital-specific 30-day all-cause mortality rates and hospital-specific RSMRs, along with their corresponding 95% confidence intervals using bootstrapping techniques, for all three conditions: AMI, HF, and pneumonia. We used the c-statistic from the logistic regression models to assess the performance of the statistical models used to calculate hospital-specific RSMRs. In addition, we calculated the intra-hospital correlation (IHC), which is the ratio of between-hospital variation to total variation in outcomes.
All analyses were conducted using SAS Software, Version 9.1.3 (SAS Institute, Inc., Cary, NC). These analyses were conducted as part of Quality Assurance/Quality Initiative (QA/QI) initiated by the VHA's Office of Quality and Performance and the Ischemic Heart Disease Quality Enhancement Research Initiative (IHD QUERI) to examine the feasibility of using the CMS RSMR measures to assess VA performance and were exempted from review by the Department of Veterans Affairs Institutional Review Board.
There were 13,046 veterans hospitalized for AMI among 123 VHA hospitals, a median of 87 per hospital (Table 1). Median age was 77.0 years and nearly all were male. Rates of co-morbid disease among AMI patients were high, although VHA hospitals varied considerably (Table 2). Anterior AMI location and prior histories of cardiopulmonary respiratory failure or major cancer were the clinical characteristics most strongly associated with mortality (Table 3) and the c-statistic for the RSMR model was 0.79 and the IHC coefficient was 0.082. Finally, average risk-standardized 30-day all-cause mortality was 14.3% (95% Confidence Interval (CI), 13.9%-14.6%) and there was modest heterogeneity among VHA hospitals (Figure).
There were 26,379 veterans hospitalized for HF among 124 VHA hospitals, a median of 188 per hospital (Table 1). Median age was 78.0 years and nearly all were male. Rates of co-morbid disease among HF patients were also high, although VHA hospitals varied considerably again (Table 2). Prior histories of cardiopulmonary respiratory failure, protein-calorie malnutrition, major cancer, or liver disease were the clinical characteristics most strongly associated with mortality (Table 3) and the c-statistic for the RSMR model was 0.73 and the IHC coefficient was 0.064. Finally, average risk-standardized 30-day all-cause mortality was 10.1% (95% CI, 9.9%-10.4%) and there was modest heterogeneity among VHA hospitals (Figure).
There were 31,126 veterans hospitalized for pneumonia among 124 VHA hospitals, a median of 221.5 per hospital (Table 1). Median age was 78.0 years and nearly all were male. Rates of co-morbid disease among pneumonia patients were also high, although VHA hospitals varied considerably again (Table 2). Prior histories of myocardial infarction, protein-calorie malnutrition, renal failure, or major cancer were the clinical characteristics most strongly associated with mortality (Table 3) and the c-statistic for the RSMR model was 0.72 and the IHC coefficient was 0.044. Finally, average risk-standardized 30-day all-cause mortality rate was 13.0% (95% CI, 12.7%-13.3%) and there was modest heterogeneity among VHA hospitals (Figure).
This study demonstrates that the statistical models currently being used by CMS to measure and publicly report 30-day mortality rates after admission for AMI, HF, and pneumonia at non-Federal U.S. hospitals demonstrate good discrimination when applied to VHA hospitals using three years of data. Restricted to patients aged 65 years or older, performances of the statistical models were similar for VHA hospitals and non-Federal hospitals using administrative claims data for fee-for-service Medicare beneficiaries. For instance, c-statistics for the three models were 0.79, 0.73, and 0.72 as compared with 0.71, 0.70, and 0.72 (2, 7). Moreover, model estimates for the patient covariates were predominantly estimated as being in the same direction and of comparable magnitude in VHA and non-Federal hospitals (2, 7). These results suggest that these statistical models could be used by the VHA to measure the quality of care delivered by its hospitals and potentially to benchmark outcomes between the VHA and private sector.
As among non-Federal hospitals caring for fee-for-service Medicare beneficiaries, we observed modest heterogeneity in 30-day RSMRs among VHA hospitals for AMI, HF, and pneumonia. Using three years of data, it is possible to identify outlying performers both at the top and bottom, either for purposes of targeting performance improvement or identifying methods or strategies implemented at top performing facilities that could be widely adopted throughout VHA. Moreover, this might be accomplished with minimal additional cost because the necessary administrative data are already routinely collected within the VHA electronic medical record.
Over the past 15 years, VHA has been a leader in quality improvement, beginning with its nationally integrated healthcare system reform in 1995 that included information technology implementation, performance measurement and reporting, service integration, and realigned payment policies (12-18). VHA has a sophisticated performance measurement system that is comprised of more than 120 performance measures and hundreds of additional process and outcome measures that are tracked on a routine basis (19). Moreover, in creating the National Surgical Quality Improvement Program (NSQIP), VHA pioneered the use of surgical RSMRs to guide quality improvement efforts (20). Similarly, VHA's Inpatient Evaluation Center measures RSMRs for intensive care units to monitor performance (21). VHA has not, however, routinely measured hospital performance using 30-day RSMRs for common medical conditions. Adopting the statistical models currently being used by CMS to monitor AMI, HF, and pneumonia outcomes provides the VHA with another tool to assess performance, compare hospitals, and drive improvement.
There are several considerations in interpreting our results. First, we did not validate the statistical models’ performance against VHA medical record data. However, prior studies of mortality following AMI in VHA using claims data and clinical information have yielded similar estimates, suggesting that this is not likely an important source of bias (22). For example, 30-day mortality following AMI was previously calculated as 16.3% using administrative claims data and 16.0% using detail clinical information (22). Second, we only examined 30-day RSMRs after AMI, HF, and pneumonia hospitalizations for patients 65 years of age or older and our findings could potentially vary when these methods are applied to patient populations that include younger individuals. Third, our study focused only on 30-day RSMRs, not other important dimensions of quality, such as processes of care or patient experiences. Fourth, our data sources do not allow for identification of patients who are admitted for comfort care nor for identification of clinical characteristics diagnosed outside of VHA among dual users. Finally, although the models used for these measures were validated against models based on medical record data, we cannot exclude the possibility of important unmeasured factors influencing our findings.
In conclusion, our study demonstrates that the statistical models currently being used by CMS to measure and publicly report hospital 30-day mortality rates after admission for AMI, HF, and pneumonia at non-Federal hospitals demonstrate similar discrimination when applied to VHA hospitals. These models could thus be used by the VHA to measure the quality of care delivered by its hospitals. These findings provide a benchmark for national VHA hospital performance and suggest that administrative claims data should be used in conjunction with the large array of performance measures that VHA already has in place to inform and promote quality improvement in an effort to achieve better outcomes within the VHA.