|Home | About | Journals | Submit | Contact Us | Français|
In 2004, the Hospital Quality Alliance began nationwide public reporting of hospital quality performance. There has been concern that public reporting of the antibiotic timing measure (percentage of patients with pneumonia receiving antibiotics within 4 hours) has led to unintended adverse consequences for patients. In a nationally-representative hospital sample, we sought to determine whether public reporting has been associated with overdiagnosis of pneumonia, excessive antibiotic use, or inappropriate prioritization of patients with respiratory symptoms.
Retrospective analyses of 13,042 emergency department (ED) visits by adult patients with respiratory symptoms in the National Hospital Ambulatory Medical Care Survey, 2001–2005.
Rates of pneumonia diagnosis, antibiotic use, and waiting times to see a physician were compared before and after public reporting. These outcomes were also compared between hospitals with different antibiotic timing scores.
Comparing outcomes before and after antibiotic timing score reporting, there were no differences in rates of pneumonia diagnosis (10% vs. 11% of all ED visits, P = .72) or antibiotic administration (34% vs. 35%, P = .21). Mean waiting times to be seen by a physician increased similarly for patients with and without respiratory symptoms (11 vs. 6 minute increase, respectively; P = .29). After adjustment for confounders, hospitals with higher 2005 antibiotic timing scores had shorter mean waiting times for all patients, but there were no significant score-related trends for rates of pneumonia diagnosis or antibiotic use.
Despite concerns, public reporting of hospital antibiotic timing scores has not led to increased pneumonia diagnosis, antibiotic use, or a change in patient prioritization.
To encourage improvement in hospitals’ quality of care, the Hospital Quality Alliance (HQA) began an initiative to collect and publicly report hospital-level performance on 10 quality measures in 2004.1–3 Over 98% of U.S. acute care hospitals supply performance data to the HQA,4, 5 but concerns have been raised about potential unintended consequences of public reporting.6–8
Hospitals’ responses to the HQA measure “Initial Antibiotic Received within 4 Hours of Hospital Arrival” have been of particular concern. Hospitals feeling pressure to improve antibiotic timing performance could potentially “play for the test” by encouraging the premature (and potentially inaccurate) diagnosis of pneumonia, giving antibiotics indiscriminately to patients with respiratory symptoms, or inappropriately prioritizing patients likely to have pneumonia ahead of others whose medical conditions may be more urgent.9–14
Prior studies from single institutions and self-selected hospitals participating in a pay-for-performance pilot program suggest that incentives tied to pneumonia antibiotic timing scores have led to increased rates of inaccurate pneumonia diagnosis and inappropriate antibiotic administration in emergency departments (EDs).15–17 However, whether public reporting on antibiotic timing has had similar effects on a national scale is unknown.
Since antibiotic timing scores are felt to reflect care delivered in EDs,18 we used a national database of ED visits and compared the care for patients with respiratory symptoms before and after the start of public reporting. We assessed whether these patients were more likely to be diagnosed with pneumonia, prescribed antibiotics, and have shorter waiting times to see a physician (as compared to patients without respiratory symptoms, reflecting patient prioritization). To test the hypothesis that hospitals with higher scores were “playing for the test,” we also assessed differences on these 3 measures between hospitals scoring higher and lower on the pneumonia antibiotic timing measure.
We used patient visit data from the Emergency Department module of the nationally-representative National Hospital Ambulatory Medical Care Survey (NHAMCS), which is administered annually by the National Center for Health Statistics (NCHS).19 A rotating panel of nonfederal, general, and short-stay hospitals participates in the NHAMCS, and visits from each hospital in the panel are sampled approximately every 15 months.
Trained hospital staff record patient and clinical data on standardized forms for each visit. Patient data include demographic information, expected source of payment, and nursing home residence. Clinical data include up to 3 reasons for visit, up to 3 physician diagnoses (in the ED), up to 8 medications administered during the ED visit (but not the timing of medication administration), waiting time to see a physician (2003–2005 only), triage vital signs, and orientation to person, place, and time.20
The NHAMCS ED module collected 182,332 patient visit records between 2001 and 2005. Among contacted hospitals over the study period, 90% to 95% participated. Visits are weighted to allow extrapolation of survey results to national estimates.19 For our analysis the NCHS created anonymous hospital identifiers that allowed longitudinal tracking of each participating hospital.
The NCHS institutional review board approved all NHAMCS protocols, and the confidentiality of the data is protected by law.21
We obtained publicly available hospital-level HQA performance data on the timing of initial antibiotics delivered to patients admitted with pneumonia for 2004 and 2005. Hospital scores were calculated as the percentage of adult patients discharged from the hospital with a diagnosis of pneumonia who received their first dose of antibiotics within 4 hours of hospital arrival. Detailed specifications for this measure are available elsewhere.22
We also linked hospitals’ pneumonia antibiotic timing scores to other hospital characteristics obtained from the 2005 database of the American Hospital Association: number of beds, geographic region, urban location, ownership (for-profit, not-for-profit, and government), status of membership in the Council of Teaching Hospitals (COTH), and percentage of patients covered by Medicare and Medicaid. Of the 507 unique hospitals participating in the NHAMCS ED module during 2001–2005, NCHS staff matched 503 (99%) to their HQA pneumonia antibiotic timing scores and other corresponding hospital characteristics.
Consistent with prior literature, we included for analysis only hospitals reporting stable antibiotic timing scores, defined as scores calculated using at least 25 patient discharges during the year 2005.2 Of the 503 NHAMCS sample hospitals, 118 (23%) were excluded based on this criterion.
Our study population included ED visits during 2001–2005 by patients aged 18 and older whose primary reason for visit was “symptoms referable to the respiratory system” or “diseases of the respiratory system,” excluding conditions limited to the upper respiratory tract (e.g., nasal congestion). Among included visits, the most common specific reasons for visit were cough (50%), shortness of breath (24%), and “labored or difficult breathing” (11%). In supplementary analyses, inclusion of visits for upper respiratory conditions did not substantively alter our results.
Visits were included regardless of patients’ dispositions at the end of each ED visit (e.g., admitted to hospital, transferred, discharged to home). Supplementary analyses limited to visits resulting in hospital admission did not substantively alter our results.
We had 3 major outcome variables: ED diagnosis, antibiotic use, and waiting time to see a physician. We used ICD-9(International Classification of Diseases, Ninth Revision) codes to classify ED diagnoses as pneumonia (with the same codes used for antibiotic timing score reporting),22 bronchitis, congestive heart failure (CHF), or other. It was possible for a single visit to carry more than 1 of these diagnoses, and we counted visits receiving diagnoses in more than 1 category towards the total in each applicable diagnostic category.
Antibiotic use was identified using the NCHS drug classification system.20, 23 As in previous studies, we classified antibiotic use in visits for asthma and CHF as inappropriate when pneumonia was not also an ED diagnosis.16 In supplementary analyses we also included antibiotic use in bronchitis as inappropriate without substantive changes to the results.
We performed 2 main comparisons. First, we analyzed nationwide longitudinal trends and differences in the outcome variables (ED diagnosis, antibiotic use, and waiting time to see a physician) before and after the start of public score reporting among ED visits for respiratory symptoms. We designated January 1, 2004 as the first day of the public reporting period because this was the first day of care that could contribute to publicly available antibiotic timing scores. Because the antibiotic timing measure was first published in October 2003,4 we designated October 1, 2003 as the first day of the reporting period in supplementary analyses with substantively similar results.
Second, we conducted a cross-sectional analysis of relationships between 2005 antibiotic timing scores and the outcome variables, restricting our analysis to ED visits in the public reporting period (2004–2005). We hypothesized that if hospitals were “playing for the test” in order to raise their scores, patients with respiratory symptoms visiting hospitals with the highest 2005 scores would experience the highest rates of pneumonia diagnosis, the highest rates of antibiotic use, and, relative to patients without respiratory symptoms, the shortest waiting times.
We assessed relationships between categorical variables using the chi-squared test. Due to the non-normal distribution of waiting time to see a physician, waiting times were modeled using generalized log-linear regression.
For the longitudinal analysis we constructed multivariable logistic regression models predicting each categorical outcome variable (ED diagnosis and antibiotic use) as a function of time period (pre-reporting vs. during reporting) and the above listed hospital-level characteristics from the American Hospital Association database. To adjust for case severity and patient characteristics known to be associated with ED processes of care,3, 24, 25 the models included patient age, gender, orientation to person/place/time (dichotomous), presence of fever (triage temperature >100.4 degrees Fahrenheit), residence in a nursing home (dichotomous), race and ethnicity (non-Hispanic white, non-Hispanic black, Hispanic, and other), anticipated source of payment (private, Medicare, Medicaid, or other), and season of the visit (winter, spring, summer, or fall). We constructed multivariable log-linear regression models for waiting time, including the same potential confounders. For the cross-sectional analysis we created similar models using hospitals’ 2005 antibiotic timing scores (continuous variable) rather than time period as the major predictor.
In order to assess the contribution of ED care processes to variation in antibiotic timing scores, we constructed supplementary linear regression models predicting the 2005 antibiotic timing score of the hospital associated with each ED visit as a function of pneumonia diagnosis, antibiotic use, and waiting time. We assessed the independent, mutually-adjusted contribution of each explanatory variable to overall variation in scores by comparing the partial R-squared values for each predictor. All analyses were performed using SUDAAN version 9.0.0 to account for data clustering and the complex NHAMCS sampling design.20
There were 385 hospitals participating in the NHAMCS from 2001 to 2005 that reported pneumonia antibiotic timing scores based on at least 25 observations in 2005. There were no significant differences between the NHAMCS hospital sample and the overall population of U.S. hospitals reporting at least 25 antibiotic timing score observations in 2005 [Table 1].
Based on the NHAMCS sample, there were an estimated 40 million (95% confidence interval, 39 to 42 million) ED visits to hospitals by adults with respiratory symptoms between 2001 and 2005. These visits for respiratory symptoms represented 11% of all ED visits, and this percentage was constant across all data years. There were no significant differences between the pre-reporting (2001–2003) and reporting (2004–2005) periods in patient sociodemographic factors (age, gender, race, and ethnicity), nursing home residence, or expected payment source [data not shown]. However, patients in the pre-reporting period were less likely to have a fever (8% pre-reporting vs. 11% during reporting, P = .003) and to be oriented to person, place, and time (77% pre-reporting vs. 85% during reporting, P < .001).
There was no significant difference in the rate of pneumonia diagnosis among ED visits for respiratory symptoms between the pre-reporting and reporting periods [Table 2]. Antibiotic use during ED visits increased among visits resulting in an ED diagnosis of pneumonia (70% pre-reporting vs. 78% during reporting, P = .01), but there were no significant changes in antibiotic use among patients with other ED diagnoses. Mean waiting times to be seen by a physician increased similarly for patients with and without respiratory symptoms (11 vs. 6 minute increase, respectively; P = .29). Adjustment for potential confounders including fever and orientation did not significantly alter these relationships.
In our second analysis we examined cross-sectional relationships between hospitals’ 2005 pneumonia antibiotic timing scores and the outcome variables during public reporting. Pneumonia diagnosis rates did not significantly increase with antibiotic timing score [Table 3]. Patients visiting higher-scoring hospitals were more likely to receive an antibiotic (31% in the lowest score quintile vs. 44% in the highest, P = .004 for trend) and had shorter mean waiting times regardless of respiratory symptoms (66 vs. 38 minutes with respiratory symptoms, P < .001 for trend; 69 vs. 38 minutes without respiratory symptoms, P < .001 for trend; P = .45 for trend in difference between symptom categories). After adjustment for confounders, only the relationship between shorter waiting times and higher-scoring hospitals remained a statistically significant trend.
Differences between hospitals in rates of pneumonia diagnosis, antibiotic prescribing, and waiting time explained only 4% of overall variation in 2005 antibiotic timing scores, suggesting a weak relationship between the measured ED processes of care and hospital antibiotic timing scores [data not shown]. Of this explainable score variation, waiting times accounted for 79%, and antibiotic usage accounted for 3%.
Despite fears that publicly reporting hospitals’ pneumonia antibiotic timing scores would lead to increased pneumonia diagnosis, indiscriminate antibiotic use, and inappropriate prioritization of patients with respiratory symptoms,6, 9–11 we found little evidence of these unintended consequences in a nationally representative cohort of ED visits. Rates of pneumonia diagnosis and overall antibiotic use did not exhibit significant changes over time. Waiting times to see a physician increased similarly for patients with and without respiratory symptoms over 2003–2005, arguing against higher prioritization of patients likely to have pneumonia.
Moreover, cross-sectional analyses of ED visits during the public reporting period revealed that after adjustment for confounders, only waiting times differed significantly between hospitals with higher and lower antibiotic timing scores. Successful efforts to shorten waiting times for all patients would be better described as quality improvements than as adverse consequences.
While hospitals in the highest antibiotic timing score quintile had the highest rates of antibiotic administration for inappropriate diagnoses, the lack of a trended relationship between antibiotic use and timing scores suggests that excessive antibiotic administration does not significantly contribute to the timing score ranking for most hospitals. Also, analysis of overall score variation revealed that only a very small percentage was attributable to differences in antibiotic administration rates. These findings are consistent with the overall stability of antibiotic administration rates before and after the start of public reporting.
However, if there are persistent concerns that hospitals seeking to achieve the highest antibiotic timing scores will, in the future, have increased rates of inappropriate antibiotic administration, then strategies focused on the top-scoring hospitals may be considered. For example, publicly reporting a score “band” rather than an exact score for hospitals scoring above a certain threshold could attenuate the incentive to achieve scores in the range generating these concerns.7, 26
The implications of our longitudinal analysis differ with those suggested by some earlier reports. However, key differences in cohort design between our analysis (which included all patients presenting to EDs with respiratory complaints) and earlier single-institution studies (which included only those patients admitted with a pneumonia diagnosis) may not allow direct comparison of findings.15, 27 A study among self-selected Premier client hospitals in the Hospital Quality Incentive Demonstration (HQID) revealed higher rates of antibiotic use for inappropriate diagnoses (heart failure, asthma, and COPD) at hospitals with higher antibiotic timing scores.16 However, HQID hospitals faced financial incentives directly tied to antibiotic timing performance, and while all hospitals in our analysis publicly reported their antibiotic timing scores, no data were available to identify which hospitals in our analysis had antibiotic timing-based financial incentives. It is possible that compared to public reporting, financial incentives could have different effects on patient treatment patterns. Prior studies agreeing with our findings have demonstrated associations between ED overcrowding (an established contributor to longer waiting times) and lower antibiotic timing scores.28–30
Our study has limitations. First, the NHAMCS does not assess the accuracy of ED diagnoses, so we cannot directly conclude that diagnostic accuracy was unaffected by public reporting. However, because rates of pneumonia diagnosis did not change, any increased inaccuracy would have had to split equally between errors of commission and omission. Second, we lacked complete clinical information (e.g., presence or absence of infiltrate on chest X-ray, comorbid illnesses) to perform fuller case-mix adjustment. Third, our measure of patient prioritization (waiting times to see a physician) did not extend to other important processes of ED care, such as immediacy of imaging.18, 31 Fourth, our analysis included ED visits by patients whose primary reason for visit was a symptom referable to the lower respiratory tract. It is possible that public reporting of antibiotic timing scores would have a different impact on patients with symptoms that were less specific for pneumonia (e.g., delirium, fever, chest pain). Finally, absence of proof is not proof of absence. Failure to detect significant unintended consequences of public antibiotic timing score reporting could be due to insufficient statistical power. However, observed rates of pneumonia diagnosis and antibiotic use were remarkably stable over time, with waiting times for all patients (regardless of respiratory symptoms) accounting for the majority of explainable between-hospital variation in antibiotic timing scores.
External incentive programs designed to encourage health care quality improvement are becoming increasingly common, and concerns about unintended consequences of these programs have surfaced in a variety of clinical settings.32–35 Many of these concerns focus on providers “playing for the test” or “gaming the system.”6, 10, 36 In response to concerns that the pneumonia antibiotic timing measure had led to adverse unintended consequences, the Joint Commission and the National Quality Forum changed the cutoff for timely initial antibiotic in pneumonia from 4 hours to 6 hours after hospital arrival and excluded cases of “diagnostic uncertainty” from score calculation.37 The Infectious Disease Society of America eliminated the time cutoff altogether, recommending only that initial antibiotics be received in the ED.38 To the extent that the outcomes we examined reflect possible adverse unintended consequences, our results do not support these changes.
In summary, we found that during the first 2 years of public reporting of hospital pneumonia antibiotic timing scores, concerns about potential widespread unintended consequences were not substantiated by the national experience. Patterns of ED pneumonia diagnosis and antibiotic use among patients with respiratory symptoms have remained stable over time. The EDs of hospitals with higher antibiotic timing scores distinguish themselves by having shorter waiting times to see a physician, suggesting that these scores communicate information of real importance to patients and payers. However, providers’ concerns about the potential adverse consequences of public reporting and pay-for-performance programs deserve attention.7, 8 Monitoring systems that target these concerns and prospectively measure the patient-level effects of quality improvement initiatives may provide valuable guidance (and reassurance) to policy makers.
Funding source: The study was supported by the Primary Care Teaching and Education Fund of the corresponding author’s hospital, a National Research Service Award from the Health Resources and Services Administration, and a Career Development Award from the Agency for Healthcare Research and Quality. No funding source had a role in the study design, analysis, or manuscript preparation.
Publisher's Disclaimer: This is the pre-publication version of a manuscript that has been accepted for publication in The American Journal of Managed Care (AJMC). This version does not include post-acceptance editing and formatting. The editors and publisher of AJMC are not responsible for the content or presentation of the prepublication version of the manuscript or any version that a third party derives from it. Readers who wish to access the definitive published version of this manuscript and any ancillary material related to it (eg, correspondence, corrections, editorials, etc) should go to www.ajmc.com or to the print issue in which the article appears. Those who cite this manuscript should cite the published version, as it is the official version of record.