Despite fears that publicly reporting hospitals’ pneumonia antibiotic timing scores would lead to increased pneumonia diagnosis, indiscriminate antibiotic use, and inappropriate prioritization of patients with respiratory symptoms,6, 9–11
we found little evidence of these unintended consequences in a nationally representative cohort of ED visits. Rates of pneumonia diagnosis and overall antibiotic use did not exhibit significant changes over time. Waiting times to see a physician increased similarly for patients with and without respiratory symptoms over 2003–2005, arguing against higher prioritization of patients likely to have pneumonia.
Moreover, cross-sectional analyses of ED visits during the public reporting period revealed that after adjustment for confounders, only waiting times differed significantly between hospitals with higher and lower antibiotic timing scores. Successful efforts to shorten waiting times for all patients would be better described as quality improvements than as adverse consequences.
While hospitals in the highest antibiotic timing score quintile had the highest rates of antibiotic administration for inappropriate diagnoses, the lack of a trended relationship between antibiotic use and timing scores suggests that excessive antibiotic administration does not significantly contribute to the timing score ranking for most hospitals. Also, analysis of overall score variation revealed that only a very small percentage was attributable to differences in antibiotic administration rates. These findings are consistent with the overall stability of antibiotic administration rates before and after the start of public reporting.
However, if there are persistent concerns that hospitals seeking to achieve the highest antibiotic timing scores will, in the future, have increased rates of inappropriate antibiotic administration, then strategies focused on the top-scoring hospitals may be considered. For example, publicly reporting a score “band” rather than an exact score for hospitals scoring above a certain threshold could attenuate the incentive to achieve scores in the range generating these concerns.7, 26
The implications of our longitudinal analysis differ with those suggested by some earlier reports. However, key differences in cohort design between our analysis (which included all patients presenting to EDs with respiratory complaints) and earlier single-institution studies (which included only those patients admitted with a pneumonia diagnosis) may not allow direct comparison of findings.15, 27
A study among self-selected Premier client hospitals in the Hospital Quality Incentive Demonstration (HQID) revealed higher rates of antibiotic use for inappropriate diagnoses (heart failure, asthma, and COPD) at hospitals with higher antibiotic timing scores.16
However, HQID hospitals faced financial incentives directly tied to antibiotic timing performance, and while all hospitals in our analysis publicly reported their antibiotic timing scores, no data were available to identify which hospitals in our analysis had antibiotic timing-based financial incentives. It is possible that compared to public reporting, financial incentives could have different effects on patient treatment patterns. Prior studies agreeing with our findings have demonstrated associations between ED overcrowding (an established contributor to longer waiting times) and lower antibiotic timing scores.28–30
Our study has limitations. First, the NHAMCS does not assess the accuracy of ED diagnoses, so we cannot directly conclude that diagnostic accuracy was unaffected by public reporting. However, because rates of pneumonia diagnosis did not change, any increased inaccuracy would have had to split equally between errors of commission and omission. Second, we lacked complete clinical information (e.g
., presence or absence of infiltrate on chest X-ray, comorbid illnesses) to perform fuller case-mix adjustment. Third, our measure of patient prioritization (waiting times to see a physician) did not extend to other important processes of ED care, such as immediacy of imaging.18, 31
Fourth, our analysis included ED visits by patients whose primary reason for visit was a symptom referable to the lower respiratory tract. It is possible that public reporting of antibiotic timing scores would have a different impact on patients with symptoms that were less specific for pneumonia (e.g
., delirium, fever, chest pain). Finally, absence of proof is not proof of absence. Failure to detect significant unintended consequences of public antibiotic timing score reporting could be due to insufficient statistical power. However, observed rates of pneumonia diagnosis and antibiotic use were remarkably stable over time, with waiting times for all patients (regardless of respiratory symptoms) accounting for the majority of explainable between-hospital variation in antibiotic timing scores.
External incentive programs designed to encourage health care quality improvement are becoming increasingly common, and concerns about unintended consequences of these programs have surfaced in a variety of clinical settings.32–35
Many of these concerns focus on providers “playing for the test” or “gaming the system.”6, 10, 36
In response to concerns that the pneumonia antibiotic timing measure had led to adverse unintended consequences, the Joint Commission and the National Quality Forum changed the cutoff for timely initial antibiotic in pneumonia from 4 hours to 6 hours after hospital arrival and excluded cases of “diagnostic uncertainty” from score calculation.37
The Infectious Disease Society of America eliminated the time cutoff altogether, recommending only that initial antibiotics be received in the ED.38
To the extent that the outcomes we examined reflect possible adverse unintended consequences, our results do not support these changes.
In summary, we found that during the first 2 years of public reporting of hospital pneumonia antibiotic timing scores, concerns about potential widespread unintended consequences were not substantiated by the national experience. Patterns of ED pneumonia diagnosis and antibiotic use among patients with respiratory symptoms have remained stable over time. The EDs of hospitals with higher antibiotic timing scores distinguish themselves by having shorter waiting times to see a physician, suggesting that these scores communicate information of real importance to patients and payers. However, providers’ concerns about the potential adverse consequences of public reporting and pay-for-performance programs deserve attention.7, 8
Monitoring systems that target these concerns and prospectively measure the patient-level effects of quality improvement initiatives may provide valuable guidance (and reassurance) to policy makers.