We found that most postacute care quality measures included in Nursing Home Compare significantly improved after the launch of Nursing Home Compare. The one measure that did not improve (delirium) had high performance levels before Nursing Home Compare and, thus, may not have had sufficient room to improve. While the quality improvements were statistically significant, in some cases the magnitude of change was small. The relative improvement in pain control attributable to Nursing Home Compare (based on the differences-in-differences specification) was <1 percent. However, with over 1.5 million patients admitted to postacute care annually, this quality improvement translates into approximately 12,000 fewer patients having moderate to severe pain. Improvement in walking had a relative improvement of 9 percent. While measured quality improved, changes in rates of potentially preventable rehospitalizations (a broader measure of quality) were inconsistent and, in some cases, worsened.
There is ample literature suggesting that in the presence of external incentives, such as public reporting, measured quality improves on average. This has been documented in the case of hospitals (Williams et al. 2005
), physicians (Hannan et al. 1994
), health plans, and previously in nursing homes (Mukamel et al. 2008
). Our findings, which improve upon the methods of prior studies, align with this prior empirical work. While there has been concern that some improvements due to public reporting might be due to changes in case mix (Werner and Asch 2005
), we found that after extensive controls for this, improvements in measured quality remain.
We also found that potentially preventable rehospitalization rates did not change or worsened slightly after the launch of Nursing Home Compare. This may be counterintuitive. However, there are several potential explanations for this finding. First, prior work has found that narrow measures of quality are often not well correlated with broader measures of quality (Bradley et al. 2006
; Werner and Bradlow 2006
;). Although strategies to improve measured quality may spill over to areas that affect quality more generally, and thus in the case of postacute care may reduce rates of hospitalization, this need not be the case. Indeed, efforts targeted at improving the three quality measures used in Nursing Home Compare may not affect rehospitalization from postacute care. Second, the divergence of Nursing Home Compare measures and potentially preventable rehospitalizations may be due to the different cohorts included in these measures. While rehospitalizations are measured for all postacute care admissions, the Nursing Home Compare measures reflect quality only among patients who stay in postacute care for at least 14 days. Characteristics of patients at 5 and 14 days may differ sufficiently to explain these divergent findings. One reason 5- and 14-day patients may differ is that SNFs may selectively discharge (and rehospitalize) postacute care patients before the 14-day assessment when Nursing Home Compare quality measurement occurs. In this way, it is possible that Nursing Home Compare contributes to higher rates of rehospitalization from postacute care. While we control for changing case mix at SNFs using propensity score matching, and control for differential rates of discharge before day 14 across SNFs, we are unable to adequately control for rates of selective discharge in this analysis. Disentangling the effects of selective discharge from true changes in quality is an important area for future research.
Our results should be interpreted in light of the study's limitations. First, results based on propensity score-matched cohorts may not be representative of the changes in quality that occurred as we do not include all patients in our analyses. (Rather, we limit our analyses to the subset of patients with similar characteristics for whom we can best estimate the effects of Nursing Home Compare.) Nonetheless, these results are important from a policy standpoint, as they provide more precise estimates of the effect of public reporting on quality of care. Second, although we extensively control for patient selection based on observed differences between patients before versus after the launch of Nursing Home Compare, unobserved differences that are uncorrelated with observed differences remain a threat to the validity of our findings. Nonetheless, prior work suggests that using propensity score matching in the setting of an exogenous treatment decision, such as the launch of Nursing Home Compare, is a valid approach to account for differences across groups (Heckman, Urzua, and Vytlacil 2006
). Finally, the quality changes we demonstrate may be due to changes in data accuracy rather than true quality changes, particularly given the subjective nature of the Nursing Home Compare quality measures. While other work has found that data changes explain some quality improvement (Green and Wintfeld 1995
; Roski et al. 2003
;), this is less likely in the nursing home quality measures based on MDS. Electronic MDS data collection started in 1998, well before Nursing Home Compare was launched, and has been used to determine Medicare payment since that time, increasing nursing homes' incentive to accurately report these data for several years before the launch of Nursing Home Compare.
Despite these limitations, our study offers important new findings with regard to the role of public reporting in quality improvement. We find that most quality measures improve in response to public reporting even after controlling for secular trends. However, the clinical significance of these improvements may be limited given that improvements in narrow measures of quality of care may not translate into broader quality improvement. To achieve more robust quality improvement, stronger incentives to improve quality may be needed. One possible way to do this is to combine public reporting with pay for performance. While public reporting of quality information is important for reasons beyond quality improvement, such as enhancing accountability of health care providers, alone it may play a positive but limited role in improving quality of care.