In this paper, we report on the impact of the publication of the Nursing Home Compare quality report card. Results are based only on the five QMs included in this analysis and not all the measures included in the report card. We conclude that two of the five QMs we studied did not exhibit a significant change following publication while three did. While we find that these changes are linked to actions undertaken by the nursing homes in response to the publication, we note that the risk adjustment of the CMS QMs is limited, and therefore some of the changes may result from unaccounted changes in patient risks.
Several observations are noteworthy. First, all the changes we find are one-time changes in the level of the QMs around the time of publication and not changes in the slope of the trend line, suggesting that the publication effect was concentrated over a short period of time. For the physical restraint QM, we observe an improvement trend even before the publication, a trend that continues following the publication. The improvement attributable to the publication, of about 1 percentage point, is roughly equivalent to three times the annual decrement in the QM observed both before and after the publication, which was about 0.3 percentage points annually. Thus, the one time impact of the publication generated an improvement equivalent to about 3 years of improvement absent report cards.
Short-term pain did not exhibit an improvement trend either before or after the publication. It did, however, exhibit a much larger one-time effect than the physical restraint QM, of about 2.5 percentage points. Nonetheless, in relative terms, the impact of the report cards on these two QMs is similar. Considering the starting prevalence level of these two QMs, physical restraints at around 10 percent and pain at around 26 percent, the improvement in both is about a 10 percent decrease relative to the starting point.
The third QM to show a response to the publication, pressure ulcers, surprisingly showed deterioration rather than improvement. This finding, while counterintuitive at first glance, might reflect the early impact of attempts to improve quality. Testimonials from quality improvement professionals in nursing homes suggest that frequently the initial response to a perceived quality problem is to focus staff attention on it, which in turn leads to better documentation, and hence higher recorded rates of adverse outcomes. QMs may actually show a decrease in quality if staff records the presence of adverse outcomes more accurately. Once past this stage, quality improvement efforts may have the desired impact on QMs.
Why are the findings heterogeneous? Why do we find evidence that the report cards affect some QMs but not others? Nursing homes may be more knowledgeable about the process changes required to improve some areas or they may have already had a head start, as in the case of physical restraints, where an improvement trend seems to have begun before the publication. Perhaps for some QMs 1 year may not provide sufficient lead time for demonstrable change. This is plausible considering the range of activities that are required to achieve measurable improvement: understanding what contributes to a poor score, identification of corrective plan, implementation of such a plan, and finally having a sufficient cumulative impact to show an improvement in the measured QM. Our study evaluated the impact of report cards 1 year following publication. A study of more recent trends in the QMs could help determine if longer lead times are required before the full effect of report cards can be observed. It could also test our explanation of the observed deterioration in the pressure ulcers QM as the potential precursor to improvement.
Despite the improvement we observe in the physical restraint and the pain QMs, we note that both are still at nonnegligible levels a year following the publication of the report cards. Physical restraints levels average about 8 percent and pain for short-term residents average 23 percent. While it is probably unrealistic to expect zero prevalence rates for these outcomes, one might hope that outcomes can improve beyond these levels. Current data (shown on Nursing Home Compare, August 1, 2007) indicate further improvement since our study period. By 2007, the national average rates declined to 6 percent for physical restraints and to 21 percent for pain for short-term residents. The improvements since 2004 are slow relative to the one-time effect of the report cards, but it does suggest that nursing homes are continuing to improve the care they provide. Further study is required to determine whether these longer-term improvements can also be attributed to the report card or whether they were spurred by other activities designed to improve quality, such as state technical assistance programs (White et al. 2003
) or the QIOs sponsored by CMS (Rollow et al. 2006
We also note that several different actions were associated with improvement, suggesting that nursing homes may have chosen among different successful paths to better quality. Differences in the nature of quality problems across nursing homes may have favored different solutions. It is also possible that cultural and other organizational differences may have influenced choice of action (Shortell, Bennett, and Byck 1998
; Zinn et al. 2005
). Further research is required to gain insights into what type of actions are more likely to be undertaken by what type of nursing home, and which are more likely to be more successful in which environment.
The fact that not all actions led to improvement may reflect lack of knowledge among nursing homes as to what actions might be most effective. This would suggest that publication of quality report cards should be accompanied by educational efforts to help nursing homes identify the best ways to improve care. CMS has contracted with the QIOs to provide such help if requested, and indeed seeking help from a QIO in this study was associated with improvement, similar to findings by Rollow et al. (2006)
. Our findings suggest that efforts of the QIOs might be expanded and that nursing homes should be encouraged to avail themselves of these resources.
The coefficients for the number of actions (see Supplementary Material Table S2) were negative for the linear term and positive for the squared term. The negative linear coefficient implies that the more actions the facility undertook the more its quality improved. The squared positive coefficient indicates diminishing returns: as the number of actions increased the marginal improvement decreased. This suggests that a judicious use of resources should lead nursing homes to adopt only a few corrective actions, rather than attempting to adopt a large number of activities, possibly spreading their resources too thin.
In summary, the analyses presented here suggest that the Nursing Home Compare quality report card led to improvement in some but not all areas. By linking specific actions undertaken by nursing homes in response to the report card to changes in the QMs, we provide more robust evidence about the potential of report cards to influence quality improvement activities compared with prior studies, as well as information about actions and areas of care that are more likely to be influenced by report cards. The policy implications, as the findings, are mixed. Report cards seem to be effective in enhancing some aspects of quality. However, because they are not effective across the board they cannot be relied upon as the only policy instrument aimed at quality improvement.