|Home | About | Journals | Submit | Contact Us | Français|
The impact of quality improvement incentives on nontargeted care is unknown and some have expressed concern that such incentives may be harmful to nontargeted areas of care. Our objective is to examine the effect of publicly reporting quality information on unreported quality of care.
The nursing home Minimum Data Set from 1999 to 2005 on all postacute care admissions.
We studied 13,683 skilled nursing facilities and examined how unreported aspects of clinical care changed in response to changes in reported care after public reporting was initiated by the Centers for Medicare and Medicaid Services on their website, Nursing Home Compare, in 2002.
We find that overall both unreported and reported care improved following the launch of public reporting. Improvements in unreported care were particularly large among facilities with high scores or that significantly improved on reported measures, whereas low-scoring facilities experienced no change or worsening of their unreported quality of care.
Public reporting in the setting of postacute care had mixed effects on areas without public reporting, improving in high-ranking facilities, but worsening in low-ranking facilities. While the benefits of public reporting may extend beyond areas that are being directly measured, these initiatives may also widen the gap between high- and low-quality facilities.
Public reporting of quality information is a potentially powerful tool to improve health care quality. Moving quality information into the public domain may improve quality of care by giving consumers the information necessary to choose high-quality providers. Additionally, health care providers may respond to this information by improving the quality of care they provide. Because improving quality in this way is theoretically appealing and the potential for a positive effect on quality is substantial, public reporting is increasingly being adopted for hospitals, health plans, nursing homes, home health agencies, and physicians.
While public reporting has the potential to improve quality of care in areas that are being measured and reported, one potential limitation is the impossibility of measuring all the important aspects of care. By necessity, measures of clinical care are limited to what is measurable, and what is measurable is not always what is most important. The impact of quality improvement incentives on unreported care is unknown, and some policy makers and clinicians have expressed concern that there may be unintended and negative consequences to public reporting, including causing providers to focus their attention on measured aspects of care while neglecting unmeasured but important areas of care (Casalino 1999; Werner and Asch 2005).
Our objective is to examine the effect of publicly reporting quality information on unreported quality of care. We do this in the setting of Nursing Home Compare, a public reporting initiative launched by the Centers for Medicare and Medicaid Services (CMS) in 2002 to address quality deficits in nursing homes. Using clinical measures of quality, Nursing Home Compare publicly rates all Medicare- and Medicaid-certified nursing homes in the United States on the care they provide for short-stay and chronic-care residents. In this setting, we examine how unreported aspects of clinical care changed in response to public reporting of other aspects of care.
Little prior work has examined the impact of performance measurement on unmeasured quality of care. One randomized, controlled trial (Mohide et al. 1988) examined the impact of a quality improvement intervention in nursing homes on areas of care that were not targeted by the intervention. The study found that while care for the targeted conditions improved, there was no change in the care for the nontargeted condition. More recently, Asch et al. (2004) examined the effects of performance measurement in the Veterans Health Administration (VHA) on targeted and nontargeted conditions. For areas of care that were targeted by performance measurement within the VHA, they found that patients from the VHA were more likely to receive recommended care than non-VHA patients. However, the difference in care between VHA and non-VHA patients in conditions that were not targeted by VHA performance measurement were smaller and barely reached statistical significance. Another recent study tested whether a quality improvement intervention changed nontargeted care for vulnerable elders in an ambulatory care setting (Ganz et al. 2007). This observational study of a practice redesign intervention found that while targeted care processes improved in the intervention practices compared with control practices, there was no change in the nontargeted care processes in either practice setting. Finally, one prior study has tested the effect of quality improvement incentives on targeted and nontargeted care, examining a hospital pay-for-performance program for acute myocardial infarction (Glickman et al. 2007). The study found that neither targeted nor nontargeted care processes significantly changed in response to pay-for-performance.
Our study contributes to this existing literature in two important ways. First, only one prior study has examined changes in nontargeted quality of care in the face of market-based quality improvement incentives (Glickman et al. 2007), but in finding no improvement in targeted measures, it provides little evidence of how nontargeted care changes when health care providers improve care in targeted areas. While other work has not found a significant change in nontargeted care, quality improvement from market-based incentives such as public reporting of quality may provide stronger incentives for improving targeted quality and therefore stronger potential for effects on nontargeted quality. Second, prior work has not directly correlated changes in targeted care to changes in nontargeted care within health care providers. While others have answered the question, “Does nontargeted quality change on average?” in this study we ask the question, “Does nontargeted quality change in response to changes in targeted quality?”
Holmstrom and Milgrom's (1991) theory of multitasking predicts that measuring and rewarding quality in some areas may harm quality in other areas. This is specifically the case when quality is multidimensional and when quality improvement efforts target only some dimensions of quality.
In the setting of health care, market-based quality improvement incentives, such as public reporting and pay for performance, typically reward only a subset of all measures. In addition, some aspects of health care quality are difficult to measure. Thus, large segments of health care quality are currently unrewarded, and in some cases unmeasured. Because quality is multidimensional, multitasking theory predicts that providers will divert resources away from these unrewarded and unmeasured aspects of quality.
The degree to which rewarded and unrewarded measures evaluate the same dimension of care may predict whether unrewarded measures improve in response to improvements in rewarded measures. If rewarded and unrewarded quality measures are related to the same quality dimension, we may expect that efforts focused on improving quality tied to incentives will spill over to unrewarded areas, causing both to improve. Conversely, measures related to different quality dimensions may be more likely to diverge when incentives are related to only one measure, as focusing limited resources on rewarded care may crowd out unrewarded care.
The approach to quality improvement may also predict whether unrewarded care improve in response to improvements in rewarded areas. If improvements in quality are driven by structural changes, such as by hiring more professional nursing staff, we may expect both rewarded and unrewarded areas of care to improve (to the extent that both are related to nurse staffing). Alternatively, if improvements in quality are driven by targeted changes, such as changes in protocols and work organization, resources may be diverted away from unrewarded quality, resulting in worsening of unrewarded quality while rewarded quality improves.
We test the effect of public reporting on unrewarded care in the setting of public reporting of nursing home quality, or Nursing Home Compare. In 2002, the CMS launched the Nursing Home Quality Initiative, an effort to improve quality of care in nursing homes. Working with measurement experts, the National Quality Forum, and a diverse group of nursing home industry stakeholders, CMS adopted a set of nursing home quality measures. CMS publicly released this information in Nursing Home Compare, a web-based resource (http://www.medicare.gov/NHCompare) that details quality of care at all Medicare- or Medicaid-certified nursing homes. Nursing Home Compare began rating nursing homes based on 10 quality measures, three of which evaluate the quality of postacute care. On November 12, 2002, the publication of these quality measures was launched nationally, allowing consumers to compare nursing home quality measures across 17,000 nursing homes nationwide (Centers for Medicare and Medicaid 2002; Harris and Clauser 2002).
We focus our analysis on the quality measures for skilled nursing facilities (SNF) related to postacute care, because a substantial number of measures of postacute care quality have been validated but not reported in Nursing Home Compare, and test whether reported and unreported aspects of care change after the launch of Nursing Home Compare in November 2002. To test whether changes in unreported care are related to quality improvement on reported measures, we stratify our analyses by whether SNFs improved or were high scoring on reported measures after Nursing Home Compare was launched. Finally, we test for changes in nurse staffing after the launch of Nursing Home Compare, as we expect increases in nurse staffing to have positive spillovers onto unrewarded aspects of care.
The primary data source for our analyses is the nursing home Minimum Data Set (MDS) for years 1999–2005. The MDS contains detailed clinical data that is collected at regular intervals for every resident in a Medicare- or Medicaid-certified nursing home. Data on residents‘ health, physical functioning, mental status, and psycho-social well-being have been collected electronically since 1998. These data are used by nursing homes to assess the needs of and develop a plan of care unique to each resident and by the CMS to calculate Medicare prospective reimbursement rates. A recent large field reliability trial of MDS was conducted to verify the quality of the data. Research nurses shown to be highly reliable among themselves undertook up to 30 reliability assessments in each of 209 randomly selected facilities. Results reveal that over 85 percent of MDS data elements have adequate interrater reliability (κ>0.6) and those below that threshold were very low prevalence binary indicators showing high levels of agreement (Mor et al. 2003). Other researchers have found considerable evidence for the consistency of clinical data in the instrument (Gambassi et al. 1998). Because of the reliability of these data and the detailed resident-level clinical information contained therein, they are considered the best available data for measuring nursing home quality and thus are the source for the quality measures reported on Nursing Home Compare.
Calculating the quality measures directly from the MDS allows us to produce the quality measures both before and after Nursing Home Compare was released. We use the technical definitions of the quality measures provided by CMS (Morris et al. 2003) to calculate each nursing home's quality measures for postacute care residents over the time period of the study. We calculated all the postacute care quality measures that were publicly reported on Nursing Home Compare when it was launched in November 2002: percent of short-stay residents who did not have moderate or severe pain, percent of short-stay residents without delirium, and percent of short-stay residents whose walking improved. We followed all the conventions of the CMS quality measures to calculate the postacute care measures: each measure is calculated quarterly over two-quarters of data on all residents with a 14-day MDS assessment; only those residents who stay in the facility long enough to have a 14-day assessment are included in the calculation of the aggregate quality measure; only facilities with at least 20 cases during the target time period are included. After calculating the measures, our results were benchmarked against the publicly reported quality measures that are available from CMS. The results of our calculations for these quality measures differed from the results reported by CMS between 0 and 0.15 percentage points, indicating an excellent match between the results of our measurement calculations and those published by CMS.
We also used MDS to calculate postacute care quality measures identified by Abt Associates Inc. as valid, but not chosen by CMS to be publicly reported. In 2004, Abt assessed the validity of a large number of measures of the quality of postacute care (Moore et al. 2005), classifying quality measures as Level I (highest validity), Level II (moderate validity), and Level III (not valid). The technical details of this validation process have been previously described (Moore et al. 2005). For the purposes of our study, we included all the unreported quality measures with Level I or Level II validity. We calculated nine valid measures of quality of care that were not reported in Nursing Home Compare—eight with Level I validity and one with Level II validity1—based on the technical definitions for these measures (Moore et al. 2005).
While we expect two of the measures of unreported quality (improved pain and locomotion) that are closely related to two reported quality dimensions (pain and walking) to change with reported quality, current evidence does not lead us to make predictions about the extent to which the remaining unreported measures may be related to or share production resources with reported measures.
All the quality measures were rescaled so that a higher score indicates higher quality of care. Reported and unreported quality measures are summarized in Table 1.
To test for changes in reported and unreported quality when Nursing Home Compare was launched, we used facility-level linear regressions where each quality measure (reported and unreported) was a function of Nursing Home Compare indicator variables, time-varying covariates, and SNF fixed effects. We tested for changes in quality measures with the launch of Nursing Home Compare in November 2002 in two ways: (1) using year indicator variables (2001–2005, omitting 2000) we tested whether quality changed at the launch of Nursing Home Compare, between 2002 and 2003; and (2) using a pre–post indicator variable (2000–2002 versus 2003–2005), we tested whether the quality level differed in the pre-Nursing Home Compare period compared with the post-Nursing Home Compare period. We included the following time-varying covariates to control for the significant changes the case mix of patients being admitted to postacute care during our study time period: the mean Cognitive Performance Scale (Morris et al. 1994), the mean activities of daily living (ADL) summary score (a measure of ADL) (Morris et al. 1999), and the percentage of SNF residents admitted in each Resource Utilization Group (RUG). Because all the quality measures are calculated only on residents who remain in postacute care for at least 14 days, we also control for the “censoring” rate at each facility using quarterly measures of the proportion of all postacute care admissions that remain in postacute care for 14 days at each SNF. In all the cases, robust standard errors were used to account for nonindependence of observations from the same facility.
To specifically test how unreported quality changed in response to changes in reported quality of care, we stratified the above analyses of unreported quality by whether facilities improved or were high ranking on reported measures. We defined facilities that improved or were high ranking on reported measures as those that improved on all the three reported measures between 2001 and 2005 plus facilities that scored in the top one-third on all the three reported measures after Nursing Home Compare was launched. Conversely, facilities that did not improve or were low ranking on reported measures were defined as those that stayed the same or worsened on all the three reported measures between 2001 and 2005 plus facilities that scored in the lower one-third on all the three reported measures after Nursing Home Compare was launched.
We also examined changes in nurse staffing over the study period to investigate whether structural improvements may be responsible for positive spillovers to unreported quality of care, calculating nurse staffing measures from On-Line Survey Certification and Reporting System. We followed standard procedures to calculate staffing ratios, assuming that each full-time equivalent staff member works 70 hours in a 2-week period and dividing the staffing hours per day by the number of residents in the facility (Abt Associates Inc. 2001). We measured skilled staffing intensity as registered nurse (RN) plus licensed practical nurse (LPN) hours per resident day.
We checked the robustness of our results to model choice. In the base model described above, we used ordinary least squares. However, because our dependent variable is a proportion, and thus bounded between 0 and 1, it may violate the assumptions of OLS. This is particularly the case for quality measures that contain a large number of zeros or ones. We test the sensitivity of our results to this possibility in two ways. First, we transform our dependent variable using the logit transformation, which results in a normally distributed dependent variable, and re-estimate these regressions using OLS. While this approach allows us to adequately test for the statistical significance of unbiased estimates, it introduces the need to retransform the dependent variable, making interpretation of the results less straightforward. Therefore, we also use generalized estimating equations with binomial family and logit link. This approach is attractive because the link function directly characterizes how the expectation on the original scale is related linearly to the predictors, which overcomes problems with retransformation of the dependent variable (Papke and Wooldridge 1996). However, this approach produces a matrix-weighted average of the between-SNF and within-SNF effects, rather than the within-SNF effects that are the goal of this analysis.
We also re-estimated our base model on a balanced panel of SNFs present through all the 6 years of the study (n=8,225 facilities). By excluding facilities that either dropped out of the panel or entered the panel in a later time period, we can confirm that the effect we observe of Nursing Home Compare on reported and unreported quality is due to changes that occurred within the same group of facilities rather than changes in the composition of the panel.
Multivariate within-SNF analyses showed that all the three reported measures of quality improved after the launch of Nursing Home Compare (Table 2). Between 2002 and 2003, when Nursing Home Compare was launched, the percentage of patients whose pain was controlled improved by 2.6 percentage points (on a base of 76 percent), the percentage of patients without delirium improved by 0.5 percentage points (on a base of 96 percent), and the percentage of patients with improved walking improved by 0.4 percentage points (on a base of 8 percent). Improvements in the quality measures averaged over the postperiod were larger.
Several of the unreported measures also improved in 2003 immediately after Nursing Home Compare was launched (Table 2): improved pain (2.5 percentage points on a base of 53 percent), locomotion (0.3 percentage points on a base of 32 percent), shortness of breath (0.6 percentage points on a base of 83 percent), and bladder incontinence (0.6 percentage points on a base of 49 percent). These improvements persisted throughout the postperiod. The measure of respiratory infections worsened slightly immediately after Nursing Home Compare in 2003 (−0.3 percentage points on a base of 95 percent), but then improved again and was on average better in the 3 years after Nursing Home Compare was launched compared with before. Several unreported measures worsened in 2003 immediately after Nursing Home Compare was launched: urinary tract infection (−0.2 percentage points on a base of 77 percent), ADL functioning (−0.9 percentage points on a base of 51 percent), and early loss ADLs (−0.8 percentage points on a base of 28 percent). Quality in these areas steadily trended downward from 2000 through 2005, suggesting these declines may not be associated with Nursing Home Compare.
To test whether changes in unreported measures were related to changes in reported measures, we stratified the analyses based on whether or not a SNF was high scoring on reported measures (Table 3). In general, we found facilities that were high scoring on reported measures improved on unreported measures. Specifically, among high-scoring facilities, the unreported measures that were related to the same quality dimension as reported measures (improved pain and locomotion) had gains in quality, as did several other measures (shortness of breath and bladder incontinence). At the same time, facilities that were low scoring on reported measures had significantly smaller improvements, had no significant change, or worsened on unreported measures after Nursing Home Compare was launched. For unreported measures that worsened on average, the decrement in quality was generally larger among low-scoring facilities than high-scoring facilities.
Finally, we examined changes in nurse staffing related to Nursing Home Compare, as nurse staffing in one way nursing homes may improve quality of care that may lead to an improvement in unreported quality, rather than a worsening. We focus on professional nurse staffing (RNs and LPNs) because skilled services are by definition of fundamental importance for postacute care in SNF. We find that professional nurse hours per resident-day declined over the study period (Table 4), for reasons that may have to do with financial pressures (Konetzka et al. 2004) and that we assume are unrelated to public reporting. However, the relative declines in professional staffing after Nursing Home Compare are consistently smaller for high-scoring facilities than for low-scoring facilities. The maintenance of relatively higher staffing in high-scoring facilities is consistent with the generally larger positive spillovers to unreported quality found at these facilities.
Our robustness checks confirmed that our choice of model—linear regression with facility fixed effects—was not driving our results. Our findings did not change qualitatively using a linear regression with logit transformation of the dependent variable, generalized estimating equations, or a balanced panel of SNFs.
There has been significant concern that quality improvement incentives such as public reporting may inadvertently harm care that is not directly targeted by the incentive. In the setting of public reporting of postacute care, we find that overall both unreported and reported care improved following the launch of public reporting. Improvements in unreported care were particularly large among facilities with high scores or that significantly improved on reported measures, whereas low-scoring facilities experienced no change or worsening of their unreported quality of care.
Our findings among high-ranked facilities are more consistent with the theory that the positive effect of public reporting spills over into other important but unreported areas of nursing home care and less consistent with the theory that measuring and reporting quality in some areas crowds out quality in other areas. The positive effect of public reporting on unreported quality suggests that public reporting may have induced SNFs to make structural and organizational changes that had a positive effect on unreported quality.2 Our staffing results are consistent with this conclusion. Within the context of a secular decline in professional nurse staffing ratios over the study period, high-scoring facilities were able to maintain higher professional nurse staffing ratios than low-scoring facilities, facilities that in some cases had worsening in unreported quality of care. These findings lend support to the role of nurse staffing in changes in both reported and unreported quality of care.
On the other hand, we found that facilities that were low scoring on reported measures also failed to improve on unreported measures and, in some cases, worsened on these measures. This could be due to a lack of resources, knowledge, or will to pursue any type of quality improvement initiative, or a failed effort to improve on reported measures that nonetheless drew resources from unreported quality. Thus, the main unintended consequence of public reporting of quality in postacute care may not be a growing divide between reported and unreported aspects of quality but rather a growing divide between providers that are more and less able to achieve quality improvement.
The relative changes in unreported quality are, in most cases, smaller than the changes observed in reported quality. Most of these changes are small in magnitude (the absolute improvements in most unreported quality measures are one percentage point or less, which translates into a relative improvement of 1–2 percent). There are multiple reasons for small improvements in quality, including measurement error and loss to follow up, biasing the results toward the null. By contrast, the relative improvements in reported quality are larger. The smaller changes we find in reported quality compared with unreported quality suggests that there were both spillovers to unreported quality from structural quality improvement efforts and that some resources were directed specifically toward improving reported quality.
Our study has several limitations. As with any observational study, there are potential sources of bias that could impact our results. It is difficult to definitively attribute changes in quality (both reported and unreported) to Nursing Home Compare using a pre–post design. Because we study the national launch of Nursing Home Compare, it is impossible to know whether observed changes would have occurred in the absence of Nursing Home Compare. There may also be unmeasured SNF characteristics that are associated with changes in both reported and unreported care other than Nursing Home Compare. For example, good management may lead to better scores on both reported and unreported measures. To account for this, we include SNF-level fixed effects so that we estimate within SNF changes in quality, accounting for time-invariant differences in SNF. In addition, we include a set of SNF-level control variables to account for time-varying changes in SNF related to changes in patient case mix; however, bias from omitted time-varying attributes may remain. It is possible that the relationship we demonstrate between reported and unreported quality is due to changes in the accuracy of the data rather than true changes in quality of care. While other work has found that changes of this nature explain some quality improvement (Green and Wintfeld 1995; Roski et al. 2003), this is less likely to happen in the nursing home quality measures calculated from the MDS. Electronic MDS data collection started in 1998, long before Nursing Home Compare was launched, and has been used to determine Medicare payment since that time, increasing nursing homes‘ incentive to accurately report these data for several years before the launch of Nursing Home Compare. Finally, we do not measure changes in overall quality of care, but rather individual elements of unreported care. Thus, our analysis is subject to the global limitation of measuring quality inherent in Nursing Home Compare and similar interventions—individual measures may not represent overall quality, reported or unreported.
Our findings provide good news for quality improvement incentives such as public reporting and pay-for-performance in that fears of serious “crowding out” of unreported quality do not appear to be substantiated in many SNFs. Reported and unreported quality within SNFs generally improved together after public reporting was initiated, particularly in high-ranking SNFs. These findings suggest that investments in quality improvement in the face of market-based incentives, if successful, may improve other areas of care as well. At the same time, lack of improvement on both reported and unreported measures was also common. Fears that public reporting will lead to a growing divide between high- and low-quality providers deserves further investigation.
Joint Acknowledgments/Disclosure Statement: This research was funded by a grant from the Agency for Healthcare Research and Quality (R01 HS016478-01) and the University Research Foundation of the University of Pennsylvania. Rachel Werner is funded in part by a VA HSR&D Career Development Award. This project is funded, in part, under a grant from the Pennsylvania Department of Health. The Department specifically disclaims responsibility for any analyses, interpretations, or conclusions.
Prior dissemination: This research was presented at AcademyHealth, Washington, DC, June 2008 and American Society of Health Economics, Durham NC, June 2008.
1All the unreported measures included in this study had Level I validity except for “percent of short-stay residents with urinary tract infections,” which had Level II validity.
2It may also be that simply knowing that facility quality would be under public scrutiny provided an impetus for facilities to direct resources to general quality improvement efforts.
Additional supporting information may be found in the online version of this article:
Appendix SA1: Author Matrix.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.