On the internet, hospital performance on the Safe Practices survey is ranked by quartiles, which may suggest to consumers that hospitals in higher quartiles are safer than hospitals in lower quartiles. In this first study of the relationship between Safe Practices survey scores and hospital outcomes, we studied a national sample of hospitals and found no relationship between quartiles of score and in-hospital mortality, regardless of whether or not we adjusted for expected mortality risk and certain hospital characteristics ().
A hospital's performance on each Safe Practice is evaluated in the survey via several questions assessing institutional systems to promote awareness, accountability, ability (capacity-building investments), and action (see Box 1). Part of the rationale for designing the survey this way originally may have been to provide “training wheels” and give hospitals credit for creating systems that could eventually support full implementation of a given Safe Practice. However, awarding survey points for hospital administrative structures raises the possibility that the survey is capturing excessive noise, which may be overwhelming an important signal.
It seemed plausible that inpatient mortality could relate to whether actions were being taken to implement the Safe Practice. For this reason the survey's creators proposed the alternative “action-based” scoring method which assigned all points for a Safe Practice based on whether or not the hospital indicated that it had implemented the key action for the practice. In our analysis, using this “action-based” scoring method slightly improved the ability of the survey ranking to predict in-hospital mortality, although the association was not statistically significant. Focusing on actions in the future may improve the survey's ability to discriminate between high-quality and low-quality hospitals.
The Safe Practices survey has recently been shortened from 27 to 13 Safe Practices, largely in response to feedback regarding the considerable time required to complete the survey. Our findings indicate that Safe Practices scores based on the 13 retained practices are unlikely to be significantly associated with inpatient mortality even if scoring is limited to actions taken, as was done in the ASPS-13. The findings do not rule out a modest association between the ASPS-13 and hospital mortality. As presented in our results (), the difference in risk-adjusted mortality between the best and worst quartiles determined by ASPS-13 was 0.27% (p-value for trend = 0.11). This difference in absolute mortality risk corresponds to a “number needed to treat” of approximately 370, which some would find clinically significant. Nonetheless, the likelihood that the observed difference arose by chance is increased by the facts that multiple statistical tests were performed and that associations were not observed in the high risk groups in which they were most strongly hypothesized.
Given the voluntary nature of this self-reported survey, it is also plausible that a lack of correlation with mortality might be due to confounding associated with a “healthy volunteer” effect. Hospitals that have already engaged in improving quality of care may be more likely to want to participate in the Safe Practices survey. However, in our study we found that participation in the survey was not predictive of lower risk-adjusted mortality. This would suggest that our negative findings are unlikely to be due to safer hospitals being more likely to participate in the survey.
To our knowledge, this is the first peer-reviewed analysis that has sought to assess whether better performance on the Safe Practices survey correlates with outcomes that are indicative of improved patient safety. Given that two recently published analyses21, 22
in the business/medical literature used Leapfrog's Safe Practices survey as the metric of hospital quality, it seems clear that additional efforts to explore the validity and value of this well-publicized1, 23, 24
quality measure are needed. As the patient-safety movement grows in importance, hospitals face increasingly complex choices regarding improvement and reporting to the public. Likewise, consumers are faced with multiple sources of information on hospital quality, and are encouraged to choose a facility based on this information. Despite a lack of evidence demonstrating the validity of the Safe Practices survey, the survey is well-known and is influential. For this reason, validating the survey rankings as a measure and as a source of accurate information for consumers and researchers is important.
Our findings suggest, however, that the survey as currently designed does not discriminate between hospitals with higher and lower inpatient mortality. Some will question our choice of overall inpatient mortality as the outcome of interest. We acknowledge that very valid concerns about use of mortality rates as a measure of hospital quality of care have been raised25-27
. Despite this, risk-adjusted mortality rates remain among the most commonly reported outcomes in both the published literature and in public reports of hospital quality. Furthermore, the Institute of Medicine has cited prevention of inpatient deaths as an important reason to focus on patient safety.28
Until consumer-oriented hospital quality reports become explicit as to what benefits consumers can expect from a “safer” hospital, consumers will almost certainly assume a safer hospital is one in which a patient is less likely to die.29
Thus, our analyses are consistent with the most likely consumer interpretation of the data presented on the internet.
Our study results suggest several points regarding the Safe Practices survey that would be valuable for the greater patient safety community to consider. An important issue is whether the SPS is measuring what needs to be measured. Many of the Safe Practices are processes to improve care, yet in its current form the survey is measuring the “processes around the process”. This often gives hospitals credit for what essentially may be good intentions. This also gives points for having structures that may support implementation of a Safe Practice, rather than only awarding credit when the Safe Practice is being consistently followed. Such a scoring system likely is vulnerable to inflation of scores. Of note, most hospitals score quite well on the survey (). It may be that all hospitals truly are doing well on the Safe Practices; however, it seems more likely that the survey as currently designed is unable to discriminate between truly high and low adherence to the Safe Practices.
It may also be that too much is being measured. Steps have already been taken to address this by reducing the survey to 13 practices, but our results suggest that this alone is unlikely to improve the survey's ability to correlate with inpatient mortality.
Finally, it is unknown how well a hospital executive's report of actions being taken to support and implement safety practices correlate with actual activity within that hospital. It may not be reasonable to assume that hospitals are doing what their executives say they are. Our study results call into question the use of a lengthy unaudited survey as a tool for measuring adherence to Safe Practices. Further research to examine how well self-reported activities regarding hospital safety practices correlate with actual activities within the hospital will be helpful in determining the value of self-reported safety data.
Our study has important limitations. The most significant limitation is that our main analysis only had enough power to detect 1% or greater differences in mortality. Although at a policy and epidemiology level the observed 0.2% difference in mortality rate between the first and fourth quartiles of performance using ASPS scoring is potentially important, we calculated that we would have needed on the order of 11 million admissions grouped in 500 hospitals to conclude statistical significance for a difference of this magnitude. Such an analysis was not feasible given the data available from the largest hospital data set in the nation. Furthermore, even if a statistically significant association were to be found with a larger sample, our results indicate that the overall magnitude of the relationship between survey score and mortality would almost certainly remain quite small, and would be of unclear utility to individual consumers.
A second limitation is that we did not study other outcomes that might be responsive to adoption of the Safe Practices, such as complications. It is possible that high performance on the survey does correlate with decreased complication rates, or other outcomes of interest to purchasers and policymakers. However, complication rates are difficult to accurately measure using administrative data. Finally, our primary analysis examines 155 of the 1075 urban hospitals that participated in the Safe Practices survey, raising the question of whether our findings can be generalized to the many survey participants which were not in the Nationwide Inpatient Sample (NIS). However, the NIS is designed to approximate a stratified random sample of U.S. hospitals, and individual hospitals cannot choose whether or not to be included, so there is no reason to believe that survey participants that were included in the NIS differed in any systematic fashion from those that were not in the NIS. Furthermore, we found that survey participants had similar survey scores whether or not they were in the NIS. Hence, although we cannot exclude the possibility that our findings are not generalizable to other hospitals participating in the survey, we have not identified any specific reason to question the generalizability of our findings.
In summary, although a recent study has found that hospitals performing well on the first 3 Leaps of the Leapfrog Hospital Survey do have lower risk-adjusted mortality12
, our analysis was unable to find a correlation between better performance on the 4th
Leap (the Safe Practices survey) and lower risk-adjusted mortality. It is possible that inviting hospitals to self-report on their patient safety practices and then assigning them to quartiles of Safe Practices Score is not an effective way to assess hospital quality and safety. Our findings should not be interpreted, however, as indicating that the Safe Practices are not important, or that the Safe Practices cannot be measured in an informative and valid way. Rather, future work should seek to establish valid methods for assessing adherence to the Safe Practices. Further research is needed to determine how performance on the Safe Practices survey or other instruments designed to measure Safe Practices performance may correlate with other outcomes of interest to patients and policymakers.