Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Surgery. Author manuscript; available in PMC 2013 May 1.
Published in final edited form as:
PMCID: PMC3414538

Variations in Surgical Outcomes Associated with Hospital Compliance with Safety Practices

Benjamin S. Brooke, M.D., Ph.D.,a Francesca Dominici, Ph.D,e Peter J. Pronovost, M.D., Ph.D.,c,d Martin A. Makary, M.D., M.P.H., F.A.C.S.,b,d Eric Schneider, Ph.D.,b and Timothy M. Pawlik, M.D., M.P.H., F.A.C.S.b



The Leapfrog Group aims to improve patient safety by promoting hospital compliance with National Quality Forum (NQF) safe practices. It is unknown, however, whether implementation of these safety practices improve outcomes following high-risk operations.


We conducted a cross-sectional analysis of 658 nationwide hospitals that responded to the 2005 Leapfrog Group Hospital Quality & Safety survey. A total of 79,462 patients were identified from Medicare claims data who underwent a pancreatectomy, hepatectomy, esophagectomy, open aortic aneurysm repair, colectomy or gastrectomy procedure from 2004 through 2006. Random-effects logistic regression models were used to estimate the association between hospital compliance with NQF safe practices and risk-adjusted odds of complications, failure rate to rescue, and mortality after adjusting for patient and hospital level confounders.


Of the 658 hospitals that responded to surveys, 41% had fully implemented NQF safe practices and 59% reported partial compliance with these standards. Compared to hospitals with partial NQF compliance, we found significant evidence that hospitals with full compliance had an increased likelihood of diagnosing a complication following any of the six high-risk operations (OR: 1.13; 95%CI: 1.03–1.25), but had a decreased likelihood of failure to rescue (OR: 0.82; 95%CI: 0.71–0.96), and a decreased odds of mortality (OR: 0.80; 95%CI: 0.71–0.91).


Despite having a higher rate of postoperative complications, hospitals fully complying with safe practices were associated with lower failure to rescue and reduced mortality following high-risk operations. These results highlight the importance of having hospitals systems in place to promote safety and manage postoperative complications.


Patients and healthcare payers over the past several decades have sought means to identify high quality hospitals for surgical procedures. The morbidity and high cost of surgical complications have put a premium on determining which hospitals deliver the highest quality of surgical care. The American College of Surgeons (ACS), the Department of Health and Human Services (HHS) and the Centers for Medicare and Medicaid Services (CMS) have all sponsored several large hospital quality initiatives designed to prevent postoperative complications.(1, 2) Despite this broad focus on improving patient safety and reducing complications, there is still considerable debate about what hospital measures actually improve surgical outcomes.

The National Quality Forum (NQF) and the Leapfrog Group are two prominent non-profit organizations formed over the past decade that have combined efforts to promote nationwide improvements in hospital quality and safety.(3, 4) They have identified and endorsed evidence-based interventions known as “safe practices”, representing a comprehensive set of hospital process measures and systematic practices. Safe practices are designed to be implemented across hospital systems as a means to recognize and prevent medical errors or complications before they lead to fatalities in all types of hospitalized patients. Hospital compliance with NQF safe practices is measured by annual Leapfrog surveys, and responding hospitals are given a safe practice score that is published online.(4)

We sought to determine whether hospital compliance with Leapfrog sponsored NQF safe practices was associated with improvements in outcomes among patients that underwent high-risk operations. Patients that undergo high-risk operations are at substantial risk of experiencing a preventable complication during their hospitalization and thus may benefit the most from standardization of hospital safety practices. To address this question, we used nationwide data from Leapfrog surveys and Medicare data to evaluate whether hospital compliance with NQF safe practices is associated with in-hospital complications, failure to rescue from complications, and all cause 30-day mortality following six high-risk procedures.


Leapfrog-NQF Safe Practice Compliance

Data regarding compliance with the NQF safe practices were obtained from the Leapfrog Group Hospital Quality and Safety Surveys that were sent to 1,960 nationwide hospitals in 2005. Both urban and rural hospitals in 41 states were targeted. Leapfrog surveys were completed on-line and certified by the CEOs or senior administrators in 978 hospitals (50% response rate). The collected data represented self-reported information regarding hospital demographics and annual compliance with each of the Leapfrog Group hospital quality and safety standards for the preceding 12–24 month period. The survey included specific questions that were used to ascertain adherence with 27 hospital measures (aka “Safe Practices”) being promoted by the National Quality Forum in 2005. Individual safe practices are listed in Table 1. Compliance with each safe practice was given a weighted value and hospitals received a safe practice score by Leapfrog that was used to rank hospitals into one of four groups: “Fully implemented”; “Good progress”; “Good early stage effort”; and “Willing to report publicly”.(5) Hospitals were stratified into two groups based on their Leapfrog NQF Safe Practice Score ranking, defined as hospitals meeting full NQF compliance (i.e. hospitals graded as “Fully implemented”) and hospitals with partial NQF compliance (i.e. hospitals graded as having “Good progress”, “Good early stage effort”, and “Willing to report publicly”). Hospitals that returned Leapfrog surveys but did not disclose compliance with NQF safe practices were excluded from analysis.

Table 1
National Quality Forum Safe Practices Used to Calculate Leapfrog Safe Practice Score, 2005.

Study Population

To determine whether full NQF compliance is associated with better surgical outcomes compared with partial NQF compliance we assessed the outcomes of patients who underwent one of six high-risk elective surgical procedures in hospitals that responded to the Leapfrog survey in 2005: esophagectomy, pancreatectomy, hepatectomy, colectomy, gastrectomy, and open abdominal aortic aneurysm (AAA) repair. These operations were selected because each can be associated with significant likelihood for perioperative complications and mortality. Data on patients who underwent any of these elective surgical procedures at U.S. hospitals from 2004 through 2006 were obtained from the CMS Medicare Provider Analysis and Review (MedPAR) nationwide database. Patients who underwent an urgent or emergent operation were excluded from analysis. Patient data was then linked by CMS hospital identification number to the hospital database containing Leapfrog Group survey results. International Classification of Diseases, 9th Revision (ICD-9) procedure codes were used to identify procedures from the CMS dataset as previously described.(6) The Johns Hopkins University School of Medicine institutional review board approved a protocol for this study.

Study Design

We conducted a cross-sectional study and estimated odds of surgical outcomes in high risk operations associated with compliance to NQF safe practices adjusted for hospital characteristics and patient characteristics. Hospital level characteristics included into the analysis were: number of total annual admissions, number of floor admissions, number of ICU admissions, number of staffed floor beds, number of licensed floor beds, number of staffed ICU beds, number of licensed ICU beds, urban vs. rural status, and hospital membership status in health organizations. Patient level covariates included into the analysis were: age, sex, race, ethnicity, and severity of patients’ comorbid conditions as categorized using the Charlson-Deyo measurement method.(7)

Main outcome measures were risk-adjusted mortality, failure to rescue, and postoperative complications. Mortality was defined as death within 30 days of the operation or before hospital discharge, whichever came first. We identified specific complications that would be most likely to vary in surgical patients in response to hospital compliance with the 27 NQF safe practices. These complications and their ICD-9 diagnostic codes included: development of pressure ulcers (707.00–707.07, 707.20–707.25); deep venous thrombosis (415.1, 451.11, 451.19, 451.2, 451.81, 453.40–453.42, 453.8, 453.89, 453.9); pulmonary embolism (415.1); aspiration (507.0, 507.1, 507.8); central line infection (999.31); malnutrition (261.0, 262.0, 263.0, 263.1, 263.8, 263.9); postoperative infection (998.5, 998.51, 998.59, 998.83); postoperative cardiac complications (997.1). Coexisting conditions were determined from secondary diagnostic codes in the MedPAR dataset.(8) Failure to rescue was defined as death in a patient with one or more of these defined complications as previously described.(9, 10)

Statistical Analysis

Analyses of hospital and patient variables as well as outcome variables were performed using chi-square tests for categorical variables and analysis of variance (ANOVA) for continuous variables that were normally distributed. The Wilcoxon signed-rank test was used to compare non-normally distributed data. We constructed a hierarchical logistic regression model with a random intercept for each hospital to estimate the effect of NQF safe practice adherence on in-hospital mortality and postoperative complications while adjusting for patient-level and hospital-level variables. These models accounted for clustering of patient-level outcomes within hospitals. Potential interactions between variables were also explored using multivariate analysis. P values less than 0.05 (two-sided) were considered to be significant for all statistical tests and models, and the Bonferroni correction was used to control for multiple comparisons. Stata 11.0 statistical software (Stata, College Station, TX) was used for all analyses.


Of the 978 nationwide hospitals that responded to the 2005 Leapfrog survey, 658 (67%) hospitals routinely performed at least one of six defined high-risk operations and provided information on their compliance with each of the 27 NQF safe practices (Table 1). Of these 658 hospitals, 273 (41%) hospitals were fully compliant with NQF safe practices, whereas 385 (59%) hospitals were partially compliant with NQF safe practices. The characteristics of these hospitals are shown in Table 2. Hospitals with full NQF compliance were larger based on admission volume, number of beds, procedure volumes for each of the six operations, and were more likely to be members of health system organizations and be located in urban areas as compared with hospitals with partial NQF compliance.

Table 2
Characteristics of Hospitals (N=658) that Provided Information on Leapfrog Surveys Regarding Compliance with NQF Safe Practices.

Within the 658 hospitals, 79,462 fee-for-service Medicare beneficiaries aged 65–91 underwent one of six high-risk procedures on an elective basis during the study period. The distribution of operations that patients underwent in these hospitals included 5,102 pancreas resections, 3,332 esophageal resections, 2,748 liver resections, 16,732 open AAA repairs, 51,052 colon resections, and 3,919 gastric resections. The characteristics of patients who underwent these six procedures are summarized in Table 3. Patient characteristics did not differ significantly based on hospital compliance with NQF safe practices, with the exception that the proportion of black patients varied with the type of procedure.

Table 3
Characteristics of Patients Undergoing any of the Six High-Risk Procedures, Stratified by Hospital Compliance with NQF Safe Practices

The unadjusted incidence of any complication (complication rate) diagnosed among patients undergoing any high-risk procedure was found to be significantly higher in hospitals with full compliance (12.4% vs. 10.9%; P<0.05) as compared with hospitals with partial compliance (Figure 1). When stratified by type of high risk procedure, the unadjusted complication rate was found to be higher in hospitals with full vs. partial compliance with the exception of patients undergoing gastrectomy, although these comparisons were not statistically significant for any individual type of operation (Figure 1). After adjusting for patient and hospital-level covariates in the logistic regression analysis, the likelihood of any postoperative complication was significantly higher in hospitals with full NQF compliance (OR: 1.13; 95%CI: 1.03–1.25) as compared with hospitals with partial compliance (Table 4).

Figure 1
Unadjusted Rate of All Complications (A) Failure to Rescue (B) and 30-day Mortality (C) for Patients Undergoing Six High Risk Operations According to Hospital Compliance with National Quality Forum Safe Practices, 2004–2006
Table 4
Risk-adjusted* Odds of Diagnosis of Complications, Failure to Rescue, and 30-day Mortality Following Six High-Risk Procedures in Hospitals with Full NQF Compliance Compared to Hospitals with Partial NQF Compliance, 2004–2006.

Hospital compliance with individual NQF safe practices did not lead to lower complication rates among the specific events they were designed to prevent(Table 5). In fact, hospitals with full compliance were significantly more likely than hospitals with partial compliance in risk-adjusted models to diagnose a deep venous thrombosis (OR: 1.25; 95%CI: 1.05–1.49), postoperative surgical site infection (OR: 1.17; 95%CI: 1.01–1.34), and postoperative cardiac event (OR: 1.15; 95%CI: 1.00–1.34), however other individual complications did not reach statistical significance.

Table 5
Mean Rate of Postoperative Complications in Hospitals Stratified By Compliance with Individual NQF Safe Practices, 2004–2006.

Despite a higher rate of complications in patients following high-risk procedures, we found that hospitals with full compliance had a significantly lower unadjusted rate of failure to rescue from any complication (7.8% vs. 9.4%; P<0.05) when compared with hospitals with partial compliance (Figure 1). In analyses stratified by type of operation, an unadjusted lower rate of failure to rescue was observed in hospitals with full compliance following all operations with the exception of patients undergoing hepatectomy (Figure 1). When analyses were stratified by type of complication, surgical site infections were the only complication with a significant lower failure to rescue in hospitals with full vs. partial compliance (OR: 0.66; 95% CI: 0.49–0.89). In the regression models that controlled for patient and hospital covariates, hospitals with full compliance were significantly less likely to have a mortality associated with any complication (OR: 0.82; 95%CI: 0.71–0.96) as compared with hospitals with partial compliance (Table 4).

For patients undergoing any of the high-risk procedures, unadjusted mortality rates were significantly lower among hospitals with full NQF compliance (2.5% vs. 3.1%; P<0.05) as compared with hospitals with partial compliance. A lower unadjusted mortality rate was associated with full compliance in any of the six high-risk procedures (Figure 1). After adjusting for patient and hospital level covariates in regression models, hospitals with full compliance had significantly lower odds of risk adjusted mortality following any of the procedures (OR: 0.80; 95%CI: 0.71–0.91) when compared with hospitals with partial compliance. The mortality benefit associated with full compliance was significant for every procedure with the exception of hepatectomy and open AAA repair, which trended towards but did not reach statistical significance (Table 4).


Hospital compliance with NQF safe practices has been heavily promoted in Leapfrog surveys over the past decade as a means to prevent medical errors or adverse events in all clinical settings.(3, 5) In the present study, we found evidence that hospitals fully implementing 27 safe practices benefited from reduced risk-adjusted mortality among patients undergoing six major operations. Moreover, our data suggests that the degree by which hospitals complied with these heterogeneous safety practices may be associated with the diagnosis and management of in-hospital complications following surgery. Patients admitted to hospitals with full safe practice compliance were more likely to be diagnosed with postoperative complications, but were less likely to die once a complication occurred.

The results of this study provide some new insights into the mechanisms by which hospital compliance with safety and quality measures leads to improvements in patient outcomes. NQF safe practices are intended not only to prevent complications, but also to develop a “culture of safety” within compliant hospitals that includes setting up the infrastructure for surveillance and management of preventable complications. The failure to manage in-hospital complications in a timely and effective manner, also known as failure to rescue, has been shown to explain a large degree of the variability in mortality following high-risk surgical procedures.(9, 10) Consistent with these studies, we showed that hospitals that fully implemented NQF safe practices were more likely to diagnose complications in surgical patients, yet had a lower rate of failure to rescue and had a lower in-hospital mortality versus hospitals with partial safe practice compliance.

Previous studies have failed to establish an association between hospital complication rates and postoperative mortality when adjusted for patient characteristics, fueling debate as to whether these outcomes are correlated at the hospital level.(9, 11, 12) However, in contrast to our analysis, prior studies have not controlled for the presence of standardized hospital safety programs set up to identify and treat complications. Hospitals that reported full NQF safe practice compliance were expected to have surveillance programs for detecting postoperative complications such as surgical site infections, adverse cardiac events, and thromboembolic events, which likely increased detection of these complications in our study (Table 4). Data from other studies have shown that implementation of hospital surveillance programs significantly influence the detection of complications.(1315) On balance, however, it is possible that the NQF safe practices were simply not effective in preventing in-hospital complications and patients achieved lower mortality from other unmeasured aspects of care related to safety. These findings underscore the limitation of using complication rates as valid measures to define hospital quality in observational studies.

It is difficult to evaluate the impact of hospital safety interventions on patient outcomes as complications may be rare, “at risk” patient populations may be hard to define, and hospital systems for surveying and diagnosing complications can vary widely.(16, 17) This may explain in part why several recent studies have failed to identify consistent improvements in surgical outcomes associated with hospital compliance with process measures as part of the Surgical Care Improvement Project and the ACS National Surgical Quality Improvement Program (NSQIP). The results of our study suggest that measuring hospital adherence to only a few selected process measures (i.e. partial compliance) may not be sufficient to differentiate hospital quality of surgical care and outcomes. We previously showed that hospital compliance with a single process measure was not an adequate surrogate measure for defining hospital quality amongst patients undergoing six other high-risk operations.(6) In a study that evaluated the association between hospital NQF safe practice scores with outcomes that occurred during medical admissions over a single year, the authors found no correlation between safe practice compliance and risk-adjusted mortality.(20) The wide variance in mortality and complication rates seen in diverse populations of medical patients, however, may have contributed to these negative findings. Rather, we chose to focus on a specific high risk surgical patient populations. In turn, our data suggest that the effectiveness of hospital safety initiatives may depend on the cumulative effect of many different coordinated care systems being evaluated among those patients at highest risk for adverse outcomes.

Our study has several important limitations. To begin with, it is cross-sectional study and is not designed to evaluate the temporal effect of NQF safe practice compliance in participating hospitals. While we controlled for differences in hospital size and procedure volume in our models, it is possible that the relationship between safe practice compliance and surgical outcomes was confounded by patient or hospital variables that were not measured. Hospitals meeting full compliance with NQF standards may be systematically different from less compliant hospitals in other ways beyond safety practices, including how they code or record complications. We do not know whether hospitals meeting full compliance with NQF standards were more likely to code complications or whether patients were more likely to suffer complications. In addition, administrative datasets such as MedPAR are not very sensitive for identifying complications. Another problem with determining the effect of hospital compliance with safety measures is that risk-adjustment models have a difficult time accounting for preventable versus inevitable complications and deaths.(16) While risk-adjusted models that estimate morbidity and mortality may suffer from measurement bias, this effect would likely bias our results to the null hypothesis of no difference. Finally, a hospital’s level of compliance with NQF safe practices is based on self-reported data and the Leapfrog survey only had a 50% response rate. To limit reporting bias, hospitals that did not submit NQF compliance data were excluded from analysis.

In conclusion, our study suggests that hospitals taking steps to implement the comprehensive set of safe practices endorsed by the Leapfrog group and NQF may benefit from improvements in managing complications among higher-risk surgical patients. These findings highlight the importance of having hospital systems in place to identify and treat postoperative complications. It is clear that more attention needs to be focused on understanding the relationship between a hospital’s ability to improve patient outcomes through safety practices.


We would like to acknowledge the Leapfrog Group for providing survey data to complete this study. Dr. Dominici is supported by EPA R83622 and NIH/NCI P01 CA134294 from the National Cancer Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute.


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.


1. Centers for Medicare & Medicaid Services. [Accessed October 18, 2010];Quality Initiatives - General Information. 2010 Available at:
2. [Accessed: September 23, 2009];Surgical Care Improvement Project Core Measure Set. Available at:
3. National Quality Forum. [Accessed October 20, 2010];NQF in the Quality Landscape. Available at:
4. [Accessed: September 25, 2010];The Leapfrog Group for Patient Safety. Available at: Available. Accessed January 22,
5. The Leapfrog Group. [Accessed September 22, 2010];The Leapfrog Safe Practices Score Leap. Available at:
6. Brooke BS, Meguid RA, Makary MA, et al. Improving surgical outcomes through adoption of evidence-based process measures: intervention specific or associated with overall hospital quality? Surgery. 2010;147:481–490. [PubMed]
7. Li B, Evans D, Faris P, et al. Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases. BMC health services research. 2008;8:12. [PMC free article] [PubMed]
8. Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Medical care. 2004;42:355–360. [PubMed]
9. Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Annals of surgery. 2009;250:1029–1034. [PubMed]
10. Ghaferi AA, Birkmeyer JD, Dimick JB. Variation in hospital mortality associated with inpatient surgery. The New England journal of medicine. 2009;361:1368–1375. [PubMed]
11. Silber JH, Rosenbaum PR. A spurious correlation between hospital mortality and complication rates: the importance of severity adjustment. Medical care. 1997;35:OS77–OS92. [PubMed]
12. Silber JH, Rosenbaum PR, Williams SV, et al. The relationship between choice of outcome measure and hospital rank in general surgical procedures: implications for quality assessment. Int J Qual Health Care. 1997;9:193–200. [PubMed]
13. Pierce CA, Haut ER, Kardooni S, et al. Surveillance bias and deep vein thrombosis in the national trauma data bank: the more we look, the more we find. The Journal of trauma. 2008;64:932–936. discussion 936–937. [PubMed]
14. Newman-Toker DE, Pronovost PJ. Diagnostic errors--the next frontier for patient safety. JAMA. 2009;301:1060–1062. [PubMed]
15. Haut ER, Noll K, Efron DT, et al. Can increased incidence of deep vein thrombosis (DVT) be used as a marker of quality of care in the absence of standardized screening? The potential effect of surveillance bias on reported DVT rates after trauma. The Journal of trauma. 2007;63:1132–1135. discussion 1135–1137. [PubMed]
16. Pronovost PJ, Colantuoni E. Measuring preventable harm: helping science keep pace with policy. JAMA. 2009;301:1273–1275. [PubMed]
17. Pronovost PJ, Goeschel CA, Marsteller JA, et al. Framework for patient safety research and improvement. Circulation. 2009;119:330–337. [PubMed]
18. Nicholas LH, Osborne NH, Birkmeyer JD, et al. Hospital process compliance and surgical outcomes in medicare beneficiaries. Arch Surg. 2010;145:999–1004. [PMC free article] [PubMed]
19. Stulberg JJ, Delaney CP, Neuhauser DV, et al. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA. 2010;303:2479–2485. [PubMed]
20. Kernisan LP, Lee SJ, Boscardin WJ, et al. Association between hospital-reported Leapfrog Safe Practices Scores and inpatient mortality. JAMA. 2009;301:1341–1348. [PMC free article] [PubMed]