|Home | About | Journals | Submit | Contact Us | Français|
Objectives To evaluate the extent to which intensive care units participating in the initial Keystone ICU project sustained reductions in rates of catheter related bloodstream infections.
Design Collaborative cohort study to implement and evaluate interventions to improve patients’ safety.
Setting Intensive care units predominantly in Michigan, USA.
Intervention Conceptual model aimed at improving clinicians’ use of five evidence based recommendations to reduce rates of catheter related bloodstream infections rates, with measurement and feedback of infection rates. During the sustainability period, intensive care unit teams were instructed to integrate this intervention into staff orientation, collect monthly data from hospital infection control staff, and report infection rates to appropriate stakeholders.
Main outcome measures Quarterly rate of catheter related bloodstream infections per 1000 catheter days during the sustainability period (19-36 months after implementation of the intervention).
Results Ninety (87%) of the original 103 intensive care units participated, reporting 1532 intensive care unit months of data and 300310 catheter days during the sustainability period. The mean and median rates of catheter related bloodstream infection decreased from 7.7 and 2.7 (interquartile range 0.6-4.8) at baseline to 1.3 and 0 (0-2.4) at 16-18 months and to 1.1 and 0 (0.0-1.2) at 34-36 months post-implementation. Multilevel regression analysis showed that incidence rate ratios decreased from 0.68 (95% confidence interval 0.53 to 0.88) at 0-3 months to 0.38 (0.26 to 0.56) at 16-18 months and 0.34 (0.24-0.48) at 34-36 months post-implementation. During the sustainability period, the mean bloodstream infection rate did not significantly change from the initial 18 month post-implementation period (−1%, 95% confidence interval −9% to 7%).
Conclusions The reduced rates of catheter related bloodstream infection achieved in the initial 18 month post-implementation period were sustained for an additional 18 months as participating intensive care units integrated the intervention into practice. Broad use of this intervention with achievement of similar results could substantially reduce the morbidity and costs associated with catheter related bloodstream infections.
Catheter related bloodstream infections cause considerable morbidity, mortality, and healthcare costs.1 2 An estimated 82000 catheter related bloodstream infections and up to 28000 attributable deaths occur in intensive care units annually,3 and each infection costs about $45000 (£28000; €31000).4 In an ongoing quality improvement project, known as the Michigan Health & Hospital Association (MHA) Keystone ICU project, these infections were substantially reduced in 103 participating intensive care units.5 The median infection rate per 1000 catheter days dropped from 2.7 at baseline to 0 within three months after implementation of an evidence based intervention. Eighteen months after implementation, infection rates had decreased by 66% from baseline. However, whether these initial results were sustained was not known.
Limited evidence assessing the sustainability of quality improvement projects beyond the initial implementation and evaluation period is available.6 7 To evaluate sustainability, a quality improvement project must have an adequate infrastructure to sustain activities beyond its initial phase. After the 18 month post-implementation evaluation period, most hospitals participating in the Keystone ICU project continued to submit data on infection rates. The objective of this study was to evaluate the extent to which intensive care units participating in the initial Keystone ICU project sustained reductions in rates of catheter related bloodstream infection. We hypothesised that the bloodstream infection rate would remain low during the sustainability period relative to baseline.
The Keystone ICU project was designed as a prospective cohort study and used a collaborative model involving the Johns Hopkins Quality and Safety Research Group, the MHA Keystone Center for Patient Safety and Quality, and participating hospitals. All hospitals in Michigan with an intensive care unit were invited to participate in the project. To participate, they had to form an intensive care unit team, agree to complete the project work, and send a signed letter of commitment from a senior hospital executive with a list of team members. At a minimum, the team had to include a senior executive, the intensive care unit director and nurse manager, and a physician and nurse in the intensive care units who could both dedicate 20% of their effort to project activities. The initial implementation and evaluation phase was funded by the Agency for Healthcare Research and Quality. During this phase, individual intensive care units implemented two cultural interventions first and then two interventions targeting patients’ safety (in any order).5 The four interventions were implemented at three month intervals. As a result, the intervention targeting catheter related bloodstream infection had a staggered timing for its implementation.
At the end of the initial funding, intensive care unit teams were instructed to sustain the bloodstream infection intervention. They were to maintain a team, integrate the intervention into staff orientation, continue collecting monthly data on catheter related bloodstream infections and catheter days in collaboration with hospital infection control staff, and continue reporting infection rates per 1000 catheter days to appropriate stakeholders. We lacked funding to formally evaluate any intensive care units’ compliance with each of these components. Informally, however, nearly all units reported that they were continuing all of these components. After the initial funding ended, intensive care units paid MHA to continue working with the Johns Hopkins team on new quality improvement interventions not related to catheter related bloodstream infections. Thirteen intensive care units declined further participation specifically because of the MHA fee. Compared with all intensive care units that continued to participate, the 13 units that dropped out of this project were more likely to be at a teaching hospital (93% v 65 %, P=0.04), but they did not vary in the median number of hospital beds (383 v 338, P=0.70) and did not have significantly different rates of catheter related bloodstream infection in the first and final quarters of the initial 18 month post-implementation period or quarterly rates of improvement over that period.
We used a conceptual model to increase use of evidence based interventions and improve safety culture. This model has been described elsewhere.8 9 10 The intervention evaluated in this study targeted clinicians’ use of five evidence based recommendations to reduce catheter related bloodstream infections from the Centers for Disease Control and Prevention (CDC).2 5 Four recommendations related to insertion of the catheter: hand washing, using full barrier precautions, cleaning the skin with chlorhexidine, and avoiding the femoral site when possible. The fifth recommendation was to remove unnecessary catheters. Strategies to increase the use of these evidence based recommendations are described elsewhere.5 11 Teams reported all catheter related bloodstream infections and number of days of use of catheters (catheter days). For the remainder of this paper we will use the term bloodstream infection to indicate catheter related bloodstream infection.
The primary outcome was the quarterly rate of bloodstream infections during the 18 month sustainability period. To test sustainability of the intervention, we analysed trends in infection rates for the 18 month initial evaluation period compared with the 18 month sustainability period. Specifically, a total of 103 intensive care units from 67 hospitals contributed data to the initial evaluation period. Some intensive care units did not contribute to all time periods, and 13 of the 103 units declined to participate in the sustainability period, leaving 90 for which we evaluated the primary outcome. All hospitals used standard definitions of catheter related bloodstream infections provided by the CDC.12 Our analysis included hospital teaching status (binary variable) and bed size (continuous variable) to show any effect on distribution of catheter days. These variables came from the American Hospital Association database.
We summarised rates of bloodstream infection as medians and interquartile ranges and as means and standard deviations. We used generalised linear latent and mixed models,13 14 with a Poisson distribution, to compare the quarterly bloodstream infection rate from baseline to the end of the initial 16-18 month evaluation period and for the subsequent 18 month sustainability period (from 19-21 months to 33-36 months post-implementation). In the models, we used robust estimation of variance, adjusted for hospital teaching status and bed number, and included two level random effects to account for nested clustering within the data13 15: intensive care units over time and hospitals within regions. We also evaluated whether the relative change in quarterly bloodstream infection rate was different during the initial evaluation period compared with the sustainability period by using a linear spline model, which allowed the relative change in quarterly infection rate to change at the end of the initial 18 month period.16
To assess the sensitivity of our results to missing data and to potential non-reporting bias, we replicated the analysis including only those 43 intensive care units that reported continuous data from baseline to the end of the sustainability period. To assess the sensitivity of our results to a few units that reported quarters with large rates of bloodstream infections (primarily driven by small numbers of catheter days), we replicated the analysis excluding the quarters of data in which catheter days were in the lowest 10% across all quarters. Finally, we replicated our models assuming an over-dispersed Poisson distribution (known as a negative binomial distribution) for the number of bloodstream infections to assess the sensitivity of our results to the Poisson assumption. All P values are two sided; we considered a P value of ≤0.05 to be statistically significant. We used Stata (version 9.1) for all analyses.
During the sustainability period, 90 (87%) of the original 103 intensive care units from 61 (91%) of the original 67 hospitals reported 1532 intensive care unit months of data and 300310 catheter days. Thirteen units dropped out because they chose not to pay the MHA’s fee. The distribution of total catheter days by hospital teaching status and bed number was consistent between the initial evaluation period and the sustainability period (table 11).
Overall, 1394 possible quarters of bloodstream infection data existed for the entire study (baseline data for 55 intensive care units, plus 13 subsequent quarters for the three month implementation period and the 36 month post-implementation period for 103 units) (table 22).). Seventy-eight quarters (roughly 5%) of data were lost when the 13 units left the collaboration, and 65 quarters (roughly 5%) of bloodstream infection rates were missing for the remaining units.
Total catheter days were consistent during each quarter in the initial evaluation period and the sustainability period. The mean and median rates of bloodstream infection decreased from 7.7 and 2.7 (interquartile range 0.6-4.8) per 1000 catheter days at baseline to 2.3 and 0 (0.0-3.0) at 0-3 months after implementation of the intervention and to 1.3 and 0 (0-2.4) at 16-18 months after implementation, and they were sustained at 1.1 and 0 (0.0-1.2) at 34-36 months post-implementation (table 22).). The multilevel Poisson regression model showed a significant decrease in bloodstream infection rates during all study periods compared with baseline rates; incidence rate ratios decreased from 0.68 (95% confidence interval 0.53 to 0.88) at 0-3 months to 0.38 (0.26 to 0.56) at 16-18 months after implementation, with a sustained improvement of 0.34 (0.24 to 0.48) at 34-36 months (table 22).
From baseline to the end of the initial 18 month evaluation period, the mean rate of bloodstream infection decreased significantly by 12% (95% confidence interval 9% to 15%) per quarter (figure(figure).). During the 19-36 month sustainability period, the quarterly bloodstream infection rate did not significantly change from the rate achieved at the end of the initial evaluation period (1% decrease, 95% confidence interval 9% decrease to 7% increase).
Forty-three intensive care units reported data continuously from baseline to the end of the sustainability period. A sensitivity analysis including only these 43 units resulted in similar findings to our main analysis including all units: an estimated 13% (9% to 16%) decrease in the mean rate of bloodstream infection per quarter during the initial evaluation period followed by a less than 1% decrease (4% decrease to 5% increase) per quarter during the sustainability period. The primary results were not substantially changed by exclusion of infection data for quarters in which the total catheter days were in the lowest 10% (<67 days). Finally, the results changed little when we used regression models assuming an over-dispersed Poisson distribution for the number of infections.
Our findings show that the markedly reduced rates of bloodstream infection achieved in the initial evaluation period of the Keystone ICU project were sustained for an additional 18 months. The median rate of infection remained at zero for the entire 18 month sustainability period, with a greater than 60% reduction in infection rates from baseline sustained at the end of the 36 month period.
Our project is one of relatively few robustly designed and evaluated large scale quality improvement projects that have shown substantial improvements. Even fewer projects have evaluated the sustainability of such results. Achieving sustainability has been a major challenge for the quality improvement field.17 18 19 The Keystone ICU project coupled a formal model to translate evidence into practice with a comprehensive patient safety intervention to improve culture, educate staff, learn from mistakes, and involve senior leaders.8 10 During interviews, intensive care unit teams noted several factors that were important in sustaining this project, including continued feedback of infection data that the team perceived as valid, improvements in safety culture that occurred as part of the overall Keystone ICU project, an unremitting belief in the preventability of bloodstream infections, involvement of senior leaders who reviewed infection data and provided teams with the resources needed, and a shared goal rather than a competition to reduce infection rates throughout the state of Michigan. Other studies describe similar methods to facilitate sustainability, such as active support from leaders and an infrastructure that supports quality improvement methods, training of staff, involving key internal and external stakeholders,6 20 21 22 23 alignment of project and organisational goals,21 multidisciplinary teams and collaborations,6 and inclusion of change management that also encourages local adaptation and rewards innovation and change.21 Quality improvement initiatives that demonstrate value are more likely to be sustained.21 24 We did not formally evaluate the factors associated with continued improvement. However, the quality improvement model used in the Keystone ICU project and our recommendations to teams at the end of the initial evaluation period to maintain a team, orient new staff, collect monthly data, and report infection rates to appropriate stakeholders target many of the recommendations described in the literature.
This research has several new limitations not reported in our evaluation of the initial evaluation period.5 Firstly, we cannot evaluate whether the results would have been sustainable if intensive care units were blinded to their infection rates, because our model to change practice included feedback of data so units could evaluate their performance.8 Secondly, we had insufficient funds to formally evaluate mechanisms that contributed to the success of this sustainability, so we cannot evaluate the relative importance during the sustainability phase of periodic meetings between participating units and the Keystone Center. Although these meetings focused on implementing new safety interventions, they may have influenced the sustainability of the bloodstream infection intervention. Also unknown was the impact of Blue Cross Blue Shield of Michigan’s quality payment incentive on a hospital’s decision to continue to participate in the Keystone ICU project. In the first year, hospitals received an incentive payment if they submitted 90% of the required bloodstream infection data. In subsequent years, the incentive payment was based on performance thresholds (statewide infection rate compared against CDC pooled mean) for continued reduction of infection rates. Thirdly, we did not evaluate the use of new technologies, such as impregnated dressings or catheters and chlorhexidine baths, on rates of infection.25 26 27 28
These findings have important public health consequences. If the multifaceted quality improvement intervention and collaborative model were implemented in all intensive care units across the United States, and the results were similar to those achieved in Michigan, substantial and persistent reductions could be made in the 82000 infections, 28000 deaths, and $2.3 billion costs attributed to these infections annually.3 Moreover, the use of this quality improvement model to reduce other complications may further improve quality and reduce costs of care.
A prominent item on the research agenda for quality improvement initiatives is identifying methods that sustain a successful project. Sustainability is often vague and may not be formally separated from the initial implementation and evaluation of the project.22 Although we do not have empirical data to support this statement, both our experience and theoretical reasons support the idea that sustainability should be examined separately from implementation. Weick and Quinn noted that organisations and teams vary in whether they are best suited for episodic or continuous change.29 Moreover, early, mid, and late implementers have differing characteristics and motivations for implementing quality improvement interventions.30 Some implementers focus efforts on short term results, whereas sustainers generally focus on long term results. Teams that are good “implementers” may not be the same as those that are good “sustainers.” More research is needed to better understand when implementation ends and sustaining begins, as well as the attributes of teams that are good implementers and good sustainers.
We thank Christine G Holzmueller for her assistance in editing the manuscript. We also thank the Michigan Health & Hospital Association (MHA) Keystone Center and all the intensive care units teams in Michigan (list of participating hospitals in web appendix) for their tremendous efforts. Their leadership and courage in this innovative effort reflects an unrelenting passion and dedication to improve quality and safety for their patients.
Contributors: All the authors were involved in preparing this manuscript and had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. PJP was the principal investigator of the project, was responsible for the overall design and supervision of the study, and wrote the initial draft of the manuscript. All other authors contributed to study design, collecting or analysing data, and redrafting the manuscript. CG was the director of the Keystone Center during the initial evaluation period, and SW was its director during the sustainability period. SW and SB managed collection and quality control of data, and PJP, EC, DJS, and DM provided data cleaning and editing, statistical analysis, and interpretation of results. SB, LL, DT, and JM were part of the Johns Hopkins team that supported the collaborative, and RH, RW, PP, and KS were members of and represented intensive care unit teams in Michigan. SC provided expertise on definitions of catheter related bloodstream infection, measuring infection rates, and interventions to reduce infections. PJP is the guarantor.
Funding: Support for this project, for the period from October 2003 to September 2005, was provided by the Agency for Healthcare Research and Quality (1UC1HS14246) and the MHA. The Agency for Healthcare Research and Quality provided financial support, and the MHA provided support for the biannual statewide meetings. The researchers were independent of the sponsors, and neither sponsor had any influence over the study design; collection, analysis, or interpretation of the data; writing of the manuscript; or decision to submit it for publication.
Competing interests: PJP and CAG received grant support from the Agency for Healthcare Research and Quality, the Robert Wood Johnson Foundation, the National Patient Safety Agency, and the World Health Organization to study and improve quality of care, including catheter related bloodstream infections. They have received lecture fees from various healthcare organisations, and CAG has also received lecture fees from government agencies to speak on quality and patient safety. SC has grant support from Cubist and Astellas, has served as a consultant for Merck, and has been on the advisory boards for Astellas, Forrest, and Cadence. JBS and JAM have grant support from the Robert Wood Johnson Foundation. PP has received lecture fees from Lilly, Merck, Edward Life Sciences, and Sage for various speaking engagements. DMN has had grant and contract support from the National Institutes of Health/National Heart Lung and Blood Institute and a clinician-scientists award from the Canadian Institutes of Health Research.
Ethical approval: The Johns Hopkins University School of Medicine Institutional Review Board reviewed and approved this research.
Data sharing: The dataset and statistical codes are available from the corresponding author.
Cite this as: BMJ 2010;340:c309