Data sources Relevant studies through Medline, Embase, PsycINFO, CINAHL, and Cochrane databases.

Study selection Two reviewers independently extracted data on topics, participants, setting, study design, and outcomes.

Data synthesis Of 1104 articles identified, 72 were included in the study. Twelve reports representing nine studies (including two randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaborative intervention on care processes or outcomes of care. Systematic review of these nine studies showed moderate positive results. Seven studies (including one randomised controlled trial) reported an effect on some of the selected outcome measures. Two studies (including one randomised controlled trial) did not show any significant effect.

Different types of multiorganisational collaboratives exist, the purpose of which are to improve care.^{1 2 3} The term quality improvement collaborative seems to be used for different multifaceted packages that focus on accelerating better outcomes.⁴ Quality improvement collaboratives are used in different clinical areas and organisational contexts and have been adopted by numerous large and small healthcare systems and individual clinics. These initiatives represent substantial investments of time, effort, and funding in the delivery of health care, although estimates of the total investment and applications of the collaborative are not available.⁵ The strength of the quality improvement collaborative seems to be the relatively efficient use of experts and peers and the exchange of best practices to facilitate and guide improvement.

The earliest well documented activities of quality improvement collaboratives are those of the Northern New England Cardiovascular Disease Study Group, established in 1986, and the Vermont Oxford Network, established in 1988. Another well known approach is the Breakthrough Series developed by the Institute of Healthcare Improvement in 1995. Many of the present approaches of the quality improvement collaborative are based on those of the Breakthrough Series.

A recent non-systematic review⁶ concluded that the collaborative methodology has important potential to improve outcomes for patients and to facilitate sustainability of quality improvement. Unfortunately this introduction to the collaborative methodology neither considers whether the evaluation of effectiveness is based on a controlled or an uncontrolled study design nor makes clear on what types of quality improvement collaborative the conclusions are based.

Of the 72 papers included in our study, 60 (83%) reports^w1-w60 used an uncontrolled study design. Fifty (83%) of these 60 reports were based on the Breakthrough Series. Several studies contained elements that went beyond a simple quality improvement collaborative. The quality improvement collaboratives in these reports were embedded in ongoing quality initiatives or used comprehensive systems for measuring performance. Of the 60 reports of uncontrolled studies 37 (almost 62%) were based on self report measures of the participating teams or sites and 14 (23%) of the studies were single case reports describing changes in an individual facility or team (a detailed table listing the uncontrolled studies is available from the authors). The study designs of the uncontrolled reports relied almost entirely on post measurement, used before and after studies without being able to account for secular trends, made use of self report measures rather than reviews of medical records, included only anecdotal information, or selected samples from self selected sites. Conclusions on effectiveness could not be drawn from these reports owing to the lack of adequate reporting procedures on data collection, analysis, and objective evaluations.

Eleven reports (eight studies)^w61-w71 used a “comparison group” study design. One study^w72 used an interrupted time series design (table 1​1).). Most studies were published in 2004 and 2005. The characteristics of the collaborative strategy used in these nine studies varied. Seven studies^{w61- w68 w72} were explicitly based on the Breakthrough Series. Four of these combined the Breakthrough Series with elements of the chronic care model (www.improvingchroniccare.org).w62 ^w64-w68 Two studies^w69-w71 were based on the Vermont Oxford Network. Box 3 gives an overview of the different collaborative strategies. Table 1​1 shows the methodological quality of the nine studies. Most studies had important flaws. Limitations included possible differences in baseline measurement, limited data on characteristics of control sites, no specification of blinded assessment, and possible contamination.

Twelve reports representing nine studies (including two recent randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaboratives intervention on processes of care or outcomes of care. The studies were based on different collaborative strategies. Seven studies evaluated the Breakthrough Series, four of these were studies on the Breakthrough Series combined with chronic care model and two were based on the Vermont Oxford Network method. A systematic review of the studies produced moderate positive results. Seven studies (including one randomised controlled trial) showed at least a positive effect of a specific selection of processes of care studied. Two studies (including one randomised controlled trial) did not show any significant effect.

As a result of flaws in the methodological quality of the studies and the heterogeneity of the intervention itself, there is no certainty that the quality improvement collaborative was responsible for an effect. Six studies reported possible differences in baseline measurement. One of the controlled studies was a Breakthrough Series embedded in a seven year quality improvement programme. Four of the studies contained elements of the chronic care model in the intervention. Two of the controlled studies were based on the Vermont Oxford Network. This type of quality improvement collaborative differs from the Breakthrough Series in that it is long term: efforts are led and supported by ongoing data collection of individual member organisations and an ongoing infrastructure of communication and meetings exists that goes beyond a particular limited time frame of a quality improvement collaborative initiative. We were unable to disentangle the different components of an intervention or to assess interactions between longitudinal activities for quality improvement or elements of the chronic care model and collaborative components.

Fifty three (88%) of the 60 uncontrolled reports highlighted specific improvements in patient care and organisational performance that resulted from participating in a quality improvement collaborative. Several reports showed dramatic improvements of 30% to 80%. Almost all of the uncontrolled reports, however, had design limitations, were methodologically weak, and were probably biased in favour of positive findings in successful teams.

The evidence of the impact of quality improvement collaboratives is positive but limited. The apparent inconsistency between the widespread belief in and use of quality improvement collaboratives and the available evidence heightens the importance of a deeper understanding of the relative strength of this intervention. Quality improvement collaboratives are, by their nature, complex and applied in many different ways. Considering that quality improvement collaboratives seem to play a key part in current strategies focused on accelerating improvement, represent substantial investments of time and funding, but may have only modest effects on outcomes at best, then further knowledge of the effectiveness of the basic components, cost effectiveness, variability within collaboratives, and success factors is crucial for determining their value. What mechanisms are responsible for the results and their variations: for example, does effectiveness depend on the topic chosen and are there specific components, supportive contextual factors, or site characteristics that enhance the effectiveness of quality improvement collaboratives? It is possible that a quality improvement collaborative works for some organisations but not for others because of inherent differences in the history and culture of organisations. The data collected in the included studies did not provide the information needed to understand and explain the findings. To understand how and why quality improvement collaboratives work it is necessary to look into the “black box” of the intervention and to study the determinants of success or failure. A detailed formative evaluation of the projects might provide additional insight into these problems. The studies needed balance between uncontrolled process oriented reports and rigorously controlled designs,^{10 11 12} and a sound integration of process and effect data is needed to understand in more detail why some quality improvement collaboratives and some organisations participating in such a collaborative are successful while others fail to change practice. We look forward to studies adding to this body of knowledge.^{13 14 15}

Limitations should be considered in interpreting the results of this review. Firstly, as in any systematic review we may have missed relevant studies. We searched multiple databases, however, and checked our search with free text words with a strategy that included MeSH terms (box 1) based on key words in the relevant studies. These searches did not add new studies. Secondly, our search was limited to quality improvement collaboratives involving the five essential features described in our inclusion criteria and to English language journals. This might have introduced bias if the effectiveness described in these studies differed systematically from those involving other features and appearing in other languages. Thirdly, the key components of some quality improvement collaboratives could have been misclassified, although our abstraction process showed good inter-rater reliability.

Despite these limitations, this review shows that the evidence underlying quality improvement collaboratives is positive but still limited and that the effects cannot be predicted with great certainty.