Worldwide, organisations are adopting the approach of the quality improvement collaborative in different settings. The evidence underlying the strategy is positive but limited and the effects cannot be predicted with certainty.
Twelve reports representing nine studies (including two recent randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaboratives intervention on processes of care or outcomes of care. The studies were based on different collaborative strategies. Seven studies evaluated the Breakthrough Series, four of these were studies on the Breakthrough Series combined with chronic care model and two were based on the Vermont Oxford Network method. A systematic review of the studies produced moderate positive results. Seven studies (including one randomised controlled trial) showed at least a positive effect of a specific selection of processes of care studied. Two studies (including one randomised controlled trial) did not show any significant effect.
As a result of flaws in the methodological quality of the studies and the heterogeneity of the intervention itself, there is no certainty that the quality improvement collaborative was responsible for an effect. Six studies reported possible differences in baseline measurement. One of the controlled studies was a Breakthrough Series embedded in a seven year quality improvement programme. Four of the studies contained elements of the chronic care model in the intervention. Two of the controlled studies were based on the Vermont Oxford Network. This type of quality improvement collaborative differs from the Breakthrough Series in that it is long term: efforts are led and supported by ongoing data collection of individual member organisations and an ongoing infrastructure of communication and meetings exists that goes beyond a particular limited time frame of a quality improvement collaborative initiative. We were unable to disentangle the different components of an intervention or to assess interactions between longitudinal activities for quality improvement or elements of the chronic care model and collaborative components.
Fifty three (88%) of the 60 uncontrolled reports highlighted specific improvements in patient care and organisational performance that resulted from participating in a quality improvement collaborative. Several reports showed dramatic improvements of 30% to 80%. Almost all of the uncontrolled reports, however, had design limitations, were methodologically weak, and were probably biased in favour of positive findings in successful teams.
The evidence of the impact of quality improvement collaboratives is positive but limited. The apparent inconsistency between the widespread belief in and use of quality improvement collaboratives and the available evidence heightens the importance of a deeper understanding of the relative strength of this intervention. Quality improvement collaboratives are, by their nature, complex and applied in many different ways. Considering that quality improvement collaboratives seem to play a key part in current strategies focused on accelerating improvement, represent substantial investments of time and funding, but may have only modest effects on outcomes at best, then further knowledge of the effectiveness of the basic components, cost effectiveness, variability within collaboratives, and success factors is crucial for determining their value. What mechanisms are responsible for the results and their variations: for example, does effectiveness depend on the topic chosen and are there specific components, supportive contextual factors, or site characteristics that enhance the effectiveness of quality improvement collaboratives? It is possible that a quality improvement collaborative works for some organisations but not for others because of inherent differences in the history and culture of organisations. The data collected in the included studies did not provide the information needed to understand and explain the findings. To understand how and why quality improvement collaboratives work it is necessary to look into the “black box” of the intervention and to study the determinants of success or failure. A detailed formative evaluation of the projects might provide additional insight into these problems. The studies needed balance between uncontrolled process oriented reports and rigorously controlled designs,10 11 12
and a sound integration of process and effect data is needed to understand in more detail why some quality improvement collaboratives and some organisations participating in such a collaborative are successful while others fail to change practice. We look forward to studies adding to this body of knowledge.13 14 15
Limitations should be considered in interpreting the results of this review. Firstly, as in any systematic review we may have missed relevant studies. We searched multiple databases, however, and checked our search with free text words with a strategy that included MeSH terms (box 1) based on key words in the relevant studies. These searches did not add new studies. Secondly, our search was limited to quality improvement collaboratives involving the five essential features described in our inclusion criteria and to English language journals. This might have introduced bias if the effectiveness described in these studies differed systematically from those involving other features and appearing in other languages. Thirdly, the key components of some quality improvement collaboratives could have been misclassified, although our abstraction process showed good inter-rater reliability.
Despite these limitations, this review shows that the evidence underlying quality improvement collaboratives is positive but still limited and that the effects cannot be predicted with great certainty.
What is already known on this topic
- The multi-institutional, quality improvement collaborative is widely accepted as a strategy in health care
- Its widespread acceptance and use are not, however, based on a systematic assessment of effectiveness
What this study adds
- Quality improvement collaboratives are complex and are applied in many different ways
- The evidence underlying the strategy is positive but limited and the effects cannot be predicted with certainty