PMCCPMCCPMCC

Search tips
Search criteria 

Advanced


 
Formats
Article sections
Authors
Related links
Logo of plosmedPLoS MedicineSubmit to PLoSGet E-mail AlertsContact UsPublic Library of Science (PLoS)View this Article
 
PLoS Med. 2009 August; 6(8): e1000086.
Published online 2009 August 11. doi:  10.1371/journal.pmed.1000086
PMCID: PMC2717209
Copyright Shepperd et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Can We Systematically Review Studies That Evaluate Complex Interventions?
Sasha Shepperd,1* Simon Lewin,2,3 Sharon Straus,4 Mike Clarke,5,6 Martin P. Eccles,7 Ray Fitzpatrick,1 Geoff Wong,8* and Aziz Sheikh9,10*
1Department of Public Health, University of Oxford, Oxford, United Kingdom
2Norwegian Knowledge Centre for the Health Services, Oslo, Norway
3Health Systems Research Unit, Medical Research Council of South Africa, South Africa
4Department of Medicine, University of Toronto, Toronto, Ontario, Canada
5UK Cochrane Centre, Oxford, United Kingdom
6School of Nursing and Midwifery, Trinity College Dublin, Ireland National Institute for Health Research, Oxford, United Kingdom
7Institute of Health and Society, Newcastle University, Newcastle Upon Tyne, United Kingdom
8Research Department of Primary Care and Population Health, UCL, London, United Kingdom
9Centre for Population Health Sciences, University of Edinburgh, Edinburgh, United Kingdom
10CAPHRI, University of Maastricht, Maastricht, The Netherlands
* E-mail: Sasha.Shepperd/at/dphpc.ox.ac.uk (SS); g.wong/at/pcps.ucl.ac.uk (GW); aziz.sheikh/at/ed.ac.uk (AS)
ICMJE criteria for authorship read and met: SS SL SS MC MPE RF GW AS. Contributed to the writing of the paper: SS SL SS MC MPE RF GW AS.
Small right arrow pointing to: This article has been cited by other articles in PMC.
Background
Over the last two decades, the application of randomised controlled trials and systematic reviews has extended to the evaluation of ever more complex interventions. A range of facets of the complexity of these interventions has been identified. While most health care interventions have some degree of complexity, interventions that include a number of components, which may be independent or inter-dependent, are at the more complex end of the spectrum [1]. Examples include case management and discharge planning, both of which aim to minimise the fragmentation of health care [2],[3]. More recently, complex interventions of a more conceptual nature have been systematically reviewed, such as continuity of care or “trust” between doctors and patients [4],[5]. These present the additional challenge of defining concepts that are often poorly developed.
Compared with single evaluations of complex interventions [1],[6], less attention has been paid to the methodological issues arising from the synthesis of data from complex interventions. Important limitations include difficulties in (1) defining the intervention within the review; (2) searching for and locating relevant evidence; (3) standardising the selection of studies for a review; and (4) synthesising data. In this Viewpoint, we describe the implications of these limitations and suggest some approaches to help systematic reviewers reflect on the conceptual and analytical challenges posed by these types of review.
1. Defining the Complex Intervention
To varying degrees, complex interventions can be standardised and defined in individual prospective studies. However, the lack of an agreed definition of complex interventions that have the same aim but are described differently, or inadequately, across studies poses inherent difficulties for systematic reviewers and users of these reviews [7]. Case management illustrates this well. Despite there being no agreed typology, case management has become a generic concept across different specialities to improve inter-professional collaboration and co-ordination of care for individual patients. Case management comes in many forms, including a brokerage model, an integrated care pathway, a liaison service, and self-managed care [2]. All these variations can occur alone or in combination.
Defining the intervention is made difficult when the core purpose of an intervention, such as behavioural counselling [8], varies according to the characteristics of the participants or the trial setting, or if the intervention aims to promote an abstract concept, such as promoting trust or continuity of care [4],[5]. In the field of service delivery, interventions become complex and difficult to define if, as is often the case, (1) they are delivered across the primary-secondary care interface; (2) they are delivered in new settings; or (3) there is an added behavioural dimension and staff perform new behaviours or current behaviours in a new context [9].
Solutions to improve the description and conceptual understanding of the content of a complex intervention include (1) typologies to guide the classification of interventions and (2) supplementary evidence, such as qualitative or descriptive data [10],[11] (see Table 1).
Table 1
Table 1
Identifying the key components of a complex intervention for a systematic review.
A. Typologies
Typologies can guide the classification of common elements of interventions into homogeneous groups. The Cochrane Effective Practice and Organisation of Care Review Group (EPOC) has developed a typology for interventions aimed at professional practice, organisational, financial, and regulatory systems (see http://www.epoc.cochrane.org/). The Cochrane Consumers and Communication Review Group has developed a typology for consumers' interactions with health care professionals, services, and researchers (see http://www.latrobe.edu.au/chcp/assets/downloads/TopicList.pdf). As an example of how such typologies are used, the EPOC typology was used to classify quality improvement strategies designed for the care of people with type 2 diabetes [12]. Subsequent correspondence highlighted the risk of misclassifying interventions due to inadequate detail [13]. In another example, a typology for heart failure disease management guided the grouping of clinical service interventions (multi-disciplinary, case management, and clinic models) [14],[15].
An alternative is to develop a typology of interventions by consensus. For example, in a systematic review of occupational therapy interventions for rheumatoid arthritis, four occupational therapists identified seven types of intervention (comprehensive therapy, training of motor function, skills training, joint protection, advice on assistive devices, counselling, and provision of splints) [16].
B. Supplementary evidence: (i) Trial-related data
Contacting trialists to obtain the protocols they followed, and identifying supplementary research related to the trial, may help define interventions. A qualitative study, conducted alongside a trial of intensive case management for people with severe mental illness, investigated the active ingredients of the intervention in terms of staff roles and organisational features [17]. Team management, comprehensive assessment, and needs-led service were regarded as the key mechanisms of this intervention. In another example, trialists contributing to a systematic review of stroke units were surveyed to build a description of the active components of stroke unit care. These included comprehensive assessment, active physiological management, early mobilisation, skilled nursing care, early rehabilitation, and discharge planning involving carers [18].
B. Supplementary evidence: (ii) Qualitative data, descriptive data, and policy documents
When trial-related evidence is inadequate [19], other sources of information may be relevant [20]. A systematic review of barriers and facilitators to healthy eating among children used qualitative evidence, unrelated to the trial data, to gain a better understanding of children's perspectives [21]. The qualitative synthesis guided the categorisation of interventions, according to the degree to which they combined health advice with the promotion of eating fruit and vegetables. A synthesis of qualitative studies aided a fuller understanding of the interventions included in a systematic review of directly observed therapy (DOT) for tuberculosis, by identifying factors that improved adherence [22]. Factors included flexible delivery systems, involving patients in decisions, and social and family support systems [23]. Policy documents can be particularly informative in understanding the development of service interventions across settings, for example, interventions designed to reduce reliance on hospital beds and interventions involving school feeding programmes [24],[25].
B. Supplementary evidence: (iii) Theory
Theory may help to explain how an intervention is related to similar interventions in a particular field [26]. However, many reviews fail to locate interventions within a theoretical model. Realist synthesis, which attempts to provide an explanatory analysis of how and why complex social interventions work (or not) in particular contexts, can aid this process. For example, a realist review of school feeding programmes identified the theory and processes that promoted the success of these interventions [25]. Theory can also guide the classification of interventions; behavioural and contingency theory successfully guided the classification of interventions designed to implement change in practice [27]. The theory of planned behaviour was used to address an individual's motivation, attitude, and perceived behavioural control; whereas contingency theory was used to take into account the fit between clinical practice and environmental constraints. However, theory can only improve our understanding of how an intervention works if it is part of an integrated body of knowledge that differentiates the explanatory role of one theory from another and provides robust predictions of causal pathways. An attempt to categorise studies in a Cochrane review of tobacco cessation for young people failed due to the complexity of the interventions and the simultaneous use of several psychosocial theories [28].
2. Searching For and Identifying All Relevant Data
The lack of consistent terminology and the inconsistent use of existing terminology to describe complex interventions means that identifying potentially eligible studies can be difficult. Search strategies may be incomplete and risk introducing bias if they identify only a proportion of all possible configurations of a complex intervention. For example, “continuity of care”, a concept that is considered to contribute to high-quality care, can be delivered through numerous mechanisms (shared care, telephone follow-up, patient-held records, and case management, to name a few) [29].
Solutions include characterising the elements of an intervention through an iterative scoping exercise and searching outside the traditional health care domains to include engineering, social sciences, and management journals. Data may be unpublished and only accessed through policy documents, conference proceedings, or book chapters, but need to be obtained to minimise the effects of publication bias [30],[31]. Contacting those working in the field, retrieving references of references, and tracking citations will also increase the efficiency of finding relevant evidence.
3. Selecting Studies for Inclusion in a Review
A major threat to validity from an imprecise definition of an intervention is the non-standardised and potentially non-reproducible selection of studies for inclusion in a review. Based on the available information, considerable judgement may be required when assessing how similar any given intervention is to the intervention of interest, particularly for multi-faceted interventions and those at the boundary of the content area. For example, reaching a common understanding of patient-centred interventions was not easily achieved in a review of interventions intended to promote patient-centred care [32].
Solutions to this problem of definition include: (1) refining the definition of an intervention through an iterative process to accommodate previously unseen configurations; (2) contacting study authors for further information; (3) recording the components of an intervention during data extraction; and (4) being explicit in the review about where disagreement occurred.
4. Synthesis of Data
Complex interventions with a large number of ill-defined elements may result in a high degree of heterogeneity. Conversely, applying a narrow definition limits generalisability by losing the potential relevance gained from examining an intervention being implemented across a range of settings. A meta-analysis of DOT for tuberculosis provided in clinics, by lay health workers, or in the home provides an example of the usefulness of exploring sources of heterogeneity [22]. The authors found no important difference between DOT and self-administered treatment (risk ratio 1.02, 95% confidence interval 0.86 to 1.21; I2 64%). However, when the trials were grouped by the location of DOT there was a small beneficial effect for delivering DOT in a home setting compared with self administration (risk ratio 1.10, 95% confidence interval 1.02 to 1.18; I2 53%). This beneficial effect allows for several possible interpretations; for example, the burden upon patients of travelling to a clinic five days a week is minimised by having their therapy supervised at home by a lay health worker or a community or family member.
Solutions to the problem of data synthesis include categorising interventions by key variables and retaining these in the analysis. For example, a meta-analysis of discharge planning and post-discharge support categorised interventions by intensity, which varied from a single home visit, to increased clinic follow-up with telephone contact through to extended home care services [3]. If meta-analyses cannot be performed, similar processes can be conducted whilst performing narrative synthesis. The quality of narrative analysis and applicability of review findings have recently received more attention [33] (see Box 1).
Box 1. Presentation of Review Findings: Information To Support Assessment of the Applicability of Evidence of Effectiveness
Intervention Content [1]
  • Describe the content (the active ingredients) of the interventions
  • Describe any interventions received by the control group, including the content of “usual care”
  • Describe how the interventions were delivered and any differences in delivery across the included trials
  • Describe the contextual similarities and differences between the trials
Intervention Fidelity
  • Include details describing whether the interventions included in a review do what is intended or if they deviate from the intended shape or form during the course of implementation
  • Include an assessment of whether an intervention failed because it was poorly implemented or it was not effective
Intervention Sustainability
  • Include details on the sustainability of intervention effects over time
Roll Out/Scaling Up of the Intervention [7]
  • Report data on accessibility, the risk of adverse events, cost-effectiveness, or budget impact of interventions
  • Address the following questions regarding the applicability of the evidence to individual patients (where applicable) [37]:
    • Have biological results (e.g., sex, co-morbidities, age) that might modify the treatment response been excluded?
    • Can consumers comply with the treatment requirements?
    • Can health care providers comply with the treatment requirements?
    • Are the likely benefits worth the potential risks and costs?
  • Address the following questions regarding the applicability of the evidence in other health systems (where applicable) [38]:
    • Are there important differences or similarities in the structural elements of health systems or of health services between where the research was done and where it will be applied that might mean that an intervention could not work in the same way?
    • Are there important differences in on-the-ground realities and constraints (i.e., governance, financial, and delivery arrangements) between where the research was done and where it could be applied that might substantially alter the potential benefits of the intervention?
    • Are there likely to be important differences in the baseline conditions between where the research was done and other settings? If so, would this mean that the intervention could have different absolute effects, even if the relative effectiveness was the same?
    • Are there important differences in perspectives and influences of health system stakeholders between where the research was done and where it could be applied that might mean an intervention will not be accepted or taken up in the same way?
Conclusion
Despite the range of supplementary methods available to improve the synthesis of complex interventions, most of these methods are infrequently used. There are several reasons for this lack of use. In some cases the theory underpinning a specific complex intervention has not been assembled. However, there are usually few data reporting the characteristics of complex interventions, and what data there are tend to be of poor quality. Although simple in concept, providing an adequate description of complex interventions can be technically difficult. The need to address this is becoming urgent as interventions with multiple components evolve in response to the complex health problems faced by health services. Current criteria to improve the reporting of health research are primarily concerned with the internal validity of studies [6], and while these include criteria related to the intervention, these guidelines can still be followed without providing adequate details of the intervention (see Table 2). It is essential that methods to improve the descriptions of complex interventions are further developed and tested with the expectation that they will complement existing systematic review methodology.
Table 2
Table 2
Current guidance for reporting complex interventions and where further research is required.
I believe that complex health interventions (CHIs) can be systematically reviewed, but only if a paradigm shift occurs in the way that these interventions are conceptualised. In this Viewpoint, I discuss an alternative way of viewing CHIs that focuses on the interactions between components of a CHI and the impact of human behaviour on the outcome of the intervention. I then discuss how this different way of viewing CHIs has given rise to a different method, called “realist review”, to systematically review CHIs.
CHIs Are Non-Linear and Produce “Irregular” Outcomes
CHIs are more than just a complicated “jumble” of components that interact in a regularly predictable but linear fashion (i.e., deterministically) to produce health outcomes. Interactions between the components are not as deterministic as might be expected because CHIs are highly dependent on human behaviour [1],[40]. The “components” in CHIs are invariably made up of people (e.g., researchers) trying to get other people (e.g., study participants) to “do” or “not do” something (e.g., to stop smoking in a CHI of smoking cessation). The actions taken by the “human components” of CHIs are influenced by the context in which the intervention takes place. Taking the example of smoking cessation, the actions that smokers can take will depend on their personal circumstances, including their health status, while the actions the clinic staff can take will be guided by factors such as the trial protocol and the clinical setting.
Thus the context in which CHIs take place is a key variable, because context influences and limits the range of people's choices and actions. Such context explains why the interactions between the components of CHIs are not deterministic and why the outcomes of CHIs can vary when the CHI is repeated. There will be a myriad of different contexts, each having a slightly different influence on those involved in the intervention. Human behaviour varies under the influence of different contexts—and so the pattern of health outcomes achieved from CHIs are best described as “demi-regular”.
Shepperd and colleagues rightly point out that the systematic review of CHIs remains challenging. From the range of solutions they have proposed, the greatest progress is likely to be made by focusing on theories (Box 2) that can explain and “predict” how certain contexts influence individuals to act in certain ways to produce certain outcomes. Pawson and colleagues have already made some progress towards such explanation and “prediction” using the “realist review” method of systematic review [41].
Box 2: Definitions of Theory
Theory:
There are multiple definitions for theory, and in this Viewpoint article: “A theory is an attempt to organize the facts—some ‘proven’, some more conjectural—within a domain of inquiry into a structurally coherent system” [45].
Middle-Range Theory:
This is a theory that lies “…between the minor but necessary working hypotheses that evolve in abundance during day-to-day research and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organization and social change…
“It is intermediate to general theories of social systems which are too remote from particular classes of social behavior, organization and change to account for what is observed and to those detailed orderly descriptions of particulars that are not generalized at all. Middle-range theory involves abstractions, of course, but they are close enough to observed data to be incorporated in propositions that permit empirical testing.” [42].
Realist Review
Realist review is a systematic review method that focuses more on trying to explain as opposed to judge CHIs. As such, it seeks not so much to answer the question of “If” a CHI works, but “How”, “Why”, “In what circumstances”, “For whom”, and “To what extent” it works. The underlying premise of realist review is that the demi-regular patterns of interactions between the components (so-called “demi-regularities”) that make up CHIs with similar goals can be explained by middle-range theory (Box 2) [42]. For any similar group of CHIs (e.g., smoking cessation interventions), the myriad of contexts influencing behaviour so as to generate outcomes are not impediments to realist review, but act as the “raw materials” from which demi-regularities can be identified. Middle-range theory (or theories) are then sought to explain why these demi-regularities occur. As the review progresses iteratively, theories that “work” (i.e., best explain sets of demi-regularities) are repeatedly “tested” against the observations reported in each CHI included in the realist review [41],[43],[44].
In exploring the feasibility of systematically reviewing studies that evaluate CHIs, I believe we have come to a crossroads. Currently, the dominant systematic review methodology and paradigm is based on the Cochrane review. Without doubt, its dominance has been well earned as its usefulness in the systematic review and meta-analysis of pharmaceutical and other “simpler” medical interventions has advanced the evidence-based practice of medicine. However, efficacy studies of pharmaceutical studies are less reliant on the human agency that we see in CHIs. In other words, a well-characterised “drug” does things to people to cause outcomes in a deterministic way, and context is much less relevant. Once human agency comes into play (as can be seen in the differences in outcomes reported between pharmaceutical efficacy and effectiveness trials), then context starts to play a bigger role. By the time we come to CHIs (where outcomes are highly dependent on human agency), then the appropriateness of the current dominant systematic review method needs to be questioned.
Conclusion
In CHIs, the action of individuals under specific contexts results in outcomes. The way these outcomes result is neither deterministic nor regular, but can be explained and “predicted” by middle-range theory. Systematically reviewing CHIs is only feasible when the review method takes into account these properties, and theory-driven reviews are our best bet.
The evaluation of complex interventions is increasingly commonplace within health services research. It is therefore important and timely that attention is given to reflecting critically on how these studies can best be identified and appraised, and their findings then synthesised and interpreted. The answer to the somewhat rhetorical question of whether it is feasible to systematically review studies of complex interventions is, at one level at least, obvious. Many of the general principles of systematic review methods can and indeed should be used when undertaking systematic reviews of complex intervention studies in health care. Sasha Shepperd and colleagues offer a number of useful suggestions in this respect, including: (1) the need to use broad search techniques in an attempt to identify and include studies that may have been poorly indexed; (2) carefully defining and describing the interventions being studied; and (3) a willingness to consider narrative synthesis if studies are found to be too heterogeneous to be synthesised quantitatively.
There are, however, a number of important conceptual and practical challenges in undertaking such reviews. One key consideration that is usefully highlighted by Geoff Wong is the importance of studying contextual considerations, both in relation to understanding the broader picture in which individual studies have been conducted, and in order to make sense of how the landscape has (often irrevocably) changed as a result of the intervention being applied. Wong also helpfully discusses the central importance of considering not only what was achieved in relation to standard parameters of interest, such as effect size, but also why this may have been achieved, i.e., a clear description of the processes that the intervention has operated through in order to achieve this effect [46]. This latter consideration is particularly important in that it is these processes that are far more likely to prove generalisable than the specifics of the intervention under study [47].
Based on my experiences of conducting a number of complex intervention trials and attempts at undertaking systematic reviews of complex interventions, I am increasingly of the opinion that there are a number of additional unresolved issues that warrant more detailed reflection.
Considering first the definition and description of the intervention, it is important that researchers not only describe the intervention in detail, but also that they capture and describe how the intervention may have evolved during the course of delivering it. A somewhat extreme example of such a complex health care intervention, but which should nonetheless illustrate the point well, is the United Kingdom's National Programme for Information Technology. This programme is the largest non-military information technology-based intervention in the world. Over the course of its relatively short life course, the programme has undergone re-branding, had several rounds of leadership change and, more fundamentally, expanded its list of core deliverables [48]. Such changes are often mirrored, albeit on a smaller scale, in the health interventions that we more routinely consider as complex interventions. The key issue here is that such modifications should not be seen as compromising the fidelity of the intervention. Instead, these modifications should be recorded and described as fully as possible in order to allow readers to make sense of what modifications were considered necessary and why, and through so doing allowing readers to appreciate what was actually delivered [1].
A second important consideration relates to what is and what is not considered a complex intervention. Most interventions do have a degree of complexity—even a “simple” aspirin effectiveness trial has some complexity—because the processes through which the delivery of the intervention leads to actual patient compliance with the treatment is inevitably made up of “several interacting components” [1]. Some argue that this in effect means that the term “complex intervention” is somewhat meaningless. I would take a somewhat different view, stressing that what we must not do is take the other extreme of excluding from our frame of reference interventions that are judged so complex that they are not amenable to study through randomised controlled trial designs. Many health policy initiatives, particularly those that are national or supra-national, are inherently multi-faceted, such as national legislation prohibiting smoking in public places and the National Programme for Information Technology. Such programme-level interventions are often not easily amenable to study using quasi-experimental designs, but their evaluation remains important and such interventions should also be studied through systematic reviews.
My third point relates to the importance of elucidating the likely mechanisms through which the effects of the intervention are mediated [49]. Carefully theorised studies/interventions can help greatly in this respect, as can accompanying embedded qualitative work, particularly if it has a longitudinal dimension [50],[51]. One way forward is to assign greater weight to studies that have described mechanisms, but a more general point is the importance of searching for and including relevant theoretical and qualitative work into complex intervention systematic reviews.
Considering then the question of meta-analysis, whilst this may be appropriate in some systematic reviews of complex interventions [52], meta-analysis may also often be somewhat inappropriate. Instead, we should really be more interested in understanding the ways in which local contextual considerations may have acted as co-factors in helping shape delivery of the intervention, rather than focusing on finding any overall summary effect of the intervention. Such summary effects may tend to obscure rather than enlighten [25]. Consider, for example, our phase III and IV studies of the accessibility, acceptability, and effectiveness of a telephone-based review service for hard-to-reach people with asthma [53],[54]. These studies could only have been possible in a climate that has a well-developed information technology infrastructure, including routine use of electronic health records, values regular review of those with long-term conditions, and rewards practitioners accordingly. Paying careful attention to such crucial contextual influences allows readers to begin to assess the role of co-factors in shaping delivery of the intervention and also in assessing its likely generalisability.
The upshot of all of this is that systematic reviews of complex health interventions can and should be done, but if they are to shed more light than darkness, the systematic reviewers need explicitly to consider doing two things. First, they should search for and include relevant theoretical and qualitative work. Second, where relevant, they should include data from a broader range of experimental study designs than is currently normally the case in most Cochrane systematic reviews. Such an approach will in turn necessitate development of better search strategies to locate this non-trial literature and also the availability of techniques for the quality assessment of such studies. Theory-driven analysis, wherever possible, should also accompany the more conventional quantitative syntheses, the emphasis on the latter being down-played. If the Cochrane Collaboration can take a lead in spearheading and supporting these developments, this would represent a considerable service to this important and expanding field of evidence synthesis.
Abbreviations
CHIcomplex health intervention
DOTdirectly observed therapy
EPOCCochrane Effective Practice and Organisation of Care Review Group

Footnotes
S Shepperd is an editor for the Cochrane Effective Practice and Organisation of Care Review Group. SL is an editor for the Cochrane Consumers and Communication Review Group. MPE is an editor for the Cochrane Effective Practice and Organisation of Care Review Group. AS is PI on a number of evaluations of complex health care interventions funded by the Chief Scientist's Office of the Scottish Government, Health Technology Assessment, National Preventative Research Institute/Medical Research Council and NHS Connecting for Health Evaluation Programme. AS is, in addition, a grant‐holder on complex intervention trials funded by Asthma UK, Intel, and the Patient Safety Research Portfolio. AS is also a grant‐holder on programme grants for the development and evaluation of complex interventions in supportive and palliative care funded by the National Cancer Research Institute and the MRC Translational Medicine Methodology Trial Hub. GW has no competing interests.
S Shepperd was supported by an NIHR Evidence Synthesis Award. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of her viewpoint. The views expressed in the Viewpoint of S Sheppard and colleagues are not necessarily those of The Cochrane Collaboration. GW's NIHR Clinical Lectureship is funded by the United Kingdom's National Institute of Health Research (via University College London). The views expressed by GW in his Viewpoint are his personal views and not necessarily those of his funders or employers.
Provenance: Sasha Shepperd and colleagues' Viewpoint was initially submitted for the Guidelines and Guidance section of PLoS Medicine and externally peer reviewed. The other two Viewpoints, by Geoff Wong and Aziz Sheikh, were commissioned and not peer reviewed.
References
1. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, et al. Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655. [PMC free article] [PubMed]
2. Zwarenstein M, Stephenson B, Johnston L. Case management: Effects on professional practice and health care outcomes (Protocol). Cochrane Database Syst Rev. 2005;1:CD002797. doi: 10.1002/14651858.CD002797.
3. Phillips CO, Wright SM, Kern DE, Singa RM, Shepperd S, et al. Comprehensive discharge planning with post discharge support for older patients with congestive heart failure: A meta-analysis. JAMA. 2004;291:1358–1367. [PubMed]
4. Freeman G, Shepperd S, Robinson I, Ehrich K, Richards S. Continuity of Care: Report of a scoping exercise for the national co-ordinating centre for NHS Service Delivery and Organisation R&D (NCCSDO) London: National Institute for Health Research Service Delivery and Organisation; 2001.
5. McKinstry B, Ashcroft R, Car J, Freeman GK, Sheikh A. Interventions for improving patients' trust in doctors and groups of doctors. Cochrane Database Syst Rev. 2006;3:CD004134. doi: 10.1002/14651858.CD004134.pub2. [PubMed]
6. EQUATOR Network. Enhancing the quality and transparency of health research. 2009. Available: http://www.equator-network org/. Accessed 10 June 2009.
7. Glenton C, Underland V, Kho M, Pennick V, Oxman A. Summaries of findings, descriptions of interventions, and information about adverse effects would make reviews more informative. J Clin Epidemiol. 2006;59:770–778. [PubMed]
8. Steptoe A, Kerry S, Rink E, Hilton S. The impact of behavioral counseling on stage of change in fat intake, physical activity, and cigarette smoking in adults at increased risk of coronary heart disease. Am J Public Health. 2001;91:265–269. [PubMed]
9. Shepperd S, Doll H, Angus R, Clarke M, Iliffe S, et al. Admission avoidance hospital at home. Cochrane Database Syst Rev. 2008;4:CD007491. doi: 10.1002/14651858.CD007491. [PubMed]
10. Campbell M, Fitzpatrick R, Haines A, Kinmouth A-L, Sandercock P, et al. A framework for development and evaluation of RCTs for complex interventions to improve health. BMJ. 2000;321:694–696. [PMC free article] [PubMed]
11. Davidson KW, Goldstein M, Kaplan RM, Kaufman PG, Knatterud GL, et al. Evidence based behavioural medicine: What is it and how do we achieve it? Ann Behav Med. 2003;26:161–171. [PubMed]
12. Shojania KG, Ranji SR, McDonald KM, Grimshaw JM, Sundaram V, et al. Effects of quality improvement strategies for type 2 diabetes on glycemic control—A meta regression analysis. JAMA. 2006;296:427–440. [PubMed]
13. Shojania KG, Ranji SR, McDonald KM, Grimshaw JM, Rushakoff RJ, et al. Quality improvement strategies for type 2 diabetes [letter]. JAMA. 2006;296:2681. [PubMed]
14. Riegel B, LePetri R. Heart failure disease management models. In: Moser DK, Riegel B, editors. Improving outcome in heart failure: An interdisciplinary approach. Maryland: Aspen; 2001. pp. 267–281.
15. Taylor S, Bestall J, Cotter S, Falshaw M, Hood S, et al. Clinical service organisation for heart failure. Cochrane Database Syst Rev. 2005;4:CD002752. doi: 10.1002/14651858.CD002752.pub2. [PubMed]
16. Steultjens EMJ, Dekker J, Bouter LM, van Schaardenburg D, van Kuyk MAH, et al. Occupational therapy for rheumatoid arthritis. Cochrane Database Syst Rev. 2004;1:CD003114. doi: 10.1002/14651858.CD003114.pub2. [PubMed]
17. Weaver T, Tyrer P, Ritchie J, Renton A. Assessing the value of assertive outreach. Br J Psychiatry. 2003;183:437–445. [PubMed]
18. Langhorne P, Pollock A. with the Stroke Unit Trialists Collaboration. What are the components of effective stroke unit care? Age Ageing. 2002;31:365–371. [PubMed]
19. Lewin S, Glenton C, Oxman AD. How are qualitative methods being used alongside complex health service RCTs? A systematic review. BMJ. In press 2009
20. Noyes J, Popay J, Pearson A, Hannes K, Booth A. on behalf of the Cochrane Qualitative Research Methods Group. Qualitative research and Cochrane reviews. The Cochrane Handbook for systematic reviews of interventions. Wiley; 2008.
21. Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, et al. Integrating qualitative research with trials in systematic reviews. BMJ. 2004;328:1010–1012. [PMC free article] [PubMed]
22. Volmink J, Garner P. Directly observed therapy for treating tuberculosis. Cochrane Database Syst Rev. 2007;4:CD003343. doi: 10.1002/14651858.CD003343.pub3. [PubMed]
23. Munro SA, Lewin SA, Smith HJ, Engel ME, Fretheim A, et al. Patient adherence to tuberculosis treatment: A systematic review of qualitative research. PLoS Med. 2007;4:e238. doi: 10.1371/journal.pmed.0040238. [PMC free article] [PubMed]
24. Shepperd S, Doll H, Angus RM, Clarke M, Iliffe S, et al. Avoiding hospital admission through provision of hospital care at home: A systematic review and meta-analysis of individual patient data. CMAJ. 2009;180:175–182. doi: 10.1503/cmaj.081491. [PMC free article] [PubMed]
25. Greenhalgh T, Kristjansson E, Robinson V. Realist review to understand the efficacy of school feeding programmes. BMJ. 2009;335:858–861. [PMC free article] [PubMed]
26. Davies P, Walker A, Grimshaw J. Theories of behaviour change in studies of guideline implementation. Proc Br Psychol Soc. 2003;11:120.
27. Leeman J, Baernholdt M, Sandelowski M. Developing a theory based taxonomy of methods for implementing change in practice. J Adv Nurs. 2007;58:191–200. [PubMed]
28. Grimshaw GM, Stanton A. Tobacco cessation interventions for young people. Cochrane Database Syst Rev. 2006;4:CD003289. doi: 10.1002/14651858.CD003289.pub4. [PubMed]
29. Shepperd S, Richards S. Continuity of care—A chameleon concept. J Health Serv Res Policy. 2002;7:130–131. [PubMed]
30. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta- analyses of randomised trials of health care interventions. Cochrane Database Syst Rev. 2007;2:MR000010. doi: 10.1002/14651858.MR000010.pub3. [PubMed]
31. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009;1:MR000006. doi: 10.1002/14651858.MR000006.pub3. [PubMed]
32. Lewin SA, Skea ZC, Entwistle V, Zwarenstein M, Dick J. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Database Syst Rev. 2001;4:CD003267. doi: 10.1002/14651858.CD003267. [PubMed]
33. Arai L, Britten N, Popay J, Roberts H, Petticrew M, et al. Testing methodological developments in the conduct of narrative synthesis: A demonstration review of research on the implementation of smoke alarm interventions. Evid Policy. 2007;3:361–383.
34. Lewin S, Swart T, Volmink J. A review of health behaviour theories: How useful are these for developing interventions to promote long-term medication adherence for TB and HIV/AIDS? BMC Public Health. 2007;7:104. [PMC free article] [PubMed]
35. Dale J, Caramlau IO, Lindenmeyer A, Williams SM. Peer support telephone calls for improving health. Cochrane Database Syst Rev. 2008;4:CD006903. doi: 10.1002/14651858.CD006903.pub2. [PubMed]
36. Cunliffe AL, Gladman JRF, Husbands SL, Miller P, Dewey ME, et al. Sooner and healthier: A randomised controlled trial and interview study of an early discharge rehabilitation service for older people. Age Ageing. 2004;33:246–252. [PubMed]
37. Dans AL, Dans LF, Guyatt G. Applying results to individual patients. In: Guyatt G, Rennie D, Meade MO, Cook DJ, editors. Users' guides to the medical literature. A manual for evidence-based clinical practice. New York: McGraw Hill; 2008.
38. Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004;364:1615–1621. [PubMed]
39. Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361–366. [PubMed]
40. Anderson R. New MRC guidance on evaluating complex interventions. BMJ. 2008;337:a1937. [PubMed]
41. Pawson R. Evidence-based policy. A realist perspective. London: Sage; 2006.
42. Merton R. On theoretical sociology. Five essays, old and new. New York: The Free Press; 1967.
43. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: An introduction. ESRC Research Methods Programme. 2004. Available: http://www.ccsr.ac.uk/methods/publications/documents/RMPmethods2.pdf. Accessed 11 June 2009.
44. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review—A new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10:21–34. [PubMed]
45. Klee R. Introduction to the philosophy of science: Cutting nature at its seams. New York: Oxford University Press; 1997.
46. Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.
47. Hawe P, Shiell A, Riley T. Complex interventions: How “out of control” can a randomised controlled trial be? BMJ. 2004;328:1561–1563. [PMC free article] [PubMed]
48. Car J, Black A, Anandan C, Cresswell K, Pagliari C, et al. The impact of ehealth on the quality & safety of healthcare: A systematic overview & synthesis of the literature. 2008. Report for the NHS Connecting for Health Evaluation Programme. Available: http://www.ehealthnews.eu/images/stories/Impact_of_eHealth.pdf. Accessed 11 June 2009.
49. Pawson R. Does Megan's Law work? A theory driven systematic review. ESRC UK Centre for Evidence Based Policy and Practice. University of London; 2002.
50. Murray SA, Kendall M, Carduff E, Worth A, Harris F, et al. Serial qualitative interviews: A method with potential for understanding patients' evolving experiences and needs. BMJ. In press 2009 [PubMed]
51. Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Elliott R, et al. A protocol for the PINCER trial: A cluster randomised trial comparing the effectiveness of a pharmacist-led IT-based intervention with simple feedback in reducing rates of clinically important errors in medicines management in general practices. Trials. 2009;10:28. [PMC free article] [PubMed]
52. Beswick AD, Rees K, Dieppe P, Ayis S, Gooberman-Hill R, et al. Complex interventions to improve physical function and maintain independent living in elderly people: A systematic review and meta-analysis. Lancet. 2008;371:725–735. [PMC free article] [PubMed]
53. Pinnock H, Adlem L, Gaskin S, Harris J, Snellgrove C, et al. Accessibility, clinical effectiveness, and practice costs of providing a telephone option for routine asthma reviews: Phase IV controlled implementation study. Br J Gen Pract. 2007;57:714–722. [PMC free article] [PubMed]
54. Pinnock H, Bawden R, Proctor S, Wolfe S, Scullion J, et al. Accessibility, acceptability, and effectiveness in primary care of routine telephone review of asthma: Pragmatic, randomised controlled trial. BMJ. 2003;326:477–479. [PMC free article] [PubMed]
Articles from PLoS Medicine are provided here courtesy of
Public Library of Science