|Home | About | Journals | Submit | Contact Us | Français|
Systematic reviews have the potential to inform clinical decisions, yet little is known about the impact of interventions on increasing the use of systematic reviews in clinical decision-making.
To systematically review the evidence on the impact of interventions for seeking, appraising, and applying evidence from systematic reviews in decision-making by clinicians.
Medline, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, and LISA were searched from the earliest date available until July 2009.
Two independent reviewers selected studies for inclusion if the intervention intended to increase seeking, appraising, or applying evidence from systematic reviews by a clinician. Information about the study population, features of each intervention, methods used to measure the use of systematic reviews and those used to measure professional performance or health care outcomes, existence and use of statistical tests, study outcomes, and comparative data were extracted.
A total of 8,104 titles and abstracts were reviewed, leading to retrieval of 189 full-text articles for assessment; five of these studies met all inclusion criteria. All five studies reported on professional performance behavior; none reported on patient health outcomes. One study reported positive outcomes in improving preventive care. Three studies focused on obstetrical care, with two reporting no impact on professional practice change, and one study reporting increases in the use of prophylactic oxytocin and episiotomy. One study found no improvement in the sealant rate of newly erupted molars among dentists in Scotland.
The small number of studies available for examination indicates the difficulty in summarizing and identifying key aspects in successful strategies that encourage clinicians to use systematic reviews in decision-making. Other concerns lay in selective reporting and lack of blinding during data collection.
The limited empirical data render the strength of evidence weak for the effectiveness and types of interventions that encourage clinicians to use systematic reviews in clinical decision making.
Ideally, clinical decisions would be made using the best available evidence. However, simple dissemination of the results of research does not guarantee implementation.1 It remains a universal challenge to implement research findings that show benefit or reduce harm.2-5
Despite the stated purpose of facilitating decision-making and the concerted effort to enhance and clarify the methods of systematic reviews,6 studies suggest their use in clinical settings is not widespread.7,8 In a systematic review of information seeking practices among clinicians, Dawes and Sampson9 found that textbooks were the most frequent source of information, followed by advice from colleagues. Surveys reported in 2001 and 2002 that physicians had limited use or awareness of the Cochrane Library and systematic reviews.10-12 In the United Kingdom, it was found that only 10% regularly referred to the Cochrane Library, and 52% had never heard of it.10,12 Similarly, in New Zealand only 15% had ever used it.11 In examining nurses’ use of knowledge sources in clinical practice, it was found they preferred to use knowledge gained through personal experience and interactions with co-workers rather than journal articles of any type.13 Concurrently, examples of misuse of therapies indicate systematic reviews are not regularly used to inform decision-making in clinical settings.3 To our knowledge, no systematic reviews of studies exist that examine interventions that help health practitioners use systematic reviews in making decisions in clinical settings. The review was carried out in two stages: (1) a formal scoping review to understand the extent to which evidence from systematic reviews is sought, appraised, understood, and used to inform decision-making in four key areas: clinical practice, health systems management, public health, and policy making; (2) a systematic review to determine the impact on professional performance and health care outcomes of interventions for seeking, appraising, and applying evidence from systematic reviews (as a source document) in decision-making by clinicians, which is reported in this manuscript.
The databases of Medline (1950 to July 2009), EMBASE (1980 to July 2009), CINAHL (1982 to July 2009), Cochrane Central Register of Controlled Trials (CENTRAL) (to July 2009), and LISA (Library and Information Science Abstracts) (1969 to July 2009) were searched using the terms systematic review, meta-analysis, evidence synthesis, methodologic review, and quantitative review combined with implement, use, utilize, seek, retrieve, appraise, and apply. No language restrictions were placed on the search strategy. The grey literature was searched after identifying key websites and search engines. Reference lists of all papers and relevant reviews were identified, and authors of relevant papers were contacted regarding any further published or unpublished work.
We included all study designs except qualitative studies. All health-care providers, including physicians, nurses, and allied health professionals (e.g., physiotherapists, speech pathologists, social workers, pharmacists) involved in providing direct patient care were included. Studies that included participants in training programs who are responsible for patient care (residents, fellows, and other pre-licensure health-care professionals) were eligible for inclusion. Allied health professionals not involved with direct patient care (e.g., medical transcriptionists), students, and persons in undergraduate training programs were not considered for this review. Any study that examined interventions intended to increase seeking, appraising, or applying evidence from systematic reviews by a clinician was included. The use of products or tools derived from systematic reviews (e.g., guidelines, clinical pathways) was not considered as the focus was the use of systematic reviews. Interventions designed to enhance the uptake of evidence from systematic reviews were included. Primary outcomes of interest were any objective measure of professional performance (e.g., prescribing patterns, use of diagnostic tests) or health outcomes for patients (e.g., adverse events, return visits, length of stay, decrease in admissions). Measures of health-care providers’ satisfaction, knowledge, or attitudes were systematically collected as a secondary outcome.
Standardized data abstraction forms were developed and pilot tested by the review team using the protocol to guide primary and secondary outcomes. The following information was extracted from each article: setting, location of care, country, unit of allocation, clinical area addressed, type of health-care professional (e.g., physician, nurse, physiotherapist, student nurse, resident), frequency and timing of the intervention, duration of intervention, format of the intervention (e.g., web-based, person-to-person contact), known effectiveness of the intervention for changing of health-care professional behaviors (e.g., evidence-based intervention), nature of the intervention (e.g., training, mode of payment, team approach), number of components included in the intervention, source and authors of the intervention (e.g., professional organization, governmental agency, health professionals training schools), mode of delivery (e.g., individuals or groups), and reliability and validity testing of outcome measurement tools and adherence (e.g., withdrawals, drop-outs). Two reviewers independently assessed each study and undertook data abstraction directly from primary studies. Disagreements were discussed until consensus was achieved. A third reviewer was available if consensus could not be reached. Authors were contacted for missing data or when clarification was required (e.g., intervention not described in sufficient detail).
Two independent reviewers assessed the methodological quality of all studies that were included for data abstraction. Any discrepancies in ratings were resolved by discussion. Reviewers were not blinded to study author, institution, or journal, as evidence indicates that little benefit is achieved through blinding.14,15 The criteria described in section 6.4 of the Data Collection Checklist from the Cochrane Effective Practice and Organisation of Care Group (available at: http://www.epoc.cochrane.org) were used. The criteria used to assess randomized trials were concealment of allocation, follow-up of professionals, follow-up of patients or episodes of care, blinded assessment of primary outcome(s), baseline measurement, reliable primary outcome measure(s), and protection against contamination.
Initial searches of electronic databases identified 12,737 records for the scoping review. Out of 8,104 de-duplicated titles and abstracts, nine studies were deemed relevant in our review.16-24 Review of full text articles led to exclusion of three studies because the intervention was not relevant22-24 and one study because it was a feasibility study.21 Five relevant randomized controlled trials met the full inclusion criteria (Fig. 1).16-20
Five cluster randomized controlled trials (RCT) examining educational interventions for seeking, appraising, and applying evidence from systematic reviews in decision-making by clinicians were identified (Tables 1).16-20 A meta-analysis of study outcomes was not possible due to the heterogeneity in the format of the interventions, the settings, and clinical areas being addressed.
Quality assessments of the studies indicate that selective reporting and other problems that could put them at a risk of bias were identified as sufficient to affect interpretation of results. All studies reported follow-up of professionals sufficiently.16-20 Allocation concealment was not done for two studies19,20 and was unclear for one study.16 For selective reporting, Gülmezoglu et al.17 were not able to report on one practice at follow-up due to the inability to measure. Survey data for controls at follow-up were deemed to be contaminated and thus not reported. Some doctors were working in more than one of the participating hospitals in Mexico, and this may have also contributed to contamination.17 Freedom from bias was unclear for Lemelin et al.16 due to inadequate information about the practices that gave informed consent to participate in the study, but later declined (19 out of 100 sites). Contact with the authors confirmed that the telephone survey was restricted by language, and the survey was terminated if the patient did not speak English, thus introducing another source of bias.
Two studies were identified following assessment for methodological quality as being of low quality.19,20 Concerns for both studies lay in the lack of rigor related to data collection.19,20 Data collectors were not blinded to whether a hospital was part of the intervention or control group in the study performed by Althabe and colleagues.19 Clinicians were aware of the treatment of interest and reported their own outcomes. which were not corroborated or assessed for accuracy in the trial conducted by Clarkson et al.20
Three cluster RCTs were based in a hospital obstetric care setting. Wyatt et al.18 recruited doctors and midwives from 25 obstetric units in two regions in the UK. Gülmezoglu and colleagues,17 building on the earlier work of Wyatt, recruited doctors and midwives from 22 hospitals in Mexico and 18 hospitals in Thailand. Althabe et al.19 conducted a trial in 17 public maternity hospitals in Argentina and 2 in Uruguay with physicians, residents, and midwives. One hundred thirty-three dentists were examined by Clarkeson et al.20 in Scotland, who were working in areas of a post-code-based system defining deprivation looking at their rates of fissure sealing of newly erupted molars. Lemelin et. al.16 examined primary care physicians located in 46 health service organizations in Canada, concentrating on the topic of preventive care.
Wyatt and colleagues18 used a single informal educational visit to obstetric units in the intervention group lasting between 1½–3 h in length. During these visits an obstetrician discussed with the senior obstetrician and midwife the principles of evidence-based obstetrics and how to find and select Cochrane reviews in order to inform clinical practice.18 Afterwards, a copy of the Cochrane database was donated to the unit for their use.18 Gülmezoglu et al.17 ran three interactive workshops encouraging participants to use the Cochrane Reproductive Health Library, along with other strategies such as the provision of computer equipment in order to access the systematic reviews and promotional brochures publicizing the Library. Similarly, Althabe and colleagues19 implemented a multi-intervention strategy including interactive workshops that ran over a 5-day period, along with computer equipment and a copy of the Cochrane Reproductive Health Library. The approach evaluated by Lemelin et al.16 involved trained facilitators who organized meetings to introduce primary care staff to audit and feedback, academic detailing, and reminder systems. An average of 33 meetings took place over the 18-month study period with each meeting lasting about 1.75 h. Contact with the authors confirmed that one component focused on academic detailing and education materials that encouraged clinicians to seek and use systematic reviews to improve preventive care; however, it did not specifically provide access to systematic reviews. Clarkson and colleagues20 studied dentists, and those randomized into the ‘education arm’ of their 2×2 factorial design were invited to a 1-day workshop with the goal to provide skills to implement an evidence-based approach to clinical practice and raise awareness of research methods in primary care. The other implementation strategies of this study were remuneration, and a combination of education and remuneration. All five trials included a control group that received no intervention.
All studies used multiple strategies in their intervention, but only Gülmezoglu et al.,17 Lemelin et al.,16 and Althabe et al.19 explicitly describe their interventions as multi-faceted.16,17 Gülmezoglu et al.17 and Wyatt et al.18 focused on ‘knowledge access’ with both interventions promoting the singular goal of helping clinicians use systematic reviews with supportive measures such as the provision of tools (e.g., computers, printers, copy of the Cochrane Reproductive Health Library). Clarkson et al.20 offered an interactive workshop that encouraged the use of systematic reviews as one component of the educational session, but did not provide supportive materials. In contrast, the goal for the intervention evaluated by Lemelin et al.16 was to improve prevention in primary care, with one strategy out of the seven helping clinicians use systematic reviews to make decisions, and Althabe et al.19 focused on facilitating the adoption of evidence-based practices with one strategy out of eight offering a component that encouraged clinicians to seek out systematic reviews for decision-making.
Wyatt and colleagues18 measured four clinical markers [rates of perineal suturing with polyglycolic acid, ventouse delivery (vacuum extraction), prophylactic antibiotics in caesarean section, and steroids in preterm delivery] using data collected through chart audits before and 9 months after an educational visit. Results indicate no difference between intervention and control units in uptake of these practices.18 Rates for ventouse delivery were reported as significant in intervention units between baseline and follow-up (risk ratio for failure to use ventouse delivery: 0.68; 95% CI 0.59–0.78), but not in control units (risk ratio for failure to use ventouse delivery: 0.96; 95% CI 0.82–1.12).
In a similar approach, Gülmezoglu et al.17 selected ten clinical practices recommended in the Cochrane Reproductive Health Library (such as antibiotic use for caesarean section and social support during labor). Data were collected at baseline, then again after 10–12 months.17 In both study sites (Mexico and Thailand), there were no statistically significant differences between the intervention and control groups for the outcomes studied.17 The largest improvement reported was a higher reduction in the use of episiotomy in the intervention hospitals in Thailand, reported as approaching statistical significance (difference in adjusted mean rate =-5.3%; 95% CI: -0.1% to 10.7%).16 Gülmezoglu et al.17 also conducted a survey to gather information about awareness and use of the Cochrane Reproductive Health Library reporting that awareness (baseline to follow-up: 24.8%–65.5% in Mexico and 33.9%–83.3% in Thailand) and use (baseline to follow-up: 4.8%–34.9% in Mexico and 15.5%–76.4% in Thailand) increased substantially.
In the third cluster randomized trial that focused on obstetrics, Althabe and colleagues19 collected data at baseline, at the end of the 18-month intervention, and at follow-up 12 months after the end of the intervention on two primary outcomes (rates of episiotomy, rates of prophylactic use of oxytocin). Post-intervention, the rate of use of prophylactic oxytocin increased (baseline to follow-up: 2.1%–83.6% at intervention hospitals, 2.6% to 12.3% at control hospitals; p=0.01 for the difference in changes). The rate of use of episiotomy decreased at intervention hospitals, but remained stable at control hospitals (baseline to follow-up: 41.1%–29.9% at intervention hospitals, baseline and follow-up: 43.5% and 44.5% at control hospitals; p<0.001 for the difference in changes). Although the absolute difference in rate changes remained increased in the intervention hospitals as compared to the control hospitals at follow-up 12 months after the intervention was completed, it did not reach statistical significance. All results were reported for both countries, Argentina and Uruguay, combined.
Lemelin and colleagues16 identified 13 preventive maneuvers and scored practices using an index of preventive performance. Baseline data were collected and follow-up data at 9, 15, and 18 months, using chart audits. Preventive performance was identified as similar at baseline between intervention and control groups, with an absolute improvement in the intervention group of 11.5% (p<0.001) at 18-month follow-up (risk ratio: 1.35). Improvement in the preventive performance index is reported at all data collection points (9 months, 15 months, and 18 months) over time for the intervention; however, this is not tracked to specific preventive measures.
Clarkson et al.20 investigated the effects of financial incentive and of educational sessions on sealant treatment on newly erupted molars of 12–14 year olds. Data were collected 12 months post-intervention. Cluster-level analysis showed a significant increase in sealant treatment in the fee arms (fee alone, or fee + education session) compared with the other arms (adjusted risk difference, 9.8%; CI 1.8%–17.8%).
To our knowledge, this is the first systematic review of the evidence on the impact of interventions for seeking, appraising, and applying evidence from systematic reviews in decision-making by clinicians. The review of five studies revealed an absence of research on interventions that encourage clinicians to use systematic reviews in decision-making. Interventions focusing on enhancing the use of systematic reviews ranged from one meeting lasting as little as 1.5 h in one study,18 to being delivered over an average of 33 meetings lasting approximately 1.75 h each over an 18-month time span.16 Wyatt and colleagues report increased rates of use of ventouse delivery in intervention units between baseline and follow-up; however, since both intervention and control groups showed similar performance at follow-up, the authors suggest that this may be due to regression to the mean.22 Both Lemelin et al.16 and Althabe et al.19 indicated positive outcomes in changing practice; however, neither examined clinician use of systematic reviews in decision-making as its sole focus, thus creating difficulty in determining the extent to which this component can account for changes in professional behavior, if at all.16,19 No studies were identified that examined health outcomes for patients.
It is not possible to know if associations in one clinical area can provide understanding for other clinical areas. Learning from Wyatt and colleagues18 and their use of a single educational visit along with support materials, Gülmezoglu and colleagues17 similarly focused on pregnancy and childbirth, but used an alternate strategy of interactive workshops with support materials. One significantly different feature between the two studies is that Gülmezoglu and colleagues completed their work in developing countries.17 Although both authors confirm a lack of success in the use of either educational strategy, these studies build upon our knowledge within the area of obstetrics by providing the opportunity to examine outcomes from different interventions. Althabe and colleagues primarily were encouraging clinicians to develop and implement guidelines; however, a component of their workshops specifically encouraged practitioners to use systematic reviews in decision-making as well as providing the tools to support this (e.g., Cochrane Reproductive Health Library).19 Similar to Lemelin et al.16, it is not possible to assess the impact of this one component on its own.
All five studies highlight several considerations for the evaluation of systematic reviews and their importance in evidence-based health care. Two studies focused on the goal of ‘knowledge access’ in obstetrical care.17,18 Both sets of researchers did not inform staff of the practices selected as outcomes, but rather focused on giving them the knowledge and skills to use available tools (such as the Cochrane Library or the Reproductive Health Library), and extract the information that they would identify as important, and implement changes. Gülmezoglu et al.17 note this may present challenges as there is no prescriptive guidance for translating messages from systematic reviews into concrete practice changes that might be recognized in other tools, such as algorithms or clinical practice guidelines. Although Lemelin et al.16 revealed their interest in changing preventive practices to the intervention group and were able to show positive outcomes, the limitation of having this one strategy embedded within a multi-component intervention does not allow us to determine the effectiveness of this approach on its own.
Several limitations in this review should be considered. First, the literature in this area is poorly indexed. This challenge was acknowledged in the choice to conduct a scoping review as a strategy to understand the overall state of research activity in the area of the use of systematic reviews in health-care decision-making. Scoping reviews are often undertaken when an area has little published research available or the area is poorly understood.25 The search strategy for the scoping review allowed for a very broad search and examination of over 8,000 articles. The small number of studies available for final assessment indicates the difficulty in summarizing and identifying key aspects in successful strategies that encourage clinicians to use systematic reviews in decision-making. The limited empirical data render the strength of evidence weak in relation to the effectiveness and the types of interventions that encourage clinicians to use systematic reviews. Second, this review is limited by the reports of methods from the included studies. When additional details were needed for data abstraction, authors of primary studies were contacted.
This review found five relevant studies that provide limited evidence that the interventions outlined changed professional behavior. Overall, there is insufficient evidence to support or refute interventions for seeking, appraising, and applying evidence from systematic reviews in decision-making by clinicians. Considerations for future research include examining the circumstances and contexts under which systematic reviews are most effective. This includes how systematic reviews are accessed under normal settings, where they are used (e.g., beside, office), when they are used (e.g., before, during, or after a consultation), and the specific characteristics that make systematic reviews easy to use in terms of presentation and format of information.
Contributors: SES created the study concept and design. LP, AM, and SES constructed and refined the search strategy. SES and LP acquired the data. Analysis and interpretation of the data were completed by LP, KM, and SES. Drafting of the manuscript and critical revision for important intellectual content was done by LP, KM, SS, MD, AM, and SES. LP wrote the final report and is the guarantor for the paper. All authors contributed to the final paper.
We are grateful to David Newton for his technical assistance.
Funders Canadian Institutes of Health Research, and each author’s institution. The funding source had no role in the study design, collection, analysis, and interpretation of results, in the writing of the report, or in the decision to submit the paper for publication.
Prior presentations May 19, 2010, Cochrane Canada 8th Annual Symposium
Conflict of interest None disclosed.
Ethical approval Not required