Systematic reviews are the most valid approach but are impractical in isolation.
Systematic reviews are protocol-driven efforts to comprehensively search for and select the most valid relevant information, critically evaluate and synthesize that information, and generate summative reports. These reviews provide the most valid method for tracking down the best evidence for a given question.
With a global network of volunteers and tremendous effort, the Cochrane Collaboration has amassed 1,837 systematic reviews addressing intervention-related questions and 11,669 abstracts of additional published systematic reviews [9
]. But 13,000 systematic reviews hardly address the number of questions that occur in practice. Cochrane personnel have estimated it would take 30 years to summarize all the current controlled trials in the form of Cochrane reviews [10
], not accounting for new evidence published during those 30 years.
Many clinical questions (e.g., diagnosis or prognosis questions) are not typically addressed through controlled trials. The medical literature contains systematic reviews for other types of clinical questions, such as the Rational Clinical Examination series in JAMA that provides systematic reviews of studies evaluating the use of patient history, physical examination, and office-based diagnostic tests for predicting or ruling out common diagnoses. When available (and done using methods of the highest methodological rigor), these systematic reviews represent the best evidence for these types of clinical questions, but these reviews are less common than systematic reviews of randomized trials.
Systematic reviews limited to randomized trials may also not provide the best available evidence for intervention-related questions. When randomized trial data are available to assess efficacy, these data represent the best available evidence. Many interventions have some evidence of efficacy (e.g., cohort studies) but have not yet been studied through randomized trials and will thus not be addressed by systematic reviews limited to randomized trials. For example, considering surgical repair for elderly patients with massive rotator cuff tears, no randomized trials determine if surgery improves symptoms, but cohort studies are available that provide the most valid current information for informed clinical decision making [11
]. Potential harms of interventions (which may be rare but serious) may also be found through cohort studies and other “lower validity” studies, where randomized trials have not documented relevant outcomes in sufficiently large samples.
Systematic reviews provide the best available evidence for answering the specific questions addressed at the time the systematic reviews are conducted, but a single systematic review can take years to perform. Two important limitations of clinical references derived solely from systematic reviews are that such reviews can be years out of date and that many significant medical topics lack systematic reviews but have research evidence. The large effort involved in conducting systematic reviews makes it impossible for systematic reviews alone to keep pace with medical information needs.
Systematic literature surveillance provides the most valid method for tracking down the best evidence not yet identified by systematic reviews.
Systematic literature surveillance is a method for supplementing systematic reviews. Systematic literature surveillance involves systematically assessing new research reports for relevance and validity and summarizing the best new research evidence. Systematic literature surveillance can be more efficient than systematic reviews for addressing a large number of information needs, because each article can be identified and evaluated once rather than be separately identified and evaluated for each question. The advantages of systematic literature surveillance, by virtue of increased efficiency, include greater currency and ability to cover more topics and more clinical questions than systematic reviews. However, systematic literature surveillance alone is inadequate for identifying the best research evidence, because it does not include older research reports that may provide better evidence, unless, of course, retrospective systematic literature surveillance is undertaken.
Complementing systematic reviews with systematic literature surveillance is necessary to provide current best research evidence for clinical references, because clinical reference content is prepared before the questions are asked. The combination of systematic reviews and systematic literature surveillance provides a balance for addressing the most information needs with the best research evidence.
Systematic literature surveillance has been developed for newsletter and alerting services.
Several services use literature surveillance to inform primary care clinicians. Journal Watch [13
] and QuickScan Reviews [14
] monitor a defined set of journals for multiple specialties (including primary care specialties) and provide brief summaries and commentaries on articles of interest. ACP Journal Club/ Evidence-Based Medicine [15
] and Evidence-Based Practice Newsletter/InfoPOEMs [16
] use explicitly defined, systematic literature surveillance methods: they monitor a defined set of journals and report on articles selected for validity and relevance, with summaries based on critical appraisal and commentaries with a focus on clinical application.
Little has been published on the size and scope of the literature that should be reviewed to inform primary care clinicians, but the Evidence-Based Practice and ACP Journal Club services have reported on their efforts. The editors of the newsletter Evidence-Based Practice reviewed 85 journals and counted 8,085 original articles over a 6-month period (January through June 1997) [17
]. They classified articles as Patient-Oriented Evidence that Matters (POEMs) if the articles addressed a clinical question encountered by a typical family physician at least once every 6 months, measured patient-oriented outcomes (e.g., mortality or symptom reduction), and presented results that would require a change in practice for the typical family physician. The editors found 211 of the articles (2.6%) to be sufficiently relevant for publication as POEMs in their newsletter.
The editors of ACP Journal Club reviewed 58 journals in 1992 with a rigorous focus on the validity of articles with abstracts [18
]. The top 20 journals had 6,837 articles with abstracts, of which 339 (5%) met ACP Journal Club criteria and 160 (2.3%) were selected for publication in ACP Journal Club.
Different systematic literature surveillance systems are needed for the clinical knowledgebase.
The Evidence-Based Practice and ACP Journal Club approaches are useful for clinical alerting, in other words, reporting new research findings to clinicians to raise awareness. However, these approaches are inadequate for identifying the best available information to be used when clinicians seek answers to questions. For example, if a physician has a patient with acute parotitis and wants guidance regarding antibiotic selection, that physician might not find information in sources limiting their reports to effects on patient-oriented outcomes from studies of the highest possible validity (i.e., randomized trials). However, having access to guidance from reviews or guidelines, or to studies of common etiologies and antimicrobial sensitivities, could provide the best available information and clinical usefulness.
A clinical knowledgebase should provide the best available information, even when the current evidence is limited to less valid studies or expert opinion, and should update that information as more valid studies are published. Systematic literature surveillance systems supporting a clinical knowledgebase must recognize when studies of limited validity or expert opinion represent the current state of knowledge and provide useful information for clinical decision making.
Systematic literature surveillance processes developed for updating clinical reference content need further development.
The authors could not identify any published literature on this subject. Several clinical reference products are promoted as up to date and evidence based, implying that product vendors conduct systematic surveillance of medical journals, but our efforts to review these products and their editorial policies and discuss methods with the product vendors have found no explicit methodology for how such surveillance is conducted.
The only clinical reference product, to our knowledge, that both provides rapid access to evidence-based information in the form of knowledge syntheses and updates syntheses through a systematic literature surveillance process with publicly stated methods is DynaMed [19
]. DynaMed is a clinical knowledgebase in development that has already been shown to answer more than half of family physicians' clinical questions [20
]. Systematic literature surveillance is used to keep DynaMed current with a process that balances validity, relevance, convenience, and affordability.
Multiple resources are monitored for updating DynaMed, including (as of August 2002) 45 original research journals, 3 systematic review collections, 9 journal review services, 5 drug information sources, and 9 sources for clinical review articles (these numbers have since increased). Article summaries vary in length based on the validity and relevance of individual articles and are incorporated directly into existing clinical topic summaries in the DynaMed database. To enhance the currency of information access, article summaries are made available immediately (the database is updated many times daily) and then secondarily peer reviewed. This process is currently text based and completely dependent on human effort.
The authors take the first steps toward defining a comprehensive systematic literature surveillance system.
While the Evidence-Based Practice and ACP Journal Club selection processes provide estimates of the scope and yield of systematic literature surveillance services for clinical alerting, the scope and yield of an identification-and-filtering process for maintaining a clinical knowledgebase is unknown. As a rough “first guess,” we created a bibliographic database to quantify the research literature pertinent to primary care and estimated the effort needed to conduct systematic literature surveillance for this literature collection.