|Home | About | Journals | Submit | Contact Us | Français|
Background: The use of clinical decision support systems to facilitate the practice of evidence-based medicine promises to substantially improve health care quality.
Objective: To describe, on the basis of the proceedings of the Evidence and Decision Support track at the 2000 AMIA Spring Symposium, the research and policy challenges for capturing research and practice-based evidence in machine-interpretable repositories, and to present recommendations for accelerating the development and adoption of clinical decision support systems for evidence-based medicine.
Results: The recommendations fall into five broad areas—capture literature-based and practice-based evidence in machine-interpretable knowledge bases; develop maintainable technical and methodological foundations for computer-based decision support; evaluate the clinical effects and costs of clinical decision support systems and the ways clinical decision support systems affect and are affected by professional and organizational practices; identify and disseminate best practices for work flow–sensitive implementations of clinical decision support systems; and establish public policies that provide incentives for implementing clinical decision support systems to improve health care quality.
Conclusions: Although the promise of clinical decision support system–facilitated evidence-based medicine is strong, substantial work remains to be done to realize the potential benefits.
Clinical decision support systems (CDSSs) have been hailed for their potential to reduce medical errors1 and increase health care quality and efficiency.2 At the same time, evidence-based medicine has been widely promoted as a means of improving clinical outcomes, where evidence-based medicine refers to the practice of medicine based on the best available scientific evidence. The use of CDSSs to facilitate evidence-based medicine therefore promises to substantially improve health care quality.
The Evidence and Decision Support track of the 2000 AMIA Spring Symposium examined the challenges in realizing the promise of CDSS-facilitated evidence-based medicine. This paper describes the activities of this track and summarizes discussions in specific research and policy recommendations for accelerating the development and adoption of CDSSs for evidence-based medicine.
We introduce a new term, “evidence-adaptive CDSSs,” to distinguish a type of CDSS that has technical and methodological requirements that are not shared by CDSSs in general. To clarify this distinction between evidence-adaptive and other CDSSs, we define the following terms as they are used in this paper:
The scientific literature is the major source of evidence for evidence-based medicine, although literature-based evidence should often be complemented by local, practice-based evidence for individual and site-specific clinical decision making. Evidence-based medicine is conducted by the health care provider and may or may not be computer-assisted.
The speakers for the Evidence and Decision Support track are listed at the end of this paper. The track consisted of three panels and two break-out discussion sessions.
The first panel addressed the role of information technology in the dissemination and critical appraisal of research evidence, the technical challenges and opportunities of evidence-adaptive computerized decision support, and the organizational and workflow issues that arise when effecting practice change through information technology (Haynes, Tang, and Kaplan, respectively).
The second panel presented two case studies of evidence-based quality improvement projects (Packer, Stone) and summarized the status of the GuideLine Interchange Format (GLIF), a developing foundational technology for distributed evidence-adaptive CDSSs (Greenes). Finally, a commentator panel expanded on some of the pitfalls to changing practice through technology (Gorman) and on the information-technology funding agenda of the Agency for Healthcare Research and Quality (Burstin).
Interspersed with these panel presentations were two moderated break-out sessions, in which participants worked to identify the research and policy needs and priorities for effective computer-supported practice change.
All conference sessions were audiotaped. Using these audiotapes, we distilled five central areas of activity that are essential to the goal of increased adoption of CDSSs for evidence-based medicine.
Clinical decision support systems can be only as effective as the strength of the underlying evidence base. That is, the effectiveness of CDSSs will be limited by any deficiencies in the quality or relevance of the research evidence. Therefore, one key step in developing more effective CDSSs is to generate not simply more clinical research evidence, but more high-quality, useful, and actionable evidence that is up-to-date, easily accessible, and machine interpretable.
Only about half the therapeutic interventions used in inpatient4,5 and outpatient6 care in internal and family medicine are supported in the research literature with evidence of efficacy. The other half of the interventions either have not been studied or have only equivocal supportive evidence. Several problems exist with using the research literature for evidence-based medicine. First, the efficacy studies of clinical practice that form the basis for evidence-based medicine constitute only a small fraction of the total research literature.7 Furthermore, this clinical research literature has been beset for decades with study design and reporting problems8,9—problems that still exist in the recent randomized trial,10 systematic review,11,12 and guidelines13 literature. As the volume of research publication explodes while quality problems persist, it is not surprising that most clinicians consider the research literature to be unmanageable14 and of limited applicability to their own clinical practices.15,16
The full promise of CDSSs for facilitating evidence-based medicine will occur only when CDSSs can “keep up” with the literature—that is, when evidence-adaptive CDSSs can monitor the literature for new relevant studies, identify those that are of high quality, and then incorporate the best evidence into patient-specific assessments or practice recommendations. Automation of these tasks remains an open area of research. In the meantime, the best electronic resources for evidence-based medicine include the Cochrane Library, Best Evidence, and Clinical Evidence, resources that cull the best of the literature to provide an up-to-date solid foundation for evidence-based practice. The drawback to these resources is that their contents are textual and thus not machine-interpretable by present-day CDSSs.
In contrast, if the research literature were available as shared, machine-interpretable knowledge bases, then CDSSs would have direct access to the newest research for automated updating of their knowledge bases. The Trial Bank project is a collaboration with the Annals of Internal Medicine and JAMA to capture the design and results of randomized trials directly into structured knowledge bases17 and is a first step toward the transformation of text-based literature into a shared, machine-interpretable resource for evidence-adaptive CDSSs.
Although the research literature serves as the foundation for evidence-based practice, it is not uncommon that local, practice-based evidence is required for optimizing health outcomes. For example, randomized trials have shown that patients with symptomatic carotid artery stenosis have fewer strokes if they receive a surgery called carotid endarterectomy.18 If complication rates from the surgery are greater than about 6 percent, however, the benefits are nullified.19 Despite this, only 19 percent of physicians know the CEA complication rates of the hospitals in which they operate or to which they refer patients.20 For clinical problems with locally variable parameters, therefore, developers of CDSSs should place a high priority on obtaining local practice-based evidence to complement the literature-based evidence.
Practice-based evidence may also be useful for the development of practice guidelines. Although the evidentiary support for individual decision steps in a guideline comes primarily from literature-based evidence, as discussed above, a guideline's process flow is usually constructed on the basis of expert opinion only. With more practice-based information on clinical processes and events, however, guideline developers may be able to improve the way they design process flows.
As useful as practice-based evidence may be, it is often not easy to come by. The informatics community can foster this much-needed research by developing information technologies for practice-based research networks to automatically capture clinical processes and events in diverse outpatient settings. Many research and policy issues concerning these research networks—from the standardization of data items to data ownership and patient privacy—are active areas of inquiry.21–24
The Internet and other sources of research evidence have provided patients with many more options for obtaining health information but have also increased the potential for patients to misinterpret or become misinformed about research results.24,25 As a result, patients are now less dependent on clinicians for information, but still trust clinicians the most for help with selecting, appraising, and applying a profusion of information to health decisions.26 Clinical decision support systems can support this growing involvement of patients in clinical decision making through interactive tools that allow patients to explore relevant information that can foster shared decision making.27,28 Systems that provide both patients and clinicians with valid, applicable, and useful information may result in care decisions that are more concordant with current recommendations, are better tailored to individual patients, and ultimately are associated with improved clinical outcomes. The actual effects of these CDSSs on care decisions and outcomes should be evaluated.
The gap between the current state of CDSSs and the full promise of CDSSs for evidence-based medicine suggests a research and development agenda. On the basis of the expert panels and discussion sessions at the Congress, we recommend the following steps for researchers, developers, and implementers to take in the five areas of activity essential to increasing adoption of evidence-adaptive CDSSs.
If clinical research is to improve clinical care, it must be relevant, of high quality, and accessible. The research should provide evidence of efficacy, effectiveness, and cost-effectiveness for typical inpatient and outpatient practice settings.29 If CDSSs are to help translate this research into practice, CDSSs must have direct machine-interpretable access to the research literature, so that automated methods can be brought to bear on the myriad tasks involved in “keeping up with the literature.” Thus, the establishment of shared, machine-interpretable knowledge bases of research and practice-based evidence is a critical priority. On the basis of discussions at the conference, we identify six specific recommendations for action:
Figure 1 depicts the informatics architecture that we suggest is needed for CDSSs to facilitate evidence-based practice. In this architecture, CDSSs are situated in a distributed environment that comprises multiple knowledge repositories as well as the electronic medical record. Vocabulary and interface standards will be crucial for interoperation among these systems. To provide patient-specific decision support at the point of care, CDSSs need to interface with the electronic medical record to retrieve patient-specific data and, increasingly, also to effect recommended actions through computerized order entry. Evidence-adaptive CDSSs also need to interface with up-to-date repositories of clinical research knowledge. No longer should CDSSs be thought of as stand-alone expert systems.
In addition to establishing standardized communication among CDSSs, electronic medical records, and knowledge repositories, we also need better models of individualized patient decision making in real-world settings. Formal models of decision making such as decision analysis are not commonly used; much methodological work remains to be done on mapping real-world decision-making challenges to tractable computational approaches.
We identify several additional priorities for evidence-adaptive CDSSs in particular. These priorities include the development of methods for adjusting for the quality of the evidence base, and efficient, sustainable methods for ensuring that CDSS recommendations reflect up-to-date evidence.
Despite the promise of CDSSs for improving care, formal evaluations have shown that CDSSs have only a modest ability to improve intermediate measures such as guideline adherence and drug dosing accuracy.31–34 The effect of CDSSs on clinical outcomes remains uncertain.32 Thus, more evaluations of CDSSs are needed to produce valid and generalizable findings on the clinical and organizational aspects of CDSS use. A wide variety of evaluation methods are available,35–37 and both quantitative and qualitative methods should be used to provide complementary insight into the use and effects of CDSSs. All types of evaluation studies, not just randomized trials, deserve increased attention and funding.38,39
In light of the current focus on errors in medicine, a special class of evaluation study deserves particular mention. These studies are ongoing, iterative reevaluations and redesigns of CDSSs that identify and amplify system benefits while identifying and mitigating unanticipated system errors or dangers. The rationale for these types of studies is that automation in other industries has not always been beneficial, and indeed, automation can interfere with and degrade overall organizational performance.40 Woods and Patterson41 offer a cautionary note from the transportation industry:
Despite the fact that these systems are often justified on the grounds that they would help offload work from harried practitioners, we find that they in fact create new additional tasks, force the user to adopt new cognitive strategies, require more knowledge and more communication, at the very times when the practitioners are most in need of true assistance .
Clinicians and health care managers must be continuously vigilant against unforeseen adverse effects of CDSS use.
Relatively few examples of CDSSs can be found in practice. In part, this limited adoption may be because CDSSs are as much an organizational as a technical intervention, and organizational, professional, and other challenges to implementing CDSSs may be as daunting as the technical challenges.
Significant financial and organizational resources are often needed to implement CDSSs, especially if the CDSS requires integration with the electronic medical record or other practice systems. In a competitive health care marketplace, financial and reimbursement policies can therefore be important drivers both for and against the adoption of effective CDSSs. As more evaluation studies become available, policy makers will be better able to tailor these policies to promote only those CDSSs that are likely to improve health care quality.
The coupling of CDSS technology with evidence-based medicine brings together two potentially powerful methods for improving health care quality. To realize the potential of this synergy, literature-based and practice-based evidence must be captured into computable knowledge bases, technical and methodological foundations for evidence-adaptive CDSSs must be developed and maintained, and public policies must be established to finance the implementation of electronic medical records and CDSSs and to reward health care quality improvement.
The authors thank the many discussion participants whose anonymous comments were included in this paper. They also thank Patricia Flatley Brennan for her helpful comments on an earlier draft of this manuscript, and Amy Berlin for her assistance in preparing the manuscript.
Panelists and Group Leaders
R. Brian Haynes, MD, PhD, Chief, Health Information Research Unit, McMaster University
Paul Tang, MD, Medical Director of Clinical Informatics, Palo Alto Medical Foundation
Bonnie Kaplan, PhD, Yale Center for Medical Informatics and President, Kaplan Associates
Marvin Packer, MD, Harvard Pilgrim Health Care
Tamara Stone, MBS, PhD, Assistant Professor of Health Management, University of Missouri
Robert Greenes, MD, PhD, Director, Decision Sciences Group, Brigham and Women's Hospital, Boston, Massachusetts
Paul Gorman, MD, Assistant Professor, Oregon Health and Science University
Helen Burstin, MD, MPH, Director, Center for Primary Care Research, Agency for Healthcare Research and Quality
Gordon D. Brown, PhD, Health Management and Informatics, University of Missouri
Richard Bankowitz, MD, MBA, University Health System Consortium
Harold Lehmann, MD, PhD, Director of Medical Informatics Education, Johns Hopkins University School of Medicine
This work was supported in part by a United States Presidential Early Career Award for Scientists and Engineers awarded to Dr. Sim and administered through grant LM-06780 of the National Library of Medicine.