The gap between the current state of CDSSs and the full promise of CDSSs for evidence-based medicine suggests a research and development agenda. On the basis of the expert panels and discussion sessions at the Congress, we recommend the following steps for researchers, developers, and implementers to take in the five areas of activity essential to increasing adoption of evidence-adaptive CDSSs.
Capture of Literature-based and Practice-based Evidence
If clinical research is to improve clinical care, it must be relevant, of high quality, and accessible. The research should provide evidence of efficacy, effectiveness, and cost-effectiveness for typical inpatient and outpatient practice settings.29
If CDSSs are to help translate this research into practice, CDSSs must have direct machine-interpretable access to the research literature, so that automated methods can be brought to bear on the myriad tasks involved in “keeping up with the literature.” Thus, the establishment of shared, machine-interpretable knowledge bases of research and practice-based evidence is a critical priority. On the basis of discussions at the conference, we identify six specific recommendations for action:
Recommendations for Clinical and Informatics Researchers
- Conduct better quality clinical research on the efficacy, effectiveness, and efficiency of clinical interventions, particularly in primary care settings.
- Continue to develop better methods for synthesizing results from a wide variety of study designs, from randomized trials to observational studies .
- Develop shareable, machine-interpretable repositories of up-to-date evidence of multiple types (e.g., from clinical trials, systematic reviews, decision models).
- Develop shareable, machine-readable repositories of executable guidelines that are linked to up-to-date evidence repositories.
- Define and build standard interfaces among these repositories, to allow evidence to be linked automatically among systems for systematic reviewing, decision modeling, and guideline creation and maintenance.
- Develop an informatics infrastructure for practice-based research networks to collect practice-based evidence.
Establishment of a Technical and Methodological Foundation
Figure 1 depicts the informatics architecture that we suggest is needed for CDSSs to facilitate evidence-based practice. In this architecture, CDSSs are situated in a distributed environment that comprises multiple knowledge repositories as well as the electronic medical record. Vocabulary and interface standards will be crucial for interoperation among these systems. To provide patient-specific decision support at the point of care, CDSSs need to interface with the electronic medical record to retrieve patient-specific data and, increasingly, also to effect recommended actions through computerized order entry. Evidence-adaptive CDSSs also need to interface with up-to-date repositories of clinical research knowledge. No longer should CDSSs be thought of as stand-alone expert systems.
Figure 1 Architecture for the capture and use of literature-based and practice-based evidence, showing the distributed nature of the knowledge and functionality involved in the use of CDSSs to support evidence-based medicine. Vocabulary and interface standards (more ...)
In addition to establishing standardized communication among CDSSs, electronic medical records, and knowledge repositories, we also need better models of individualized patient decision making in real-world settings. Formal models of decision making such as decision analysis are not commonly used; much methodological work remains to be done on mapping real-world decision-making challenges to tractable computational approaches.
We identify several additional priorities for evidence-adaptive CDSSs in particular. These priorities include the development of methods for adjusting for the quality of the evidence base, and efficient, sustainable methods for ensuring that CDSS recommendations reflect up-to-date evidence.
Recommendations for Researchers and Developers
- Continue development of a comprehensive, expressive clinical vocabulary that can scale from administrative to clinical decision support needs.
- Continue to develop shareable computer-based representations of clinical logic and practice guidelines.
- Develop tools for knowledge editors to incorporate new literature-based evidence into CDSS knowledge bases; specify the clinical context in which that knowledge is applicable (e.g., that a rule is for the treatment of stable outpatient diabetic patients only); and customize the literature-based evidence for local conditions (e.g., factoring in local surgical complication rates).
- Explore and develop automatic methods for updating CDSS knowledge bases to reflect the current state and quality of the literature-based evidence.
- Develop more flexible models of decision making that can accommodate clinical evidence of varying methodological strength and relevance, so that evidence from randomized trials (Level I evidence by U.S. Preventive Services Task Force criteria30) is accorded more weight than evidence from case reports or expert opinion (Level III evidence).
- Develop models of decision making that can simultaneously accommodate the beliefs, perspectives, and values of multiple decision makers, including those of physicians and patients.
- Develop methods for constructing and selecting among decision models of scalable granularity and specificity that are neither too general nor too specific for the case at hand.
Recommendations for Current CDSS Developers
- Adopt and use standard vocabularies and standards for knowledge representation (e.g., GLIF) as they become available.
- We consider it axiomatic that CDSSs must be based on the best available evidence. Incorporate into the CDSS knowledge base the current best literature-based and practice-based evidence, and either provide mechanisms for keeping the knowledge base up-to-date or explain why keeping up with the evidence is not applicable.
- Explicitly describe the care delivery setting and clinical scenarios for which the CDSS is applicable (e.g., that a CDSS for diabetes treatment is intended for the management of stable outpatient diabetics only).
- Integrate CDSSs with electronic medical records and other relevant systems using appropriate interoperability standards (e.g., HL-7).
- Develop more CDSSs for outpatient settings. Approximately 60 percent of U.S. physicians practice in outpatient settings, where an aging population is requiring increasingly complex diagnostic, treatment, and supportive services.
Recommendation for Policy Makers, Organizations, and Manufacturers
- Fund development and demonstration of inter-organizational sharing of evidence-based knowledge and its application in diverse CDSSs.
Evaluation of Clinical Decision Support Systems
Despite the promise of CDSSs for improving care, formal evaluations have shown that CDSSs have only a modest ability to improve intermediate measures such as guideline adherence and drug dosing accuracy.31–34
The effect of CDSSs on clinical outcomes remains uncertain.32
Thus, more evaluations of CDSSs are needed to produce valid and generalizable findings on the clinical and organizational aspects of CDSS use. A wide variety of evaluation methods are available,35–37
and both quantitative and qualitative methods should be used to provide complementary insight into the use and effects of CDSSs. All types of evaluation studies, not just randomized trials, deserve increased attention and funding.38,39
In light of the current focus on errors in medicine, a special class of evaluation study deserves particular mention. These studies are ongoing, iterative reevaluations and redesigns of CDSSs that identify and amplify system benefits while identifying and mitigating unanticipated system errors or dangers. The rationale for these types of studies is that automation in other industries has not always been beneficial, and indeed, automation can interfere with and degrade overall organizational performance.40
Woods and Patterson41
offer a cautionary note from the transportation industry:
Despite the fact that these systems are often justified on the grounds that they would help offload work from harried practitioners, we find that they in fact create new additional tasks, force the user to adopt new cognitive strategies, require more knowledge and more communication, at the very times when the practitioners are most in need of true assistance .
Clinicians and health care managers must be continuously vigilant against unforeseen adverse effects of CDSS use.
Recommendations for Evaluators
- Evaluate CDSSs using an iterative approach that identifies both benefits and unanticipated problems related to CDSS implementation and use: all CDSSs can benefit from multiple stages and types of testing, at all points of the CDSS life cycle.
- Conduct more CDSS evaluations in actual practice settings, including ambulatory settings.
- Use both quantitative and qualitative evaluation methodologies to assess multiple dimensions of CDSS use and design (e.g., the correctness, reliability, and validity of the CDSS knowledge base; the congruence of system-driven processes with clinical roles and work routines in actual practice; and the return-on-investment of system implementation). Qualitative studies should incorporate the expertise of ethnographers, sociologists, organizational behaviorists, or other field researchers from within and without the medical informatics community, as applicable.
- If preliminary testing suggests that a CDSS could improve health outcomes, the CDSS should be evaluated to establish the presence or absence of clinical benefits. Any randomized clinical trials that are conducted should have sufficient sample sizes to detect clinically meaningful outcomes, should randomize physicians or clinical units rather than patients, and should be analyzed using methods appropriate for cluster randomization studies.
- Establish partnerships between academic groups and community practices to conduct evaluations.
Promotion of the Implementation of CDSSs
Relatively few examples of CDSSs can be found in practice. In part, this limited adoption may be because CDSSs are as much an organizational as a technical intervention, and organizational, professional, and other challenges to implementing CDSSs may be as daunting as the technical challenges.
Recommendations for CDSS Implementers
- Establish a CDSS implementation team composed of clinicians, information technologists, managers, and evaluators to work together to customize and implement the CDSS.
- Develop a process for securing clinician agreement regarding the science underlying the recommendations of a CDSS. For evidence-adaptive CDSSs, a process is also needed for maintaining clinician awareness of and agreement with any changes in CDSS recommendations that may result from new evidence.
- Plan explicitly for work flow re-engineering and other people, organizational, and social issues and incorporate change management techniques into system development and implementation. For example, a CDSS that recommends immediate angioplasty instead of thrombolysis as a new treatment option for acute coronary syndromes will necessitate a major restructuring of the hospital's resource use and work practices.
Establishment of Public Policies That Provide Incentives for Implementing CDSSs
Significant financial and organizational resources are often needed to implement CDSSs, especially if the CDSS requires integration with the electronic medical record or other practice systems. In a competitive health care marketplace, financial and reimbursement policies can therefore be important drivers both for and against the adoption of effective CDSSs. As more evaluation studies become available, policy makers will be better able to tailor these policies to promote only those CDSSs that are likely to improve health care quality.
Recommendations for Policy Makers
- Develop financial and reimbursement policies that provide incentives for health-care providers to implement and use CDSSs of proven worth.
- Develop and implement financial and reimbursement policies that reward the attainment of measurable quality goals, as might be achieved by CDSSs.
- Promote coordination and leadership across the health care and clinical research sectors to leverage informatics promotion and development efforts by government, industry, AMIA, and others.