|Home | About | Journals | Submit | Contact Us | Français|
Computerized physician order entry (CPOE) systems represent an important tool for providing clinical decision support. In undertaking this systematic review, our objective was to identify the features of CPOE-based clinical decision support systems (CDSSs) most effective at modifying clinician behavior. For this review, two independent reviewers systematically identified randomized controlled trials that evaluated the effectiveness of CPOE-based CDSSs in changing clinician behavior. Furthermore, each included study was assessed for the presence of 14 CDSS features. We screened 10,023 citations and included 11 studies. Of the 10 studies comparing a CPOE-based CDSS intervention against a non-CDSS control group, 7 reported a significant desired change in professional practice. Moreover, meta-regression analysis revealed that automatic provision of the decision support was strongly associated with improved professional practice (adjusted odds ratio, 23.72; 95% confidence interval, 1.75-∞). Thus, we conclude that automatic provision of decision support is a critical feature of successful CPOE-based CDSS interventions.
A significant gap exists between actual clinical practice and optimal patient care. For example, in a recent systematic review of the quality of health care in the United States, Schuster et al. found that only about 70% of patients received recommended acute care, and that only about 60% of patients received recommended care for chronic conditions.1 Moreover, the Institute of Medicine recently estimated that 44,000 to 98,000 patients die each year in hospitals as a result of preventable medical errors.2 Even when using the lower estimate, this statistic implies that more Americans die each year from medical errors than from motor vehicle accidents (43,458), breast cancer (42,297), or AIDS (16,516).3
Given this gap between actual clinical practice and ideal patient care, the Institute of Medicine and other key stakeholders have identified computerized physician order entry (CPOE) as an important strategy for improving professional practice and reducing medical errors.2 Indeed, by requiring clinicians to directly enter orders online, CPOE systems can virtually eliminate medical errors due to lost, incomplete, or illegible orders.4 Moreover, CPOE systems can significantly improve professional practice through the integration of clinical decision support systems (CDSSs). For example, a recent time series study found that a CPOE system with decision support features reduced the incidence of serious medication errors in a large hospital by 86%.5
While there is significant evidence for the effectiveness of CPOE-based CDSS interventions as a whole, little is known with regard to the specific CDSS elements that are most important in producing a desired change in clinician behavior. Thus, despite the insights gained from relevant qualitative studies,6 there remains a lack of evidence-based understanding as to why some CPOE-based CDSS interventions succeed, while others fail. Indeed, this is a problem that affects CDSS interventions in general,7 and we have conducted a systematic review of CDSSs provided both within and outside of CPOE systems in order to address this issue.8 The current systematic review represents a subset analysis of this larger work, in which special focus is placed on CDSSs provided in the context of CPOE systems. In undertaking this endeavor, our objective was to help guide the efforts of CPOE designers by providing a rigorous, evidence-based assessment of the CDSS features that are most effective at influencing clinician behavior in the context of computerized physician order entry.
We defined a computerized physician order entry (CPOE) system as a computer-based system that allows clinicians to enter orders directly. Also, we defined a clinical decision support system (CDSS) as any system designed to directly aid in clinical decision making, in which characteristics of individual patients are matched to a knowledge base for the purpose of generating patient-specific assessments or recommendations that are then presented to clinicians for consideration.9 In selecting studies, we included randomized controlled trials that evaluated the effectiveness of a CPOE-based CDSS or one of its features in changing an important clinician behavior in a real clinical setting. We considered physicians, physician assistants, and nurse practitioners to be valid clinician subjects. Moreover, we excluded studies with less than 7 units of randomization per study arm and studies scoring less than 5 points on the 10-point quality rating scale described below. In addition, we excluded studies not in English, studies in which compliance with the CDSS was mandatory, studies that did not describe the content of the decision support provided to clinicians, and studies that did not describe how clinicians interacted with the system.
We searched MEDLINE (1966-December 2002), CINAHL (1982-October 2002), and the Cochrane Controlled Trials Register (Fourth Quarter, 2002) for relevant studies. We did not limit the search by study type, and we used combinations of multiple search terms, which included the following: decision support systems, clinical; decision making, computer-assisted; reminder systems; feedback ; guideline adherence; medical informatics; communication; physician’s practice patterns; reminder$; feedback$; decision support$; and expert system. We also systematically searched the reference lists of included studies and of relevant reviews for potential studies.
We created a screening algorithm based on a subset of the inclusion criteria in order to facilitate the selection of papers. Using this algorithm, two independent reviewers examined all titles, and index terms and abstracts if available, and rated each paper as “potentially relevant” or “not relevant.” We rated the citation as being “potentially relevant” if there was any uncertainty regarding the inclusion of the paper. The raw agreement at this stage was 99.8% and the level of agreement beyond chance was 96.3% (κ = 96.3%; 95% confidence interval [CI], 94.8%–97.8%). Disagreements were resolved by discussion, and full-text articles were retrieved for all studies considered to be “potentially relevant.” Two reviewers then independently evaluated the full-text articles using the screening algorithm to determine whether the papers were “potentially relevant” or “not relevant,” and disagreements were resolved by discussion. The raw agreement at this stage was 98.8% and the level of agreement beyond chance was 82.7% (κ = 82.7%; 95% CI, 65.8%–99.6%). Next, all studies still considered to be “potentially relevant” were assessed independently by two reviewers to determine their inclusion status using the full set of inclusion criteria. The raw agreement at this stage was 100%.
Finally, all remaining studies were assessed independently by two reviewers for methodological quality. For this purpose, we used a 10-point rating scale derived from the metric used by Hunt et al. in a previous systematic review of electronic CDSSs.9 This scale assessed for 5 potential sources of bias, including the method of allocation to study groups, the unit of allocation, the presence of baseline differences between groups that were potentially linked to the study outcome, the type of outcome measure, and completeness of follow-up. Disagreements at this stage were resolved by discussion. There was 100% raw agreement as to whether a study possessed a total quality score of at least 5 points. Only studies meeting the minimum score of 5 points were included in the final analysis.
For each included study, two reviewers independently assessed for the presence or absence of a statistically and clinically significant desired change in clinician behavior. Effect size was considered as an alternative outcome measure, but the use of this metric was ruled out for two reasons. First, we anticipated that the use of effect size would have led to the exclusion of relevant trials, as studies oftentimes fail to report all of the statistical elements necessary for effect size reconstruction. Second, we felt that the use of effect size would be misleading given the significant heterogeneity among the outcome measures reported by the included studies.
In addition to the outcome, each comparison was assessed for general study characteristics, including the setting, subjects, and domain of decision support. Moreover, the two reviewers independently determined the presence or absence of 14 specific CDSS features, which included general system features, system-clinician interaction features, communication content features, and auxiliary features (Table 1). These 14 features were abstracted because they had been suggested as being important in the primary or secondary literature, and because it was felt that the presence or absence of these features could be abstracted reliably from most studies. Any disagreements at this stage were resolved by discussion. The raw agreement for the outcome measure was 100%. For the assessment of the 14 interventions features, the raw agreement ranged from 81.8% to 100%, and the level of agreement beyond chance ranged from 62.1% to 100% (κ = 62.1%–100%).
We sought to identify the CDSS features important in changing clinician behavior through the use of two approaches. As one approach, we conducted a meta-regression analysis, wherein the presence or absence of a statistically and clinically significant desired change in clinician behavior constituted the binary outcome variable, and where the presence or absence of the intervention features constituted binary explanatory variables. For this analysis, we only included studies in which the CPOE-based CDSS was compared against a control group that did not receive decision support. Moreover, given the limited sample size, we restricted the set of candidate explanatory variables to the three features that were most strongly associated with a significant outcome in our larger systematic review of CDSSs provided both within and outside of CPOE systems (noted with asterisks in Table 1).8 The logistic regression analysis was conducted using LogXact-5,10 a commercial statistical program that uses the exact permutation distribution of sufficient statistics to generate parameter estimates.11 Variables were included into the model using forward selection and a significance level of 0.05.
As a second approach for identifying critical CDSS features, we looked for direct exp erimental evidence supporting the importance of specific CDSS features. This assessment was done through the systematic identification of studies in which a given CPOE-based CDSS was directly compared against the same CDSS with one or more additional features.
Of 10,023 potentially relevant citations screened, 12 papers describing 11 studies met our inclusion criteria.12–23 Of the 11 included studies, 10 trials compared a CPOE-based CDSS against a control group that did not receive decision support (control-CDSS comparisons). One study compared a CPOE-based CDSS against the same system with additional features (CDSS-CDSS comparison).
Table 1 summarizes the distribution of the 14 abstracted CDSS features among the 10 control-CDSS comparisons. Of note, 2 of the features were not present in any of the CDSS interventions.
Furthermore, Table 2 provides a more detailed description of the interventions and outcomes of the 10 control-CDSS comparisons. Among these studies, 70% reported a statistically and clinically significant desired change in clinician behavior (the first 7 studies listed in Table 2), whereas 30% failed to find a significant improvement (the last 3 studies in Table 2). Among these 10 studies, decision support was provided most frequently for laboratory test ordering (70%), pharmacotherapy (60%), and radiology requisition (40%). Overall, 70% of the CPOE-based CDSSs supported decision making related to acute medical conditions, and 70% supported decision making related to chronic medical conditions.
As one approach for identifying important features of successful CPOE-based CDSS interventions, we conducted a meta-regression analysis on the 10 control-CDSS comparisons included in our study. As shown in Table 3, the meta-regression analysis revealed a significant association between the automatic provision of decision support and the finding of a statistically and clinically significant desired change in clinician behavior (adjusted odds ratio, 23.72; 95% confidence interval, 1.75-∞). Indeed, of the 10 studies described in Table 2, all 7 of the successful studies provided the decision support automatically, without the need for clinician initiative. On the other hand, this critical feature was absent from all 3 of the unsuccessful studies. For these studies, delivery of the decision support was dependent on the presence of user initiative.
As a second approach for identifying important CDSS features, we systematically searched for studies in which a CPOE-based CDSS intervention was compared directly against the same CDSS with additional features. As shown in Table 4, we identified one such study. In this study,23 the control clinicians received real-time critiques regarding the appropriateness of abdominal radiograph orders through a CPOE system, whereas the intervention clinicians received modified critiques which were worded more strongly, and which were accompanied by institutional evidence to support the critiques. However, this study failed to find a change in clinician behavior attributable to the provision of these additional CDSS features.
Through a systematic review of 11 randomized controlled trials, we have identified automatic provision of decision support as a critical feature of successful CPOE-based CDSS interventions. Indeed, meta-regression analysis revealed that automatic provision of decision support was strongly associated with a significant desired change in clinician behavior (adjusted odds ratio, 23.72; 95% confidence interval, 1.75-∞). This finding is consistent with the results from our systematic review of electronic as well as non-electronic CDSSs; even in this larger data set of over 75 randomized controlled trials, we have not identified a single CDSS intervention that resulted in a significant desired change in clinician behavior in the absence of the automatic provision of decision support.8 We speculate that this factor is critical because the demands on clinicians preclude them from actively seeking out advice from a clinical decision support system.
In interpreting the findings of this review, several limitations should be kept in mind. First, we used a binary outcome measure, instead of a continuous measure such as effect size. Because of this approach, we were unable to account for variations in the magnitude of effects. Second, the scope and power of our meta-regression analysis were limited by the relatively small sample size. Thus, we cannot rule out the potential importance of the 13 other CDSS features that we abstracted for this study. Finally, our analysis was limited to features that could be abstracted reliably. Thus, we were unable to assess the significance of several other potentially important features, including system response time, consistency of the system interface, and commitment from top levels of management.4, 6
Despite these limitations, we feel that this systematic review makes a significant contribution to our understanding of the specific features important in the success of CPOE-based CDSS interventions. We speculate that our findings will be of use to the designers and implementers of CPOE-based decision support systems as they leverage this technology to influence clinician behavior and improve patient care.
We thank Caitlin Houlihan for assisting with the study selection process and Dr. Andrew Balas for assisting with the study design. This research was supported by NIH grants T32-GM07171, R01-HS10472, and R03-HS10814.