We reviewed 65 RCTs of CCDSSs for drug therapy management reported over a 34-year span. Most trials measured process of care outcomes and results supported the use of CCDSSs to improve these outcomes in a majority of cases (improvement was based on at least 50% of the relevant study outcomes being statistically significantly positive). However, while nearly one-half of (29 studies) included studies measured a patient outcome, only a small proportion demonstrated any direct benefit to patients. While improvement in process outcomes could lead to benefits for patients, no consistent link was observed here. In the absence of data needed for an economic analysis, improved process of care measures alone are not sufficient to recommend adoption of these systems. The success rates we found for processs of care (64%) and clinical outcome measures (21%) are similar to those in our previous review [2
] and also a recent umbrella review of systematic reviews of computerized decision support (57% and 30% respectively) by Jaspers et al.
Several possible predictors of CCDSS success were examined. In most cases, these a priori
factors did not explain success or failure across included studies. Our previous review [2
] concluded that successful trials of CCDSS were more likely to have been conducted by the developers of the system under study. In our current review, no such association was noted. Previously, a significant trend towards increased study quality over time was noted, but not replicated in this update, and we attribute this to a more restrictive inclusion criterion (randomized controlled trials). Counter to our expectations, we found that integration of CCDSSs with EMRs and use in an academic setting was associated with CCDSS failure. This trend was not statistically significant when tested using multi-variate techniques and so we are unable to determine whether this finding represents a true association or is better explained by the lack of power in our multi-variate analysis. We report these findings as hypothesis generating only and suggest they be examined in future.
Compared with the review of Kawamoto et al.
], we did not find that automatic provision of advice as part of the existing clinical workflow predicted CCDSS success. Because both the current analysis and that of Kawamoto were underpowered to detect such associations, we have refrained from drawing any conclusions in this regard.
Prospective data on the possible harms of CCDSSs are needed to facilitate informed adoption decisions. Only two trials quantitatively reported on harm from CCDSSs [36
] with one trial ending early due to increased risk of harm with the CCDSS. We suggest this absence of evidence of harm should not be taken as proof that CCDSSs are safe to employ for drug management in patient care.
Strengths and limitations of review
The results of our review should be interpreted with consideration of methodological strengths and limitations, including steps taken to mitigate the risk of bias. We based our review on the strongest studies available, RCTs. Reviews are necessarily retrospective and we employed multiple methods to limit the introduction of bias, including: duplicate study eligibility assessment, duplicate data abstraction, solicitation of study author feedback on abstracted data, and objective selection of outcomes used to determine improvement. We cannot exclude the possibility that a different method of selecting outcomes from each study to measure improvement could lead to different results, although sensitivity analyses did not suggest this to be the case. Several pre-specified analyses of possible predictors of system success were conducted. Several analyses demonstrated statistically significant results using univariate techniques that were not substantiated using a multi-variate model. Therefore, the few associations we reported between possible predictors of success and improved outcomes with CCDSS should be interpreted with caution.
We have relied upon vote counting as our method of obtaining an estimate of how often CCDSS for drug therapy management improve process or patient outcomes. Significant limitations to this approach as described by Hedges [93
] include a tendency to inflate type II error and inadequate incorporation of the effect of unequal study sizes in overall results. The heterogeneity between studies included in our review precluded the use of more robust combination techniques. Formal assessment for publication bias using funnel plots was not possible with the vote-counting technique.
The effectiveness of any CCDSS will be determined in part by the efficacy of the underlying action suggested by the system. Where no benefit was detected with a particular CCDSS, we cannot exclude the possibility that the negative finding is due to a lack of efficacy of the intervention suggested by the system. Measurement of the concordance between decision advice given and followed would be a useful measure to address this issue. These outcomes were included in our analyses of process of care outcomes. It does not necessarily follow, however, that an effective CCDSS that recommends the appropriate prescription of an efficacious intervention will necessarily improve patient care. A multitude of intervening factors (e.g. patient non- or over-adherence or new errors introduced by CCDSS) may mitigate (or exaggerate) estimates of CCDSS effectiveness.
Finally, the systems reviewed constitute a heterogeneous group with differing functionality and clinical intent. While we have attempted to usefully divide the systems for the reader, we acknowledge other divisions were possible.
Implications for practice and research
Because CCDSSs have not been shown to reliably and positively impact patients, and in the absence of useful data on potential harms, costs, and clinician impacts, we cannot recommend the general adoption of CCDSSs for drug therapy management. It is possible that these systems are still evolving and success will improve with time. Clearly further innovation is needed if these systems are to be dependably useful in clinical practice. Rigorous trials of these innovations will be necessary, and we suggest that future research explicitly address patient outcomes, including potential harms, and costs and adverse clinician impacts of CCDSSs. Given the availability of effective non-computerized approaches for promoting safe and effective medication use [5
], future studies may wish to incorporate these interventions as active comparators to CCDSSs.