Of 2036 citations identified, we excluded 1662 at the initial stage of screening and an additional 374 after review of the full-text articles. A total of 28 articles (reporting 32 comparisons) met all of our inclusion criteria ().19–46
The full review has recently been published in The Cochrane Library.47
Figure 1 Results of literature search. *Excluded topics included expert systems (e.g., artificial intelligence or neural network applications) for facilitating diagnosis or for estimating prognosis; decision support not directly related to patient care (e.g., (more ...)
Of the 32 comparisons, 19 were in the United States and 8 occurred in inpatient settings (, located at the end of the article). Only six comparisons involved a quasi-randomized design, typically allocating intervention status on the basis of even or odd provider identification numbers. Twenty-six comparisons allocated intervention status to providers or provider groups (cluster trials); 12 of these comparisons accounted for clustering effects in the analysis. Seventeen trials reported a power calculation that included a target effect size. Twelve trials reported a target improvement in adherence to processes of care; 10 of these trials specified an absolute increase of at least 10% ().
Description of 28 studies (32 comparisons) included in a systematic review of the effects of point-of-care computer reminders on physician behaviour
displays the median improvements in adherence to processes of care for each included study (for details about the results from each study, see Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.090578/DC1
). Pooling data across studies (), we found that the median improvement in adherence associated with computer reminders was 4.2% (IQR 0.8%–18.8%). Prescribing behaviours improved by a median of 3.3% (IQR 0.5%–10.6% [21 trials]), adherence to target vaccinations by 3.8% (IQR 0.5%–6.6% [6 trials]) and test-ordering behaviours by 3.8% (IQR 0.4%–16.3% [13 trials]). also shows the results obtained when we used the best outcome from each study instead of the median improvement.
Figure 2 Median absolute improvements in adherence to processes of care between intervention and control groups in each study. Each study is represented by the median and interquartile range for its reported outcomes; studies with single data points reported only (more ...)
Improvements in adherence to processes of care across the 28 studies (32 comparisons) included in the review
Across eight comparisons that reported dichotomous clinical outcomes (e.g., achievement of target treatment goals), patients in the intervention groups experienced a median absolute improvement of 2.5% (IQR 1.3%–4.2%). For blood pressure control, the single most commonly reported outcome, patients in the intervention groups experienced a median reduction in systolic blood pressure of 1.0 mm Hg (IQR 2.3 mm Hg reduction to 2.0 mm Hg increase) and a median reduction in diastolic blood pressure of 0.2 mm Hg (IQR 0.8 mm Hg reduction to 1.0 mm Hg increase).
Study features and effect size
We found no significant correlation between effect size and the following study features: publication year, country (United States v. other), study design (randomized v. quasi-randomized) or sample size (whether calculated on the basis of patients or providers) (). We considered that studies with high adherence rates in control groups (a marker for baseline adherence) might achieve smaller improvements in care, because they had smaller opportunities for improvement. Surprisingly, studies with control-group adherence rates that were higher than the median across all studies showed larger effect sizes (). When we analyzed the potential impact of baseline adherence in various other ways (e.g., focusing on the highest and lowest quartiles of baseline adherence), we found no evidence that small improvements reflected high baseline quality of care.
Figure 3 Median effects for adherence to processes of care by study feature. *Kruskall–Wallis test; all other p values reflect Mann–Whitney test. †Quasi-RCT refers to randomized controlled trials in which intervention status was assigned (more ...)
We observed a trend toward larger improvements with inpatient interventions than with outpatient interventions (median 8.7% [IQR 2.7%–22.7%] v. 3.0% [IQR 0.6%–11.5%]; p = 0.34). All inpatient interventions occurred at two institutions that had well-developed, “homegrown” computerized systems for order entry by providers. Moreover, the recipients of computer reminders from these institutions consisted primarily of physician trainees.
Our grouping of studies on the basis of track records in clinical informatics did not result in significant differences, except that the studies from Brigham and Women’s Hospital in Boston, USA, reported a median improvement of 16.8% (IQR 8.7%–26.0%),26,31,37,40,46
compared with 3.0% (IQR 0.5%–11.5%) for studies from the other institutions (p
Features of computer reminders and effect size
We analyzed a number of reminder characteristics to look for associations with effect size (). Only the requirement for providers to enter a response to the reminder showed a trend toward larger improvements (median 12.9% [IQR 2.7%–22.7%] v. 2.7% [IQR 0.6%–5.6%] for no response required; p = 0.09). No trends toward larger effect sizes existed based on the type of targeted problem (underuse v. overuse of a targeted process of care), inclusion of patient-specific information, provision of an explanation for the alert, inclusion of a specific recommendation with the alert, development of the reminder by the study authors, or the type of system used to deliver the reminder (CPOE [computerized provider order entry] v. electronic medical records).
Figure 4 Median effects for adherence to processes of care by reminder feature. *Underuse = targeting improvements to increase the percentage of patients who receive targeted process of care (e.g., increasing the percentage of patients receiving the influenza (more ...)
Reminders that were “pushed” onto users (i.e., users automatically received the reminder) did not achieve larger effects than reminders that required users to perform some action to receive them (i.e., users had to “pull” the reminders); only 4 of the 32 comparisons involved “pull” reminders. A three-armed cluster randomized controlled trial of reminders for screening and treatment of hyperlipidemia45
directly compared these two modes of delivering reminders. Patients cared for at practices randomly assigned to deliver automatic alerts were more likely to undergo testing for hyperlidemia and receive treatment than were patients at clinics where reminders were delivered to clinicians only “on demand.”
We re-analyzed the potential predictors of effect size (study features and characteristics of reminders) using a variety of choices for the representative outcome from each study, including the outcome with the middle value (rather than a calculated median) and the best outcome (the outcome associated with the largest improvement in adherence to the process). None of these analyses substantially altered the main findings.