|Home | About | Journals | Submit | Contact Us | Français|
To synthesise current evidence for the influence on clinical behaviour of patient-specific electronically generated reminders available at the time of the clinical encounter.
PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC were searched for relevant articles.
We included controlled trials of reminder interventions if the intervention was: directed at clinician behaviour; available during the clinical encounter; computer generated (including computer generated paper-based reminders); and generated by patient-specific (rather than condition specific or drug specific) data.
Systematic review and meta-analysis of controlled trials published since 1970. A random effects model was used to derive a pooled odds ratio for adherence to recommended care or achievement of target outcome. Subgroups were examined based on area of care and study design. Odds ratios were derived for each sub-group. We examined the designs, settings and other features of reminders looking for factors associated with a consistent effect.
Altogether, 42 papers met the inclusion criteria. The studies were of variable quality and some were affected by unit of analysis errors due to a failure to account for clustering. An overall odds ratio of 1.79 [95% confidence interval 1.56, 2.05] in favour of reminders was derived. Heterogeneity was high and factors predicting effect size were difficult to identify.
Methodological diversity added to statistical heterogeneity as an obstacle to meta-analysis. The quality of included studies was variable and in some reports procedural details were lacking.
The analysis suggests a moderate effect of electronically generated, individually tailored reminders on clinician behaviour during the clinical encounter. Future research should concentrate on identifying the features of reminder interventions most likely to result in the target behaviour.
Computer generated reminder systems are commonly used to support routine health care. They utilise electronic data to identify clinical errors and opportunities for screening, preventive interventions, improved prescribing, and both diagnostic and monitoring tests. Previous studies have found that the response of clinicians to such reminders is variable, and a number of reviews have described existing tools, where possible measured their impact, and in some cases attempted to identify factors influencing effect size.1–11 Reminder systems are diverse in their design. Some are used to support specific clinical areas of care (e.g. diabetes), presenting current recommendations or evidence, and do not require patient specific data. Others are triggered simply by an attempt to prescribe a specific drug therapy, for instance reminding the prescriber of lithium that blood monitoring is required. Shojania et al. studied the impact of ‘on-screen’ reminders as a Cochrane systematic review,10 and excluded computer generated paper-based reminders and email alerts occurring outside clinical encounters. They hypothesised that this approach would identify a more consistent effect, in contrast to the variable results reported in previous reviews. This group derived a median absolute change in adherence of 4.2% with IQR 0.8-18.8%, suggesting significant variation in response, and factors predicting effect size were difficult to identify.
A subset of reminder system draws on patient specific data in the electronic record and is therefore tailored to the individual. For this review we were interested specifically in individually tailored reminders and in the impact of these tools on decision making. We concurred with Shojania et al. over the importance of the ‘point of care’ setting, but hypothesised that tailored reminders might provide a more consistently positive effect. Although the reminders that we studied were exclusively clinician directed, individual tailoring might conceivably carry greater impact, as the resulting behaviour often requires patient involvement for completion (e.g. uptake of screening).
We chose to study both on-screen and paper-based reminders provided that they were generated by electronic information specific to the individual in a health record and available at the clinical encounter. In contrast to the Shojania review, we chose the odds ratio technique to estimate effect size as we were interested in the relative likelihood of achieving the desired outcome in the presence of a reminder rather than the absolute change in outcome. This approach may be more appropriate where baseline activity varies significantly between different trial settings, as relative benefit tends to be more stable across risk groups than absolute benefit.12 We were also interested in detecting any variation in response according to clinical area and in changes in responsiveness over the past 40 years, during which the use of electronic records has become widespread. A review protocol was written but not published.
We systematically examined the literature from 1970 to February 2011 describing controlled trials of computer generated reminder interventions that draw on patient specific information and are available to clinicians during clinical encounters. We searched the following databases for relevant articles: PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC. The following search strategy (or adaptations of it) was used in each database:
We looked at reference lists of retrieved articles and past systematic reviews of similar interventions. We included non-randomised controlled trials, provided data collection from both arms was contemporaneous. We did not consider ‘before/after’ studies to be sufficiently valuable, given the potential for secular trends (including health policy changes) to confound the influence of the effect, and such studies were excluded.
The inclusion criteria were applied to each paper by two reviewers, with disagreements resolved by the third reviewer.
For each identified paper, two reviewers assessed methodological quality and extracted the outcome data using a formatted extraction sheet. Where necessary, study authors were contacted for clarification. We assessed risk of bias according to inadequate random sequence generation (at study level); and incomplete outcome data, selective reporting, and unit of analysis error (at the outcome level). The latter was used as a basis for a correction for clustering in the meta-analysis.
Changes in process or clinical outcome included rates of screening, vaccination, diagnostic tests, blood pressure measurement, blood pressure control, rate of venous thrombo-embolism, and measures of prescribing quality.
Odds ratios were derived for all binary outcomes where available. We used a random effects model with the Mantel–Haenszel method in RevMan version 5.2 to combine the data. Where multiple outcomes were reported, we derived a pooled outcome measure for each study. Heterogeneity was measured using the Tau2 and I2 statistics. Tau2 is a measure of between study variance appropriate for a random effects meta-analysis.12 I2 gives the proportion of the variability that is attributable to heterogeneity rather than chance.12
Trials of reminder interventions may be affected by ‘unit of analysis errors,10 through failure to correct for clustering. For instance, a trial may use as its outcome the proportion of patients achieving a clinical target at the end of the study, but it was the clinicians, clinical teams or clinics (not the patients individually) that had been randomised to use or not to use the reminders. If uncorrected, the precision of effect size measurement will be over-estimated by this error.
We tested the effect of introducing a correction factor where clustering had not been accounted for, using a recommended technique.12 An assumed intra-class correlation co-efficient of 0.03 was identified as appropriate from a published source.13 This was used to derive a design effect estimate for each study based on its mean cluster size, and the numerator and denominator values for each trial arm were divided by this factor. The pooled odds ratio was then re-estimated to account for clustering. Recognising the risk of applying a single ICC to many studies, we undertook an analysis to measure the sensitivity of the pooled odds ratio and its confidence interval to a range of assumed ICC values.
We also examined subgroups of reminder intervention according to pre-specified clinical areas and distinguished articles according to whether the trial was ‘explanatory’ or ‘pragmatic’ in design. ‘Explanatory’ studies were those in which the denominator was the reminder opportunity, i.e. the clinical encounter in which the reminder was triggered. The outcome was the proportion of all examples in which a clinician actually encountering a patient and presented with a reminder, responded to it. ‘Pragmatic’ studies used as their outcome the proportion of a population of patients whose clinicians were potentially exposed to a reminder intervention in whom the recommended care occurred. Some of the outcome denominator population might not have presented to the clinician during the study period, whilst others might have presented a number of times. Whilst some studies were difficult to categorise, we considered these groups to represent methodologically distinct designs worthy of separate analysis.
Finally, we sub-grouped studies according to the decade of publication, looking for a secular trend in the responsiveness of clinicians to such reminders, and assessed risk of publication bias using a funnel plot.
We initially identified 683 articles following removal of duplicates. Abstracts were examined to remove obviously irrelevant papers, leaving 234 for full text examination. Of these, 192 articles were excluded by at least two reviewers (Fig. 1). This left 42 trial reports in the final group.14–55 Forty-one of these used binary outcomes. The other46 used length of hospital stay. Two papers reported clinical outcomes (control of blood pressure24 and rate of venous thrombo-embolism)29 and all the rest involved process outcomes. One paper19 reported three different intervention arms and one control arm. This study was entered as three separate comparisons and the numbers in the control arm were divided by three to avoid over-weighting. A further two papers47,53 reported two equally important forms of reminder that were both included as separate comparisons. Where possible we aggregated separately reported subgroups of outcome within the same trial to provide an estimate of overall effect. For instance, a single reminder intervention might promote screening tests, clinical measurement and immunisations, with each outcome reported separately. One paper21 reported multiple outcomes with no primary outcome and was not included in the meta-analysis as it was not possible using this method to aggregate the outcome data from this paper reliably. There were therefore 44 comparisons using a binary outcome available for the meta-analysis. There were examples in which the desired effect of the reminder was to reduce rather than increase the outcome measure15,29,32,43,47,48,50. In such cases we used the method described by Shojania et al.10 to impute a corrected numerator in order that the effect was measured in the same direction as for the other studies. There was only one example49 of a trial that was controlled but not randomised.
For the 44 binary outcome comparisons an overall odds ratio of 1.79 [95% confidence interval 1.56, 2.05] was derived in favour of the reminders. Heterogeneity was high presumably due to clinical and methodological diversity, with an overall Tau2=0.18, Chi2=1530.40, p<0.00001, I2=97%. The one study using a continuous outcome46 reported a non-significant difference in length of hospital stay. The study that was excluded on the basis of multiple outcomes reported no effect of the reminder system on clinical care.21 For our included studies, 32 out of 44 comparisons showed a significant positive effect and 11 showed no significant effect. One study48 appeared to show a significant negative effect but this was dependent on the definitions of intervention and control in a study comparing two different reminder systems.
To reduce clinical diversity we attempted subgroup analyses based on area of care (although there was much overlap). There was evidence (of borderline significance, Chi2=11.47, p=0.04) of subgroup differences in effect size. Odds ratios ranged from 1.24 [95%CI 1.01–1.52] for condition specific but multiple reminders to 4.69 [95%CI 1.25–17.53] for vaccination reminders (Fig. 2). The condition specific but multiple reminders subgroup had a relatively low Tau2 score of 0.03 with Chi2=7.84, p=0.05. The odds ratios in favour of the intervention for the explanatory and pragmatic sub-groups were 1.90 and 1.71, respectively, and there was no significant improvement in heterogeneity scores.
There was no evidence that odds ratios were different between the 1980s, 1990s and 2000s, and only one study from the 1970s was included.
Characteristics of the reminder interventions were examined to look for factors likely to influence the effect size, including clinical priority, remunerative factors, and ease of use. These are explored in the Discussion section below (Table 1).
For many studies procedural details such as randomisation techniques were unreported. Trials of reminder interventions sometimes randomise at the level of the clinician or clinical team, but analyse using patient level outcome data. Some form of unit of analysis issue potentially affected 28 studies14,18–22,24–28,31,32,35,37–39,41–45,47–49,51–53 and 32 comparisons. In sixteen cases14,19–21,24,26,31,37–39,45,47,48,51–53 this was discussed and corrective action taken to adjust confidence intervals or p values appropriately. However, the raw data that we extracted had not undergone this correction and we therefore applied our own adjustment as described above. For the 32 comparisons affected, the initial odds ratio in favour of reminders was 1.87 [95% CI 1.54, 2.28]. Following our adjustments the odds ratio for these studies had changed to 1.90 and the confidence interval had widened slightly to [1.54, 2.33]. There was no change in the overall pooled odds ratio of all studies combined (1.79, [1.58, 2.02]). The results of our adjustment for clustering are given in Fig. 3. Table 2 gives the results based on a range of assumed ICC values, suggesting that the analysis was not sensitive to this assumed value over a 100 fold scale. Figure 4 shows risk of bias tables (a) for each study (b) aggregated.
We derived a funnel plot which was broadly symmetrical with no evidence of substantial publication bias.
The majority of interventions in our review produced significant changes in measured outcomes, but there were numerous examples of no effect and it appears that reminders are often ignored. There is no evidence that such tools were more effective in the 2000s than in the 1980s or 1990s, and our effect size estimate is very similar to a previously published value from 19967, albeit using different inclusion criteria.
Characteristics of individual studies are given in Table 1. We examined these to see whether specific features associated with a more consistent effect could be identified. Kawamoto et al.4 have reported four features believed to be relevant in clinical decision support systems: automatic provision of decision support as part of clinical workflow; provisions of recommendations rather than just assessments; provision of decision support at the time and location of decision making; and computer-based decision support. Whilst all our trials involved computer generated reminders, some of these were paper-based. We looked at whether this feature influenced success, and also considered a number of other potentially relevant issues suggested by other investigators.56–62 These included clinical priority and relevance, cost-effectiveness considerations, accessibility, intrusiveness, and the time required to respond.
Computer generated but paper-based reminders were involved in 12 of our 44 comparisons.16–19,33–36,40–42,54 The remainder were displayed either exclusively on a computer screen or in both formats. There was no significant difference in the odds ratios obtained between these subgroups.
It is difficult to judge which issues physicians are likely to consider most important clinically. Vaccination reminders might in most situations be considered less urgent than immediate prescribing safety or laboratory monitoring issues, but in fact were associated with a stronger effect, albeit based on a small number of studies. However the one trial reporting a significantly positive result for a clinical (rather than a process) outcome involved the prevention of venous thrombo-embolism in hospitalised patients identified and flagged as ‘at risk’ of this serious condition.29
None of our included trials specifically reported ‘payment by result’ as a direct consequence of responding to a reminder, but this may have been an unreported factor in settings where remuneration is partly based on quality or efficiency of care. In some cases the electronic record itself had been established at least partly for the purpose of gathering billing information. Shea et al.46 mention financial pressures relevant to their length of hospital stay outcome. Others mentioned the health economic benefits of cost-effective monitoring and prescribing, promoted by reminders, and Tierney 198750 included charges per visit as a secondary outcome.
It is difficult to interpret from a published study exactly how much time clinicians had available and how onerous the recommended action might have been. In a large study based in Canada, the reminder requiring activation by the clinician was in fact more effective than the one appearing spontaneously.48 Van Wyk et al. arrived at the opposite conclusion in their trial.53 They included an ‘on-demand’ arm that required the user to actively seek the recommendation by accessing an overview screen in the patient’s record. In this arm responsiveness was significantly lower than in the ‘alerting’ arm which required no positive action. Eccles et al. reported a similar finding that highlights the difficulties in successfully embedding the reminder into the workflow.21 The negative results in this study were attributed by the authors to low usage of the system, despite its integration into the clinical software.
Other interesting phenomena were reported in the studies we examined. Chambers et al.18 included an arm in which the reminders only appeared ‘sometimes’ (in addition to the ‘always reminded’ arm whose data were used in our meta-analysis). The clinicians reminded ‘sometimes’ had a lower adherence than those reminded ‘never’ (i.e. controls), suggesting that they had become dependent on the alerts to remember to arrange influenza immunisation for eligible patients.
Our study is limited partly due to heterogeneity of effect sizes and by difficulties in synthesising data from diverse trial designs. The effect under investigation is likely to depend on the health care setting, the detailed design of the reminder, and the priorities of both clinician and patient. Attempts to substantially reduce heterogeneity through subgroup analyses were unsuccessful but our measurement of effect size is nevertheless meaningful. We focussed specifically on ‘reminder’ interventions and may have missed some studies of more generalised decision support systems in which reminders were a minor element. A further limitation is the lack of detail given in some trial reports over how the system actually operated in practice and what was required of the user in practical terms.
Our review provides data specific to tailored reminders available during clinical encounters, and is the only recently published example of a meta-analysis using a relative (odds ratio) technique rather than an absolute change method in this area of care. This technique provides a more consistent measure of effect across diverse studies, but is more sensitive to outliers than the median absolute benefit technique.11 Trial reports accounted for clustering effects in some cases, risking unit of analysis errors in others. We applied our own correction for clustering in the analysis of the raw trial data to estimate the effect of clustering on our pooled odds ratio.
Most individual reminder trials are designed to find out whether a system works rather than why it works. Mayo-Smith and Agrawal used a mixed method to investigate this area, conducting an observational study of reminder completion rates followed by a questionnaire survey of users.63 They also reviewed literature reporting this issue specifically, and included studies using qualitative methods. They reported a number of possible features of reminders, settings and users that appear to facilitate or obstruct response, and such clues might become the basis for a more extensive programme of investigation.
Individually tailored, computer generated reminders generally produce positive but modest effects on clinicians’ behaviour. Such interventions are inexpensive, widely available, and offer the potential both to improve clinical care and to impact health outcomes. There is now an extensive literature demonstrating these benefits. The specific features of such tools and the particular settings that determine their effect are still unclear but should become the focus of future research in this area.
We thank Dr Simon Gates for analytical advice and Samantha Johnson for assistance with database searching and retrieval of papers. We also thank the authors of studies for which clarification was required for providing useful information.
The authors declare that they do not have a conflict of interest.
Funded through internal sources. No external funding.