We chose to study both on-screen and paper-based reminders provided that they were generated by electronic information specific to the individual in a health record and available at the clinical encounter. In contrast to the Shojania review, we chose the odds ratio technique to estimate effect size as we were interested in the relative likelihood of achieving the desired outcome in the presence of a reminder rather than the absolute change in outcome. This approach may be more appropriate where baseline activity varies significantly between different trial settings, as relative benefit tends to be more stable across risk groups than absolute benefit.12
We were also interested in detecting any variation in response according to clinical area and in changes in responsiveness over the past 40 years, during which the use of electronic records has become widespread. A review protocol was written but not published.
We systematically examined the literature from 1970 to February 2011 describing controlled trials of computer generated reminder interventions that draw on patient specific information and are available to clinicians during clinical encounters. We searched the following databases for relevant articles: PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC. The following search strategy (or adaptations of it) was used in each database:
- Reminder systems [MeSH] AND (Health OR Medic* OR Clinical) AND (Computer*
- [text word] OR Electronic* [text word])
We looked at reference lists of retrieved articles and past systematic reviews of similar interventions. We included non-randomised controlled trials, provided data collection from both arms was contemporaneous. We did not consider ‘before/after’ studies to be sufficiently valuable, given the potential for secular trends (including health policy changes) to confound the influence of the effect, and such studies were excluded.
Selection of Articles
The inclusion criteria were applied to each paper by two reviewers, with disagreements resolved by the third reviewer.
Extraction of Data
For each identified paper, two reviewers assessed methodological quality and extracted the outcome data using a formatted extraction sheet. Where necessary, study authors were contacted for clarification. We assessed risk of bias according to inadequate random sequence generation (at study level); and incomplete outcome data, selective reporting, and unit of analysis error (at the outcome level). The latter was used as a basis for a correction for clustering in the meta-analysis.
Changes in process or clinical outcome included rates of screening, vaccination, diagnostic tests, blood pressure measurement, blood pressure control, rate of venous thrombo-embolism, and measures of prescribing quality.
Odds ratios were derived for all binary outcomes where available. We used a random effects model with the Mantel–Haenszel method in RevMan version 5.2 to combine the data. Where multiple outcomes were reported, we derived a pooled outcome measure for each study. Heterogeneity was measured using the Tau2
is a measure of between study variance appropriate for a random effects meta-analysis.12
gives the proportion of the variability that is attributable to heterogeneity rather than chance.12
Trials of reminder interventions may be affected by ‘unit of analysis errors,10
through failure to correct for clustering. For instance, a trial may use as its outcome the proportion of patients achieving a clinical target at the end of the study, but it was the clinicians, clinical teams or clinics (not the patients individually) that had been randomised to use or not to use the reminders. If uncorrected, the precision of effect size measurement will be over-estimated by this error.
We tested the effect of introducing a correction factor where clustering had not been accounted for, using a recommended technique.12
An assumed intra-class correlation co-efficient of 0.03 was identified as appropriate from a published source.13
This was used to derive a design effect estimate for each study based on its mean cluster size, and the numerator and denominator values for each trial arm were divided by this factor. The pooled odds ratio was then re-estimated to account for clustering. Recognising the risk of applying a single ICC to many studies, we undertook an analysis to measure the sensitivity of the pooled odds ratio and its confidence interval to a range of assumed ICC values.
We also examined subgroups of reminder intervention according to pre-specified clinical areas and distinguished articles according to whether the trial was ‘explanatory’ or ‘pragmatic’ in design. ‘Explanatory’ studies were those in which the denominator was the reminder opportunity, i.e. the clinical encounter in which the reminder was triggered. The outcome was the proportion of all examples in which a clinician actually encountering a patient and presented with a reminder, responded to it. ‘Pragmatic’ studies used as their outcome the proportion of a population of patients whose clinicians were potentially exposed to a reminder intervention in whom the recommended care occurred. Some of the outcome denominator population might not have presented to the clinician during the study period, whilst others might have presented a number of times. Whilst some studies were difficult to categorise, we considered these groups to represent methodologically distinct designs worthy of separate analysis.
Finally, we sub-grouped studies according to the decade of publication, looking for a secular trend in the responsiveness of clinicians to such reminders, and assessed risk of publication bias using a funnel plot.