Answers to the study questions
According to the 69 participating CPOE experts the top-five useful context factors for prioritizing alerts are (in descending order of usefulness): (1) severity of the effect of the ADE the alert refers to; (2) clinical status of the patient; (3) probability of occurrence of the ADE the alert refers to; (4) risk factors of the patient; and (5) strength of evidence on which the alert is built (see for details).
All of the experts agreed on the potential to prevent ADE by using CPOE systems with active alerting (see ). The experts estimated that on average every fourth preventable ADE can be avoided by integrating an active alerting module or a proactive prescription simulation module into the hospital information system architecture (see ).
Strengths and weaknesses of the study
Only a third of all the invited experts completed both rounds. This can be seen as a potential limitation. On the other hand, we did not use a convenience sample of easily available experts (eg, only experts personally known to us), but contacted all researchers worldwide who we identified as first authors of CPOE papers. This method of identifying experts can be viewed as a strength, because it reduced potential bias in the selection of experts. Furthermore, in comparison with other Delphi studies with less than 30 participants,26–28
69 participants seems quite high. The identification method and the number of participants should have yielded reasonable diversity in professional points of view.
The final expert panel consisted of CPOE experts from 15 countries spread over five continents, with approximately half from the USA and one-third from Europe. This mirrors the distribution of scientific papers on CPOE systems in the literature. Two-third of participants self-assessed their level of expertise with CPOE systems to be advanced, which is an indication that we did reach CPOE experts.
The sample was mostly restricted to experts who are publishing in scientific journals. This means that the results represent the point of view of researchers. The perspectives of CPOE users, CPOE developers and CPOE implementers may be different, and they were not investigated in this study.
The experts had to read and judge a list of 20 suggested factors. This could have contributed to a ‘serial position effect’, meaning that the first factors listed are treated differently than factors listed further down (eg, when participants are losing concentration). To minimize this effect, we chose an automatic random order of factors for each expert in the survey.
Another possible source of bias is the way of describing the factors. For each factor, a short definition was given together with a short prime example (see supplementary appendices 1and 2, available online only). It may be that the experts only judged a factor based on the one example given without considering other situations. We cannot assess the impact of this source of bias; it seems clear, however, that examples are needed to explain the meaning of the factors and that the number of examples that can be given is limited.
Both non-weighted and weighted frequencies showed the same top five factors. We view this as an indication for the stability of the results. Some experts expressed concerns about the validity of their estimates about ADE reductions. This, however, is the task of a Delphi study: to gather estimates of unsure future developments.
The context factors were developed for ‘normal’ inpatient settings; usefulness of context factors may be different in intensive or outpatient units.
Results in relation to other studies
Study objective 1: usefulness of different context factors
There are reviews on the effectiveness of different types of alerts (eg, drug–drug-interaction alerts and drug–laboratory alerts),29
studies on how alerts are handled by clinicians30
and studies on frequencies and reasons for alert overriding.13
Several of these studies provide specific suggestions on how to improve alert presentation in order to improve alert effectiveness and reduce alert fatigue (eg, see next paragraphs). To our knowledge, however, there are no studies that first define and then compare context factors in a controlled trial (eg, comparing the usefulness of ‘severity’ vs ‘professional experience’). Obviously, comparing all factors with each other in individual simulations or field studies would require quite a number of controlled studies. We therefore chose the Delphi study to compare all 20 factors with each other in a unified way to determine which factors may be most beneficial to be studied further in controlled trials.
Our list of 20 factors was identified by a literature review, so each factor has been discussed in the literature. However, different points of view were often expressed on the usefulness of the factors. For example, there is some controversy in the literature about the ‘severity of the effect of the ADE the alert refers to’, which was the factor given the highest ranking by experts in our study. Some authors such as Kuperman et al10
(p. 37) and Weingart et al31
(p. 2625) discussed the usefulness of this factor; others such as van der Sijs et al32
(p. 446) and Tamblyn et al33
(p. 437) viewed it differently. This controversy could depend on one's interpretation of ‘severity’. Physicians may rate the severity of an alert differently depending on whether there are organizational or clinical rules that could prevent the manifestation of the related ADE (eg, by monitoring certain laboratory values) (p. 446).32
The clinical status of the patients, ranked highly in our survey, has also been considered important by other researchers such as van der Sijs et al12
(p. 144). It seems quite clear that the inclusion of more clinical parameters such as laboratory values can help to tailor alerts better.
Another highly ranked factor, ‘strength of evidence on which the alert is built’ is also mentioned by other authors such as Kuperman et al10
(p. 37). It is not surprising that our participating experts rate this as an important factor, given their research background.
‘Probability of occurrence of the ADE the alert refers to’ and ‘risk factors of the patient’ were highly ranked by our experts, but are mentioned seldom in the literature.
An initially unexpected result was the poor ranking achieved by the context factor ‘professional experience of the user’. This factor is mentioned quite frequently in the literature, for example, by van der Sijs et al
and is often mentioned as a key example to describe the contextualization of drug safety alerts (eg, alerts for senior physicians vs alerts for junior physicians). In our survey, it is ranked lowest. Our sample mostly consisted of CPOE researchers, not clinical practitioners outside a research context; this might explain the low ranking of this factor.
Study objective 2: impact of different ways of delivering alert information on ADE rates
Our experts estimated that, depending on the tools used, between 10% and 25% of ADE could be prevented. It must be noted that these numbers are subjective judgments, as no empirical evidence is available for most of the tools as yet. As shows, experts often gave comparable estimates; we see this as an indicator for the potential validity of these estimates.
In our study, the highest estimates of approximately a 25% ADE reduction were given for active alerting modules. This is not surprising as sufficient evidence for their benefit in reducing medication errors and ADE rates is available.8
However, all studies included in the review by Ammenwerth et al8
compared the intervention with a paper-based control situation; here, ADE risk reductions of 13–84% were found. We did not find studies that assessed ADE risk reductions comparing computer-based ordering with and without an active alerting module. We can now provide a first estimate of 25%.
External drug information is also a frequently used tool in many hospitals—Sharp et al34
identified over 30 major drug information resources. While evaluations of the quality of the content of these services and of their usage already exist,17
we are not aware of any systematic evaluations of the impact of these tools on patient outcomes. Our experts estimated an ADE risk reduction of less than 10%. Drug information services have the drawback of not using individual patient data. Moreover, these tools are typically used voluntarily. This all may reduce their impact on patient safety.
Passive alerting modules, proactive prescription simulation modules and ADE epidemiology information are new tools just being developed in the PSIP project.15
No empirical evaluation results are available yet. Our experts judged their potential impact to be between 10% and 25%. The high estimates for the simulation modules point to the fact that this is seen as an interesting support within the ordering process.
The experts estimated that a patient medication module may lead to an ADE reduction of approximately 15%. General drug information services available to patients already exist, for example,35
but the concept of the patient portal envisions an extended tool that offers personal, tailored, drug-related information based on a patient's actual prescriptions. The participating experts obviously support this idea by estimating an ADE risk reduction of approximately 15%.
Meaning and generalizability of the results
To our knowledge, the ‘context model’ used in this study is the first attempt to systematically describe and structure information on the clinical and user context that can be used to optimize alert prioritization and alert presentation. It may help to improve the ‘alert logistics’ in clinical settings and therefore support a reduction of over-alerting and resulting alert fatigue.
Unanswered and new questions
Although some of the methods to reduce ADE presented to experts in this study are relatively new, many of them are already available or under development. Future research will have to show in empirical evaluation studies whether the estimated benefits can be obtained in real practice.
The set of context factors for alert prioritization recommended by the experts who participated in this study can be seen as a starting point to develop more effective alert mechanisms within CPOE systems. Systematic trials should be conducted to determine whether alert overload can be reduced without reducing the sensitivity of the alerts too much.
The experts proposed some additional context factors, such as whether a prescription is based on a clinical protocol; these ideas need to be investigated further. In addition, the list of factors should also be complemented and validated from the point of view of practising clinical users, which may be different from those of experts.
In this study we did not investigate the information logistics needed to provide CPOE systems with the necessary information (such as professional experience and workload of the user, severity of the expected effect or clinical status of the patient). Obviously, some of this information will have to come from clinical or administrative systems, others from drug information sources. In addition, some context factors are quite vague and general (such as ‘clinical status of a patient’) and need to be defined precisely. Research on how to provide this information through adequate information systems architectures and interfaces is ongoing work within the PSIP project. We also did not ask the experts about the feasibility of the proposed tools or the way these tools can be integrated into the clinical workflow.
This study focused on each factor individually. It has not yet been investigated how the most important factors can and should be combined in a clinical situation. For this, additional studies are needed.
System developers and researchers should collaborate to develop these tools and to implement and evaluate them in field studies. Some of this is ongoing research within the PSIP project. Future research will show whether the estimated ADE reduction can be achieved and at what cost.