How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study

doi:10.1136/amiajnl-2010-000006

J Am Med Inform Assoc. 2011 Nov-Dec; 18(6): 760–766.

Published online 2011 June 22. doi: 10.1136/amiajnl-2010-000006

PMCID: PMC3197982

How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study

Daniel Riedmann, Martin Jung, Werner O Hackl, and Elske Ammenwerth

Institute for Medical Informatics, UMIT-University for Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria

Corresponding author.

Correspondence to Dr Elske Ammenwerth, Institute for Medical Informatics, Eduard Wallnöfer-Zentrum I, Hall in Tirol 6060, Austria; Email: elske.ammenwerth/at/umit.at

Received November 12, 2010; Accepted June 1, 2011.

This article has been cited by other articles in PMC.

Abstract

Objectives

To determine what information can be helpful in prioritizing and presenting medication alerts according to the context of the clinical situation. To assess the usefulness of different ways of delivering medication alerts to the user.

Design

An international Delphi study with two quantitative rounds. 69 researchers with expertise in computerized physician order entry (CPOE) systems were asked to estimate the usefulness of 20 possible context factors, and to assess the potential impact of six innovative ways of delivering alert information on adverse drug event (ADE) rates.

Results

Participants identified the following top five context information items (in descending order of usefulness): (1) severity of the effect of the ADE the alert refers to; (2) clinical status of the patient; (3) probability of occurrence of the ADE the alert refers to; (4) risk factors of the patient; and (5) strength of evidence on which the alert is built. The ways of delivering alert information with the highest estimated ADE reduction potential are active alerting, proactive prescription simulation and a patient medication module that gives patient-oriented alert information.

Limitations

Most participants had a research-oriented focus; therefore the results may not reflect the opinions of CPOE users or CPOE implementers.

Conclusion

The study results may provide CPOE system developers and healthcare institutions with information on how to design more effective alert mechanisms.

Keywords: classic experimental and quasi-experimental study methods (lab and field), demonstrating return on IT investment, human–computer interaction and human-centered computing, improving healthcare workflow and process efficiency, measuring/improving patient safety and reducing medical errors, medical informatics, nursing informatics, personal health records and self-care systems, qualitative/ethnographic field study

Patient safety is a serious public health issue.¹ ² The Council of Europe defines patient safety as ‘the identification, analysis and management of patient-related risks and incidents, in order to make patient care safer and minimize harm to patients’ (p. 8).³ Estimates from the WHO show that one out of 10 patients is harmed while receiving hospital care.²

Medication-related events are among the most common adverse events.⁴ These adverse drug events (ADE) are defined as ‘any injury occurring during the patient's drug therapy and resulting either from appropriate care or from unsuitable or suboptimal care' (p. 1).³ ADE resulting from a medication error are considered to be preventable ADE.³ ⁵

It is estimated that in the USA, over 770 000 people are injured annually or die in hospitals due to ADE.⁶ A review article by von Laue et al⁷ in 2003 found ADE rates to range from 0.7% to 6.5% per admission of hospitalized patients.

The use of computerized physician order entry (CPOE) systems can reduce medication errors and ADE.⁸ ⁹ CPOE systems can be equipped with different levels of clinical decision support.¹⁰ However, drug safety alerts generated by CPOE systems often show low specificity due to too many false-positive warnings.¹¹

Overriding drug safety alerts in CPOE systems is very common and occurs in 49–96% of cases.¹² Constant over-alerting may cause alert fatigue. Alert fatigue is described ‘as the mental state that is the result of too many alerts consuming time and mental energy, which can cause important alerts to be ignored along with clinically unimportant ones’ (p. 139).¹²

Some proposals have recently been made to prioritize and differently present alerts according to the respective patient, for example, based on his age, gender, weight, clinical laboratory values or allergies in order to reduce over-alerting.¹¹ ¹² Another possibility is to tailor alerts to the level of professional experience of the user¹² ¹³—for example, a senior cardiologist may want to see fewer or different alerts than a junior cardiologist. It seems to be helpful if a CPOE system were aware of this kind of patient and user context. In computer science, ‘context’ refers to the idea of systems sensing and reacting based on their environment, with the context being defined as ‘any information that can be used to characterize the situation of an entity’ (p. 3).¹⁴

At the moment, however, it is unclear what types of context information a CPOE system should take into account in order to prioritize and differently format a drug-related alert before presenting it to the user.

This study is part of the European patient safety through intelligent procedures in medication (PSIP) project.¹⁵ This project aims at developing innovative ways to deliver alert information in order to improve medication safety, addressing different timings (during prescription, dispensing, administration …) and different receivers (physician, nurse, patient …). The tools developed include a prescription simulation tool, a passive alerting tool, an active alerting tool, a patient medication module and an ADE statistics report resulting from data mining of clinical data. These tools represent interesting ideas for delivering alert information in different ways. It is unclear, however, whether these new ways of delivering alert information can help to reduce ADE rates and, if so, how much reduction of ADE can be expected.

The objectives of this study were to obtain expert opinion on two questions:

What information about the clinical and user context can help to prioritize and differently present medication alerts?
Can different ways of delivering alert information, addressing different timings and different receivers, help to reduce ADE rates?

Background

A context model for CPOE alerts

Within the PSIP project, a context model was created that was used for this study.¹⁶ The context model was developed based on a literature review and expert interviews. Twenty types of context information were identified that may be useful to define the clinical and user context, to prioritize medication alerts and to present them differently to the user.

To develop this context model, we identified CPOE papers that deal with issues of alert presentation. First, 10 major health informatics journals were hand-searched by two researchers. Sixty-one papers discussing alerts within CPOE systems were identified. Second, CPOE-related papers were identified in a PubMed search by using the MeSH terms ‘decision-support system, clinical’, ‘drug-therapy, computer-assisted’ and ‘medical order entry system’ or by searching for ‘CPOE’ in the title or abstract. Papers retrieved also had to contain terms related to ‘alerting’ in the title or abstract. In this PubMed search, 184 papers were identified. Combining the results of the hand search and PubMed search, 224 unique papers were identified. The abstracts of these papers were reviewed, and 67 papers possibly containing information on medication alert presentation issues were identified. The full texts of these papers were then read by two researchers, and context factors discussed (positively or negatively) anywhere in the papers were identified. The identified factors were then aggregated by two researchers using an inductive approach. Overall, 20 different context factors were extracted from the papers. The details of the process and the steps for internal validation of this context model can be found in Riedmann et al.¹⁶

The 20 identified context factors can be grouped into three main axes:

Context of organizational unit: information on the organizational setting, such as information on the hospital (eg, speciality), the department or the user (eg, experience).
Context of patient and case: information on the patient and the case, such as demographic data, laboratory data, diagnostic data or risk factors.
Context of alert: information on the severity or probability of an ADE the alert refers to.

Supplementary appendix 1 (available online only) presents the 20 context factors in more detail.

The first objective of our study was to assess whether these different context factors can be used to prioritize and differently present medication alerts.

Providing decision support within CPOE systems

Several approaches are discussed in the literature to improve patient safety by CPOE systems. Additional approaches have been developed by the European PSIP project.¹⁵ The following ways of delivering alert information are either already available or are currently being developed within the PSIP project (see supplementary appendix 2, available online only, for a more complete description):

External drug information for clinicians: this allows a doctor, nurse or pharmacist to review detailed drug-related information. These tools typically do not have access to patient data. This type of basic decision support has been in use for many years in many healthcare organizations.¹⁷ ¹⁸
Passive alerting module: these tools developed within PSIP allow a clinician to review all recent alerts on a given patient, such as drug–drug–interaction alerts, at any time he/she wants.¹⁹ This tool is called ‘passive’ because the alerts do not interrupt the prescription process, but are just displayed passively in a list that can be retrieved at any time.
Active alerting module: these modules present patient-related drug safety alerts in an interruptive way, typically during the prescription process. These modules are already in use as part of CPOE systems and have been shown to be quite effective in increasing patient safety.⁸
Prescription simulation module: in these modules developed within PSIP,¹⁹ a clinician can try several combinations of drugs, the module immediately shows all alerts that are relevant for the given combination of drugs. The user can then try another combination. This simulation is typically performed before the final prescription decision is made and entered into a CPOE system.
Patient medication module: these personalized modules tailored for patients and their relatives have also been developed within PSIP.²⁰ Patients can review information on their prescribed medication as well as all related alerts. Information is presented in a patient-understandable way.
ADE epidemiology information: this concept, which was also developed within PSIP, comprises a summarized overview on the number and type of ADE that occurred in a given clinical unit.²¹ This information is presented to clinicians and aims at influencing future prescribing patterns by learning from earlier incidents.

The second objective of our study was to assess the potential impact of these different approaches on ADE rates, each approach addressing different timings and different receivers of a medication alert.

Methods

The approach: a Delphi study

We used a Delphi study to address our two study questions:

What information about the clinical and user context can help to prioritize and differently present medication alerts?
Can different ways of delivering alert information, addressing different timings and different receivers, help to reduce ADE rates?

The Delphi study comprises a systematic, iterative collection of expert opinions.²² ²³ Compared with other methods, such as interviews, it enables the inclusion of a larger number of experts.²⁴ We conducted a type 3 Delphi study²⁵ comprising two rounds. In the first round, the participants were asked to answer predefined standardized questions with regard to the study questions (see details below). In the second round, the results of the first round were fed back to the participants and they had the opportunity to revise their opinion.

The Delphi study was conducted between May 2010 and August 2010. A third round was not conducted as the results were stable between the first and second round (eg, the top five factors did not change order).

Selection of the appropriate expert panel

We conducted a systematic PubMed search to identify CPOE experts from different countries and different professional perspectives. We searched for the terms ‘CPOE’ in all fields and for the terms ‘medical order entry systems’, ‘computer-assisted drug therapy’ and ‘clinical decision support system’ as major MeSH headings. The first authors of the retrieved papers were invited only if they had published at least two CPOE papers or if they were personally known to us as CPOE experts. Overall, we identified 217 authors who we then invited to participate; these experts were also asked whether they could recommend further experts who were then also invited. After this invitation, we received 14 new recommendations. From all 231 invited experts, 15 could not be contacted due to wrong e-mail addresses and two chose not to participate. Overall, 214 experts were finally invited. These invited experts came from 27 countries with the large majority from the USA and Europe. Experts who participated in both rounds were awarded with a €50 book voucher for their efforts.

Content of the Delphi study

A web-based survey (using LimeSurvey, http://www.limesurvey.org) was developed that comprised three parts:

Delphi study part 1: selecting and ranking of useful context factors

All 20 potential context factors were presented to the experts (see supplementary appendix 1, available online only). The experts were then asked: ‘In some hospitals, systems for prescribing drugs already provide automatic medication safety alerts. These systems tend to produce a large number of alerts that may not be relevant for a given clinical situation. We would like to have your opinion on which factors might be useful to prioritize and filter alerts.’ The experts could then mark one or more factors that they found useful.
The experts were then asked: ‘If you think that the preceding list of factors is incomplete, are there other factors that could be used to filter irrelevant alerts?’
Finally, the experts were asked to identify the top five factors: ‘From the factors you have chosen, which are the five most useful factors?’ Here, the expert could see all the factors he or she had identified earlier in step 1, and then could rank the top five of those by adding numbers (1–5) to them.

Delphi study part 2: estimated impact of electronic tools on ADE

The experts were introduced to the six ways of delivering alert information discussed in section 2.2 (for details, see supplementary appendix 2, available online only). The experts were then asked two questions:

‘Do you think that this tool has the potential to prevent ADEs compared to a hospital where electronic prescribing is already in use, but without any kind of electronic decision support?’ (yes/no)
‘Based on your personal judgment: from 100 preventable ADE cases, how many do you think could be avoided in the best case by this tool in a non-specialized hospital where electronic prescribing is in use, but without any kind of electronic decision support?’ (0–100).

Part 3: professional expertise

Finally, the experts were asked about their ‘professional perspective’ (healthcare provider, university, other), about their ‘professional role when dealing with CPOE’ (physician, pharmacist, informatics specialist, other), about their level of expertise with regard to CPOE (foundation, intermediate, advanced) and about the percentage of their CPOE activities that dealt with research (0–100%).

Data analysis

We first calculated the non-weighted frequency and then the weighted frequency for each factor. Here, a top five factor received between five points and one point, based on the number assigned by the expert. All other factors judged as ‘useful’ received one point. The weighted frequency was then calculated as a sum of all the points received.

For each of the six tools that we presented, we calculated the absolute and relative frequencies for their potential (yes or no). Estimated reductions in ADE rates were analyzed using box plots.

Results

Participants and return rate

We invited 214 experts to the first round, of whom 73 completed the first round (34.1%) and 69 the second round (32.2%). These 69 participants came from 15 countries, mostly from North America (50.7%) and from Europe (39.1%).

Over half of the participants held a university perspective and approximately one-third a healthcare provider perspective (table 1). The large majority indicated an intermediate or advanced level of CPOE expertise. The participants were mostly medical informatics specialists, physicians or pharmacists. The participating experts considered their CPOE-related activities mostly as research (median: 60%, mean: 61%; range 33–100%).

Table 1

Professional perspective, role, and level of CPOE expertise of the 69 participants

Usefulness of context factors

Figure 1 shows how many participants judged each factor as ‘useful’. Those factors with more than 50% agreement are highlighted.

Figure 1

Percentage of participants who found that a context factor is useful to prioritize and filter alerts (n=69).

After assigning the weights as described above, the list of top five factors and the list of the lowest five factors remained the same.

The factors were mostly stable between the first and second round of the Delphi study. For example, the top five factors and the lowest five factors were the same in both rounds.

The participants gave 35 free-text statements. Seven of them contained suggestions for missing factors: drug history (including stopping of a drug); whether a prescription is based on a clinical protocol; whether there is advice that may reduce the ADE risk; the degree to which the alert is actionable; and already planned clinical actions (such as the monitoring of laboratory values).

Impact of ways of delivering alert information on ADE rates

Participants felt that each of the proposed ways of delivering alert information has a potential to reduce ADE rates (see figure 2 for details).

Figure 2

Answers to the question: ‘Do you think that this tool has the potential to prevent ADE?’ (n=69). ADE, adverse drug event.

Figure 3 shows the estimated amount of ADE reduction for each tool. Active alerting modules as well as prescription simulation modules were ranked highest (median for both 25% ADE reduction).

Figure 3

Answer to the question: ‘From 100 preventable ADE, how many could be avoided by the tool?’ (n=69). ADE, adverse drug event.

In their free-text comments, the participants indicated three other possibilities for reducing ADE: automatic decision support by proposing the drugs that are best suited for a given patient's diagnosis; review of medication orders of high-risk patients by a second expert who is informed automatically; and voluntary ADE reporting systems by physicians.

Discussion

Answers to the study questions

According to the 69 participating CPOE experts the top-five useful context factors for prioritizing alerts are (in descending order of usefulness): (1) severity of the effect of the ADE the alert refers to; (2) clinical status of the patient; (3) probability of occurrence of the ADE the alert refers to; (4) risk factors of the patient; and (5) strength of evidence on which the alert is built (see figure 1 for details).

All of the experts agreed on the potential to prevent ADE by using CPOE systems with active alerting (see figure 2). The experts estimated that on average every fourth preventable ADE can be avoided by integrating an active alerting module or a proactive prescription simulation module into the hospital information system architecture (see figure 3).

Strengths and weaknesses of the study

Only a third of all the invited experts completed both rounds. This can be seen as a potential limitation. On the other hand, we did not use a convenience sample of easily available experts (eg, only experts personally known to us), but contacted all researchers worldwide who we identified as first authors of CPOE papers. This method of identifying experts can be viewed as a strength, because it reduced potential bias in the selection of experts. Furthermore, in comparison with other Delphi studies with less than 30 participants,^26–28 69 participants seems quite high. The identification method and the number of participants should have yielded reasonable diversity in professional points of view.

The final expert panel consisted of CPOE experts from 15 countries spread over five continents, with approximately half from the USA and one-third from Europe. This mirrors the distribution of scientific papers on CPOE systems in the literature. Two-third of participants self-assessed their level of expertise with CPOE systems to be advanced, which is an indication that we did reach CPOE experts.

The sample was mostly restricted to experts who are publishing in scientific journals. This means that the results represent the point of view of researchers. The perspectives of CPOE users, CPOE developers and CPOE implementers may be different, and they were not investigated in this study.

The experts had to read and judge a list of 20 suggested factors. This could have contributed to a ‘serial position effect’, meaning that the first factors listed are treated differently than factors listed further down (eg, when participants are losing concentration). To minimize this effect, we chose an automatic random order of factors for each expert in the survey.

Another possible source of bias is the way of describing the factors. For each factor, a short definition was given together with a short prime example (see supplementary appendices 1and 2, available online only). It may be that the experts only judged a factor based on the one example given without considering other situations. We cannot assess the impact of this source of bias; it seems clear, however, that examples are needed to explain the meaning of the factors and that the number of examples that can be given is limited.

Both non-weighted and weighted frequencies showed the same top five factors. We view this as an indication for the stability of the results. Some experts expressed concerns about the validity of their estimates about ADE reductions. This, however, is the task of a Delphi study: to gather estimates of unsure future developments.

The context factors were developed for ‘normal’ inpatient settings; usefulness of context factors may be different in intensive or outpatient units.

Results in relation to other studies

Study objective 1: usefulness of different context factors

There are reviews on the effectiveness of different types of alerts (eg, drug–drug-interaction alerts and drug–laboratory alerts),²⁹ studies on how alerts are handled by clinicians³⁰ and studies on frequencies and reasons for alert overriding.¹³ Several of these studies provide specific suggestions on how to improve alert presentation in order to improve alert effectiveness and reduce alert fatigue (eg, see next paragraphs). To our knowledge, however, there are no studies that first define and then compare context factors in a controlled trial (eg, comparing the usefulness of ‘severity’ vs ‘professional experience’). Obviously, comparing all factors with each other in individual simulations or field studies would require quite a number of controlled studies. We therefore chose the Delphi study to compare all 20 factors with each other in a unified way to determine which factors may be most beneficial to be studied further in controlled trials.

Our list of 20 factors was identified by a literature review, so each factor has been discussed in the literature. However, different points of view were often expressed on the usefulness of the factors. For example, there is some controversy in the literature about the ‘severity of the effect of the ADE the alert refers to’, which was the factor given the highest ranking by experts in our study. Some authors such as Kuperman et al¹⁰ (p. 37) and Weingart et al³¹ (p. 2625) discussed the usefulness of this factor; others such as van der Sijs et al³² (p. 446) and Tamblyn et al³³ (p. 437) viewed it differently. This controversy could depend on one's interpretation of ‘severity’. Physicians may rate the severity of an alert differently depending on whether there are organizational or clinical rules that could prevent the manifestation of the related ADE (eg, by monitoring certain laboratory values) (p. 446).³²

The clinical status of the patients, ranked highly in our survey, has also been considered important by other researchers such as van der Sijs et al¹² (p. 144). It seems quite clear that the inclusion of more clinical parameters such as laboratory values can help to tailor alerts better.

Another highly ranked factor, ‘strength of evidence on which the alert is built’ is also mentioned by other authors such as Kuperman et al¹⁰ (p. 37). It is not surprising that our participating experts rate this as an important factor, given their research background.

‘Probability of occurrence of the ADE the alert refers to’ and ‘risk factors of the patient’ were highly ranked by our experts, but are mentioned seldom in the literature.

An initially unexpected result was the poor ranking achieved by the context factor ‘professional experience of the user’. This factor is mentioned quite frequently in the literature, for example, by van der Sijs et al,¹² and is often mentioned as a key example to describe the contextualization of drug safety alerts (eg, alerts for senior physicians vs alerts for junior physicians). In our survey, it is ranked lowest. Our sample mostly consisted of CPOE researchers, not clinical practitioners outside a research context; this might explain the low ranking of this factor.

Study objective 2: impact of different ways of delivering alert information on ADE rates

Our experts estimated that, depending on the tools used, between 10% and 25% of ADE could be prevented. It must be noted that these numbers are subjective judgments, as no empirical evidence is available for most of the tools as yet. As figure 3 shows, experts often gave comparable estimates; we see this as an indicator for the potential validity of these estimates.

In our study, the highest estimates of approximately a 25% ADE reduction were given for active alerting modules. This is not surprising as sufficient evidence for their benefit in reducing medication errors and ADE rates is available.⁸ However, all studies included in the review by Ammenwerth et al⁸ compared the intervention with a paper-based control situation; here, ADE risk reductions of 13–84% were found. We did not find studies that assessed ADE risk reductions comparing computer-based ordering with and without an active alerting module. We can now provide a first estimate of 25%.

External drug information is also a frequently used tool in many hospitals—Sharp et al³⁴ identified over 30 major drug information resources. While evaluations of the quality of the content of these services and of their usage already exist,¹⁷ we are not aware of any systematic evaluations of the impact of these tools on patient outcomes. Our experts estimated an ADE risk reduction of less than 10%. Drug information services have the drawback of not using individual patient data. Moreover, these tools are typically used voluntarily. This all may reduce their impact on patient safety.

Passive alerting modules, proactive prescription simulation modules and ADE epidemiology information are new tools just being developed in the PSIP project.¹⁵ No empirical evaluation results are available yet. Our experts judged their potential impact to be between 10% and 25%. The high estimates for the simulation modules point to the fact that this is seen as an interesting support within the ordering process.

The experts estimated that a patient medication module may lead to an ADE reduction of approximately 15%. General drug information services available to patients already exist, for example,³⁵ but the concept of the patient portal envisions an extended tool that offers personal, tailored, drug-related information based on a patient's actual prescriptions. The participating experts obviously support this idea by estimating an ADE risk reduction of approximately 15%.

Meaning and generalizability of the results

To our knowledge, the ‘context model’ used in this study is the first attempt to systematically describe and structure information on the clinical and user context that can be used to optimize alert prioritization and alert presentation. It may help to improve the ‘alert logistics’ in clinical settings and therefore support a reduction of over-alerting and resulting alert fatigue.

Unanswered and new questions

Although some of the methods to reduce ADE presented to experts in this study are relatively new, many of them are already available or under development. Future research will have to show in empirical evaluation studies whether the estimated benefits can be obtained in real practice.

The set of context factors for alert prioritization recommended by the experts who participated in this study can be seen as a starting point to develop more effective alert mechanisms within CPOE systems. Systematic trials should be conducted to determine whether alert overload can be reduced without reducing the sensitivity of the alerts too much.

The experts proposed some additional context factors, such as whether a prescription is based on a clinical protocol; these ideas need to be investigated further. In addition, the list of factors should also be complemented and validated from the point of view of practising clinical users, which may be different from those of experts.

In this study we did not investigate the information logistics needed to provide CPOE systems with the necessary information (such as professional experience and workload of the user, severity of the expected effect or clinical status of the patient). Obviously, some of this information will have to come from clinical or administrative systems, others from drug information sources. In addition, some context factors are quite vague and general (such as ‘clinical status of a patient’) and need to be defined precisely. Research on how to provide this information through adequate information systems architectures and interfaces is ongoing work within the PSIP project. We also did not ask the experts about the feasibility of the proposed tools or the way these tools can be integrated into the clinical workflow.

This study focused on each factor individually. It has not yet been investigated how the most important factors can and should be combined in a clinical situation. For this, additional studies are needed.

System developers and researchers should collaborate to develop these tools and to implement and evaluate them in field studies. Some of this is ongoing research within the PSIP project. Future research will show whether the estimated ADE reduction can be achieved and at what cost.

Conclusion

The serious problem of alert overload and alert fatigue and how to improve drug safety alerting in CPOE systems is a current issue in health informatics. The results of this study may provide CPOE system developers and healthcare institutions with insights on how to develop more effective alert mechanisms based on context information and new ways of delivering alert information.

Acknowledgments

The authors would like to thank all of the participants for actively contributing to this research. They also wish to thank the anonymous reviewers for their constructive comments on earlier versions of this paper.

Footnotes

Funding: The research leading to these results has received funding from the European Community's Seventh Framework Program (FP7/2007–2013) under grant agreement no 216130.

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Byers J, White S, editors. , eds. Patient Safety: Principles and Practice. New York, NY: Springer, 2004.

2. WHO 10 Facts on Patient Safety. Geneva: World Health Organization, 2010. http://www.who.int/features/factfiles/patient_safety/patient_safety_facts/en/index.html (accessed 1 Jun 2011).

3. Council of Europe Committee of Experts on Management of Safety and Quality in Healthcare (SP-SQS)—Expert Group on Safe Medication Practices: Glossary of Terms Related to Patient and Medication Safety. 2005. http://www.who.int/patientsafety/highlights/COE_patient_and_medication_safety_gl.pdf (accessed 1 Apr 2008).

4. European Commission Communication from the Commission to the European Parliament and the Council on Patient Safety, Including the Prevention and Control of Healthcare-Associated Infections. Brussels: European Commission, 2008. http://www.ec.europa.eu/health/ph_systems/docs/patient_com2008_en.pdf (accessed 1 Jun 2011).

5. Aspden P, Wolcott JA, Bootman JL, et al., editors. , eds. Institute of Medicine. Committee on Identifying and Preventing Medication Errors. Preventing Medication Errors. Washington, DC: National Academies Press, 2007.

6. Shojania KG, Duncan BW, McDonald KM, et al., editors. , eds. Making Healthcare Safer: A Critical Analysis of Patient Safety Practices, Evidence Report/Technology Assessment No. 43, AHRQ Publication No. 01-E058. Rockville, MD: Agency for Healthcare Research and Quality, 2001.

7. von Laue NC, Schwappach DL, Koeck CM. The epidemiology of preventable adverse drug events: a review of the literature. Wien Klin Wochenschr 2003;115:407–15. [PubMed]

8. Ammenwerth E, Schnell-Inderst P, Machan C, et al. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008;15:585–600. [PMC free article] [PubMed]

9. Hug BL, Witkowski DJ, Sox CM, et al. Adverse drug event rates in six community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med 2010;25:31–8. [PMC free article] [PubMed]

10. Kuperman GJ, Bobb A, Payne TH, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007;14:29–40. [PMC free article] [PubMed]

11. Khajouei R, Jaspers MW. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010;49:3–19. [PubMed]

12. van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13:138–47. [PMC free article] [PubMed]

13. Grizzle AJ, Mahmood MH, Ko Y, et al. Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 2007;13:573–8. [PubMed]

14. Dey AK. Understanding and using context. Personal and Ubiquitous Computing 2001;5:4–7.

15. Beuscart R, Hackl W, Nohr C. Detection and Prevention of Adverse Drug Events—Information Technologies and Human Factors. Amsterdam: IOS Press, 2009.

16. Riedmann D, Jung M, Hackl WO, et al. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011;11:35 http://www.biomedcentral.com/1472-6947/11/35(accessed 6 Oct 2011). [PMC free article] [PubMed]

17. Costerison EC, Graham AS. Developing and promoting an intranet site for a drug information service. Am J Health Syst Pharm 2008;65:639–43. [PubMed]

18. Polen HH, Clauson KA, Thomson W, et al. Evaluation of nursing-specific drug information PDA databases used as clinical decision support tools. Int J Med Inform 2009;78:679–87. [PubMed]

19. Nies J, Koutkias V, Kilintzis V, et al. Information contextualization in decision support modules for adverse drug event prevention. In: Koutkias V, Nies J, Jensen S, et al., editors. , eds. Patient Safety Informatics. Amsterdam: IOS Press, 2011:95–104.

20. Lawton K, Skjoet P. Assessment of three systems to empower the patient and decrease the risk of adverse drug events. In: Koutkias V, Nies J, Jensen S, et al., editors. , eds. Patient Safety Informatics. Amsterdam: IOS Press, 2001:246–53.

21. Marcilly R, Chazard E, Beuscart-Zéphir MC, et al. Design of adverse drug events-scorecards. Stud Health Technol Inform 2011;164:377–81. [PubMed]

22. Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. 2002. http://www.is.njit.edu/pubs/delphibook (accessed May 2009).

23. Dalkey NC. The Delphi Method: An Experimental Study of Group Opinion. United States: Air Force Project Rand, 1969. http://www.rand.org/pubs/research_memoranda/RM5888/ (accessed May 2009).

24. Green KC, Armstrong JS, Graefe A. Methods to elicit forecasts from groups: Delphi and prediction markets compared. Foresight: The International Journal of Applied Forecasting 2007;8:17–21.

25. Häder M. Delphi Befragungen—Ein Arbeitsbuch [Delphi Surveys]. Wiesbaden, Germany: Westdeutscher Verlag GmbH, 2002.

26. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol 2005;5:37. [PMC free article] [PubMed]

27. Pare G, Sicotte C, Jaana M, et al. Prioritizing the risk factors influencing the success of clinical information system projects. A Delphi study in Canada. Methods Inf Med 2008;47:251–9. [PubMed]

28. Sittig DF, Campbell E, Guappone K, et al. Recommendations for monitoring and evaluation of in-patient Computer-based provider order Entry systems: results of a Delphi survey. AMIA Annu Symp Proc 2007:671–5. [PMC free article] [PubMed]

29. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 2009;16:531–8. [PMC free article] [PubMed]

30. van der Sijs H, van Gelder T, Vulto A, et al. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform 2010;79:361–9. [PubMed]

31. Weingart SN, Toth M, Sands DZ, et al. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163:2625–31. [PubMed]

32. van der Sijs H, Aarts J, van Gelder T, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15:439–48. [PMC free article] [PubMed]

33. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc 2008;15:430–8. [PMC free article] [PubMed]

34. Sharp M, Bodenreider O, Wacholder N. A framework for characterizing drug information sources. AMIA Annu Symp Proc 2008:662–6. [PMC free article] [PubMed]

35. Huber M, Kullak-Ublick GA, Kirch W. [6-year experience with a drug information service for patients] [in German]. Med Klin (Munich) 2009;104:220–4. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association