The Effect of Computerized Physician Order Entry with Clinical Decision Support on the Rates of Adverse Drug Events: A Systematic Review

doi:10.1007/s11606-008-0504-5

J Gen Intern Med. 2008 April; 23(4): 451–458.

Published online 2008 March 29. doi: 10.1007/s11606-008-0504-5

PMCID: PMC2359507

The Effect of Computerized Physician Order Entry with Clinical Decision Support on the Rates of Adverse Drug Events: A Systematic Review

Jesse I. Wolfstadt,¹ Jerry H. Gurwitz, MD,² Terry S. Field, DSc,² Monica Lee, RPh,¹ Sunila Kalkar, MBBS, MD,¹ Wei Wu, MSc,¹ and Paula A. Rochon, MD, MPH^1,³

¹Kunin-Lunenfeld Applied Research Unit, Baycrest Centre, Toronto, Ontario Canada

²Meyers Primary Care Institute, University of Massachusetts Medical School, Fallon Clinic Foundation, and Fallon Community Health Plan, Worcester, MA USA

³University of Toronto, Toronto, Ontario Canada

Paula A. Rochon, Phone: +1-416-7852500, Fax: +1-416-7854230, Email: paula.rochon/at/utoronto.ca.

Corresponding author.

This article has been cited by other articles in PMC.

Abstract

Context

Computerized physician order entry (CPOE) with clinical decision support (CDS) has been promoted as an effective strategy to prevent the development of a drug injury defined as an adverse drug event (ADE).

Objective

To systematically review studies evaluating the effects of CPOE with CDS on the development of an ADE as an outcome measure.

Data Sources

PUBMED versions of MEDLINE (from inception through March 2007) were searched to identify relevant studies. Reference lists of included studies were also searched.

Methods

We searched for original investigations, randomized and nonrandomized clinical trials, and observational studies that evaluated the effect of CPOE with CDS on the rates of ADEs. The studies identified were assessed to determine the type of computer system used, drug categories being evaluated, types of ADEs measured, and clinical outcomes assessed.

Results

Of the 543 citations identified, 10 studies met our inclusion criteria. These studies were grouped into categories based on their setting: hospital or ambulatory; no studies related to the long-term care setting were identified. CPOE with CDS contributed to a statistically significant (P

≤

.05) decrease in ADEs in 5 (50.0%) of the 10 studies. Four studies (40.0%) reported a nonstatistically significant reduction in ADE rates, and 1 study (10.0%) demonstrated no change in ADE rates.

Conclusions

Few studies have measured the effect of CPOE with CDS on the rates of ADEs, and none were randomized controlled trials. Further research is needed to evaluate the efficacy of CPOE with CDS across the various clinical settings.

KEY WORDS: computerized physician order entry, clinical decision support, adverse drug event

INTRODUCTION

Computerized physician order entry (CPOE) has been defined as an electronic application used by physicians to order drugs, laboratory tests, and requests for consultations,¹ ensuring that all orders are legible and complete.² Clinical decision support (CDS) encompasses a wide range of computerized tools directed at improving patient care, including computerized reminders and advice regarding drug selection, dosage, interactions, allergies, and the need for subsequent orders.³ CPOE linked with CDS has been promoted as having great potential for reducing medication errors and adverse drug events (ADEs).⁴^–⁸ Clinical decision support can provide either basic (e.g., drug-allergy checking) or advanced (e.g., drug dosing support for renal insufficiency) guidance to the prescriber.⁸ The implementation of CPOE with CDS can be a challenging process necessitating careful planning, pilot-testing, and training, as well as appropriate policy and process changes, sufficient infrastructure, and continuous updating and revisions.⁸^–¹³

Adverse drug events are defined as drug-related injuries to patients.⁷ Preventable ADEs are the subset of these injuries that are associated with errors that occur during the ordering, administering, dispensing, and monitoring of drugs and were deemed to have been preventable.⁷ Many preventable ADEs occur in the hospital,¹⁴^,¹⁵ ambulatory,¹⁶^,¹⁷ and long-term care (LTC) settings,⁷ accounting for substantial health care costs.¹⁸^–²⁰ A 2007 report from the Institute of Medicine on medication errors estimated that between 380,000 and 450,000 preventable ADEs occur annually in the hospital setting, resulting in a cost of $3.5 billion annually in the United States.²¹ With the inclusion of estimates from ambulatory and long-term care setting, this report projected that over 1.5 million preventable ADEs occur annually in the United States.²¹

Much of the research relating to CPOE with CDS has evaluated the process that leads to the development of an adverse event. For example, studies have evaluated potential adverse events (e.g., penicillin being prescribed to an individual with a penicillin allergy) as the outcome measure or the role of CPOE with CDS on changing physicians’ behavior (e.g., prescribing penicillin, receiving an alert identifying this as a potential prescribing error, and discontinuing the penicillin) as the outcome. More important than measuring the effects of CDS on processes of care are its effects on patient outcomes, specifically drug-related injuries. Accordingly, we conducted a systematic review to examine the evidence documenting the relation between CPOE with CDS on the development of an ADE as the outcome measure.

METHODS

Search Strategy

We conducted a systematic review to identify all English-language original investigations that evaluated the benefit of CPOE with CDS on the development of an ADE as an outcome measure. We used a 3-step process to identify original investigations for inclusion in our sample as outlined in Figure Figure1.1. First, we searched MEDLINE using PUBMED (from 1966 to March 2007) using all fields for the following MeSH terms: Medical Order Entry Systems or Decision Support Systems, Clinical or Drug Therapy or Medication Errors. We then searched the following textwords: Physician Order Entry or CPOE or POE or CDS or Adverse Effects of Drug Therapy. A total of 543 citations were identified for further review by two of us (S.K. and M.T.). Second, we obtained all of the available abstracts and excluded those that were not original investigations (i.e., 72 citations without an abstract as these were unlikely to represent original investigations and 58 review articles), case reports (n

29), cost benefit analyses (n

21), and those that focused on CPOE or CDS but where ADE was not an outcome measure (e.g., management, system characteristics, guidelines, medication errors, changes in physician behavior; n

356). A total of 7 original investigations were identified for inclusion. Finally, we searched the reference list of these 7 original investigations and identified 3 additional studies meeting our criteria. These new original investigations were not identified by our search strategy because they were published before 1998, which was before the MeSH heading “decision support systems” was introduced.

Figure 1

Process used to identify original investigations for inclusion in our sample

Data Abstraction and Analysis

Study Characteristics

The original investigations were grouped into 3 broad categories based on the setting of care: hospital care, ambulatory care, or long-term care. Studies were further characterized as to whether a focus was placed on a specific patient population (i.e., pediatric, older adults, or intensive care.

Clinical Decision Support

The type of CPOE/CDS system was classified as homegrown or commercially sold. Homegrown systems were defined as systems developed internally by the clinical entities in which they were being used. Commercially sold systems were products purchased from a software developer that may have been modified for use in a particular clinical setting. We identified the type of drug categories being targeted with the CDS. These categories were classified as: general (e.g., all drug orders) or specific (e.g., antibiotics or psychotropic medication orders).

The studies were evaluated to determine whether the CDS system was basic or advanced using the approach outlined by Kuperman et al.⁸Basic CDS systems performed drug-allergy checking, basic dosing guidance, formulary decision support, duplicate therapy checking, and drug–drug interaction checking functions. Advanced CDS systems performed more complex functions including dose adjustment for renal insufficiency, guidance for medication-related laboratory testing, drug-disease contraindication checking, and drug-pregnancy checking.

ADE Capture

We described the data sources and the process used to capture ADEs. Data sources that were used to identify possible ADEs included medical records, incident reports, signals from abnormal serum drug levels (e.g., digoxin toxicity), abnormal laboratory value (e.g., elevated potassium level), pharmacy orders for possible antidotes (e.g., kayexalate), and information on possible ADEs solicited from staff. We also described the process used to adjudicate ADEs. This could be an ADE identified by a computer alert or could be an ADE that was identified as part of a formal review process such as has been described by Bates et al.¹⁵ ADEs were considered to be preventable if they were caused by an error or were classified as preventable using a standard approach.¹⁵ We summarized the effect of CPOE with CDS implementation on the rates of ADEs.

Quality Scoring

We used the approach outlined by Downs et al.²² to assess study quality as this approach can be used to assess both randomized and observational study designs. This quality assessment included 10 items related to reporting, 3 items related to external validity, 7 items related to internal validity—bias, and 6 items related to internal validity—confounding. For 25 of the 26 items, a score of 1 point was assigned if the item was present. One item in the reporting section was assigned 2 points. Accordingly, the maximum score that could be obtained for an original investigation was 27.

Quality scores were independently obtained by two of us (S.K. and W.W.). The initial agreement between the reviewers was 79%. All differences were subsequently resolved with discussion.

RESULTS

Setting, Type of Care Provided, and Study Design

We identified 10 studies that evaluated the effect of CPOE with CDS on ADEs for inclusion in our sample as outlined in Table 1.²³^–³² Nine were performed in hospital settings: 3 hospital-wide CPOE with CDS,²³^,²⁴^,²⁸ 1 focused on older patients,³⁰ 1 specific to pediatric patients,³² 4 evaluated patients in the intensive care unit setting (ICU).²⁵^–²⁷^,²⁹ One study was performed in the ambulatory care setting.³¹ No studies were performed in the long-term care setting. None of the identified studies were randomized controlled trials (RCTs).²⁷ The quality of the studies was relatively good with scores ranging from 13 to 20 (mean

17, SD

2.7). Of the 10 studies, 7 evaluated homegrown CPOE with CDS systems²³^,²⁴^,²⁶^–³⁰ and 3 studies used commercially sold systems.²⁵^,³¹^,³² Four of the studies were conducted using the LDS HELP system,²⁶^–²⁹ and 3 were conducted using the Brigham Integrated Computing System (BICS).²³^,²⁴^,³⁰

Table 1

Description of Original Investigations Evaluating CPOE/Computerized Decision Support on the Development of an ADE as an Outcome Measure

Effect of CPOE with CDS on ADEs

Hospital Care

Hospital-wide CPOE with CDS implementation Evans et al.²⁸ prospectively evaluated the prevention of ADEs with a computer alert program that provided alerts of drug allergies at the time of drug ordering. The study used a quasiexperimental pre/post design and found a significant reduction in the rate of ADEs due to allergic reactions from 56 in the 1-year baseline period to 8 and 18 during 2 subsequent 1-year study periods that incorporated CPOE with CDS (P

.002). There were no ADEs in years 2 and 3 of the study in patients whose drug allergies were known and displayed compared with 13 in the first year when known drug allergies were not displayed. Severe ADEs were significantly reduced from 41 in the first study period to 12 and 15 during the 2 CPOE implementation periods (P

.001). Bates et al.²³ assessed the effectiveness of CPOE with CDS for reducing preventable ADEs. The investigators reported a nonsignificant reduction of 17% in preventable ADEs during the intervention period (P

.37). A second study by Bates et al.²⁴ demonstrated a reduction in the rates of both total ADEs and preventable ADEs per 1,000 patient-days. The trend in total ADEs nonsignificantly fell from 14.7 to 9.6 between the baseline and the third study period (P

.09), and the trend in preventable ADEs significantly decreased from 2.9 in the baseline period to 1.1 in the third study period (P

.05).

Older patients Peterson et al.³⁰ evaluated the implementation of a CDS system for patients 65 years of age or older on the rate of ADEs measured as fall-related injuries per 100 patient-days. Fall injuries were nonsignificantly reduced from 0.17 per 100 patient-days in the 2 baseline periods to 0.06 per 100 patient-days in the 2 intervention periods (P

.09).

Pediatric patients Upperman et al.³² evaluated the effect of a CPOE system with CDS implemented at the Children’s Hospital of Pittsburgh on the rates of ADEs and medication errors. There was no significant change in the rate of total ADEs between baseline and intervention periods, but there was a significant reduction in harmful ADEs per 1,000 doses from 0.05

0.017 in the baseline period to 0.03

0.003 in the intervention period (P

.05).

Intensive care unit patients Evans et al.²⁶ evaluated the effects of a CDS system in the ICU setting. This study found a nonsignificant decrease in the number of antibiotic-related ADEs from 15 ADEs (2.4% of all control patients) in the baseline period to 3 ADEs (0.9% of all study patients) in the intervention period (P

.164). A second study by Evans et al.²⁷ assessed whether a CPOE system with CDS could improve patient care when prescribing antiinfective agents. CPOE with CDS significantly reduced the number of adverse drug reactions to antiinfective agents from 28 in the baseline period to 4 in the intervention period, a reduction of over 70% (P

.018). Colpaert et al.²⁵ reported a significant reduction in the rate of total ADEs per 100 drug orders from 1.0 in a paper-based unit to 0.15 in a computer-based unit consisting of CPOE with CDS (P

.01).

Pediatric intensive care unit patients Mullett et al.²⁹ evaluated the impact of a CDS used to determine antiinfective drug therapies in a pediatric ICU. The investigators found no significant change in the number of ADEs attributable to antiinfective agents during the intervention period.

Ambulatory Care

Clinic-wide CPOE with CDS implementation Steele et al.³¹ investigated the use of CPOE with CDS in a medical outpatient clinic for reducing medication errors and ADEs. The study demonstrated a nonstatistically significant reduction from 12 ADEs in the baseline period to 2 in the intervention period (P

.35).

ADE Capture

Table 2 describes the type of CDS that was being evaluated and outlines the data sources used to identify ADEs and the process that was used to determine whether an ADE occurred. Eight of the studies used multiple sources of data to identify possible ADEs.²³^–²⁶^,²⁸^–³¹ These ranged from studies that used only medical records to identify possible ADEs to studies that used medical records, incident report, and solicited information.²⁵^,²⁶^,³⁰^,³¹ Formal processes were used to determine whether ADE occurred. These included adjudication of ADEs by 2 trained physician reviewers and classification of events using standard criteria in 2 studies²³^,²⁴ and use of the Narajno classification system to identify ADEs in 2 studies. ²⁸^,³¹

Table 2

Type of CDS Alerts and Data Sources and Mechanisms Used to Identify the ADEs

CDS Features

Table 3 summarizes the features of the CDS system and classifies them into whether these features are consistent with a basic or advanced CDS. All of the studies used CDS systems that incorporated drug-allergy checking, a criteria for basic CDS.²³^–³² Two studies used CDS systems that included all 5 components of basic CDS.²⁴^,³⁰ Six studies used CDS systems with 4 components of basic CDS, but did not feature duplicate therapy checking.²³^,²⁵^–²⁷^,²⁹^,³² One study used a CDS system that did not contain formulary decision support or duplicate therapy checking,³¹ and 1 study used an extremely basic CDS system that included only drug-allergy checking.²⁸ None of the studies of adult patients used CDS systems that contained drug-pregnancy checking, a feature of advanced CDS. Five studies used CDS systems with 2 or more features of advanced CDS,²⁵^–²⁷^,²⁹^,³⁰ 4 studies incorporated only 1 feature of advanced CDS,²³^,²⁴^,³¹^,³² and 1 study had none of the criteria for advanced CDS.²⁸

Table 3

Description of Basic and Advanced Medication Related CDS

COMMENT

Three systematic reviews have summarized the effects of CPOE and CDS on practitioner performance and patient outcomes,³^,³³^,³⁴ focusing on changes in prescribing practices,³ hospital length of stay,³³ and costs of stay.³⁴ None of these reviews have focused specifically on ADEs as the outcome of interest. Through our comprehensive search, we identified 10 articles for inclusion in this review. Five of the 10 studies reported a statistically significant (P

≤

.05) reduction in the number of ADEs through the use of CPOE with CDS.²⁴^,²⁵^,²⁷^,²⁸^,³² Another 4 studies showed a trend towards a reduction in the rate of ADEs with CPOE and CDS, but these did not achieve statistical significance.²³^,²⁶^,³⁰^,³¹ Only 1 study reported no effect on the number of ADEs when a CPOE with CDS was implemented.²⁹

Our review demonstrates that 70% of the studies evaluating the rates of ADEs used homegrown CPOE with CDS systems. Customization is required to make a CPOE with CDS system work for a specific environment. Accordingly, it is understandable that most of the systems evaluated in our study were homegrown. A successful system often requires input from all of the staff members who will use the system (e.g., nurses, physicians, and pharmacists) and can take years to develop.⁴ Results from 7 studies evaluating homegrown CPOE with CDS systems demonstrated statistically significant reductions in ADEs in 3 studies,²⁴^,²⁷^,²⁸ nonstatistically significant reductions in ADEs in 3 studies,²³^,²⁶^,³⁰ and no effect on ADEs in 1 study.²⁹ Since 2003, only 3 published studies assessing the effect of CPOE with CDS on ADEs have evaluated commercial systems.²⁵^,³¹^,³² Of these, 2 studies found a statistically significant decrease in ADEs.²⁵^,³² The third study showed a nonstatistically significant reduction in ADEs.³¹ Accordingly, we know relatively little about the benefit of commercially developed CPOE/CDS systems on reducing ADEs. Our results are similar to a previous systematic review, which concluded that more research was needed to evaluate commercially sold CPOE systems.³ Knowing more about the benefit of commercially developed CPOE/CDS systems will be increasingly important for health care settings planning to select and implement CPOE/CDS system in the future.

We demonstrate that there is considerable variability in the way that ADEs are captured across the studies. Using multiple sources of data as have been done in studies included in our sample will make it more likely that ADEs will be captured. Further different approaches are being used to determine whether ADEs occurred. At present, these approaches are very labor intensive requiring collection of data on events that may represent ADEs from multiple sources, presentation of these cases to groups of 2 trained reviewers, and classification of these events using standard criteria.¹⁵ This may in part explain why relatively few studies have used ADE as their outcome measure.

We identified no RCTs in our systematic review. Whereas RCTs are considered to be the gold standard in terms of study design, they are difficult to conduct in this context. When CPOE is being implemented in a clinical setting, it is often not possible to limit this to only some groups of patients. Despite the difficulty, it is important to know if the CDS rules work. RCT designs, potentially involving cluster randomization techniques, need to be considered to provide the best evidence to foster the development and implementation of these systems.³⁵

Studies that evaluate the efficacy of CPOE with CDS have been conducted predominantly in hospital settings. While there have been descriptions of the development of CPOE with CDS,³⁶ changes in physician’s behavior,³⁷ and the rates of ADEs in the long-term care setting,⁷ no study has reported on the effectiveness of CPOE with CDS in reducing ADEs. Elderly patients often take multiple medications and are at an increased risk of ADEs.³⁷^,³⁸ Long-term care facilities may benefit from CPOE with CDS if computerized entry can be proven to reduce ADEs. Future studies should focus on examining the benefits of CPOE with CDS across clinical settings.

Limitations

Many studies performed to evaluate CPOE with CDS were not eligible for inclusion in our systematic review because they did not include a comparison group. Researchers must incorporate adequate control arms into their studies or a valid assessment of the benefits and potential risks associated with these interventions may not be possible. Some of the studies included in our systematic review were published more than 10 years ago and the CDS systems they were evaluating were relatively simple. Clinical decision support systems being used in future studies will be more sophisticated and more likely to be consistent with newly created standards. Furthermore, 7 of 10 studies included in our sample were conducted using 1 of 2 CPOE systems further restricting the generalizability of our results.²³^,²⁴^,²⁶^–³⁰

CONCLUSION

Only 10 studies have evaluated the effect of CPOE with CDS on reducing ADEs, and half of these found a significant reduction in the number of ADEs. None of the studies were RCTs. The medical settings in which CPOE with CDS have been studied have focused on the hospital. No studies were conducted in the long-term care setting. Future research should focus on the full range of clinical settings in which CPOE with CDS may be employed.

Acknowledgments

The authors would like to thank Mary Thomson Ph.D. for her assistance with the identification of original investigations for inclusion in our sample and Peter Anderson for his assistance with the manuscript preparation. The study was supported by grants (HS010481 and HS15430) from the Agency for Healthcare Research and Quality, Bethesda, MD, USA.

Conflict of Interest None disclosed.

References

1. Poon EG, Blumenthal D, Jaggi T, Honour MM, Bates DW, Kaushal R. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals. Health Aff. 2004;23:184–90.

2. Kaushal R, Bates DW. Information technology and medication safety: what is the benefit? Qual Saf Health Care. 2002;11:261–5. [PMC free article] [PubMed]

3. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409–16. [PubMed]

4. Rochon PA, Field TS, Bates DW, et al. Computerized physician order entry with clinical decision support in the long-term care setting: insights from the Baycrest Centre for Geriatric Care. J Am Geriatr Soc. 2005;53:1780–9. [PubMed]

5. Kohn LT. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000.

6. Fortescue EB, Kaushal R, Landrigan CP, et al. Prioritizing strategies for preventing medication errors and adverse drug events in pediatric inpatients. Pediatrics. 2003;111:722–9. [PubMed]

7. Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities. Am J Med. 2005;118:251–8. [PubMed]

8. Kuperman GJ, Bobb A, Payne TH, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14:29–40. [PMC free article] [PubMed]

9. Gesteland PH, Nebeker JR, Gardner RM. These are the technologies that try men’s souls: common-sense health information technology. Pediatrics. 2006;117:216–7. [PubMed]

10. Sittig DF, Ash JS, Zhang J, Osheroff JA, Shabot MM. Lessons from “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics. 2006;118:797–801. [PubMed]

11. Jacobs BR, Brilli RJ, Hart KW. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117:1451–2. [PubMed]

12. Longhurst C, Sharek P, Hahn J, Sullivan J, Classen D. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117:1450–1. [PubMed]

13. Rosenbloom ST, Harrell FE Jr., Lehmann CU, Schneider JH, Spooner SA, Johnson KB. Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics. 2006;117:1452–5. [PubMed]

14. Bates DW, Boyle DL, Vliet MVV, Schneider J, Leape L. Relationship between medication errors and adverse drug events. J Gen Intern Med. 1995;10:199–205. [PubMed]

15. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. JAMA. 1995;274:29–34. [PubMed]

16. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003;348:1556–64. [PubMed]

17. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289:1107–16. [PubMed]

18. Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA. 1997;277:307–11. [PubMed]

19. Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA. 1997;277:301–6. [PubMed]

20. Field TS, Gilman BH, Subramanian S, Fuller JC, Bates DW, Gurwitz JH. The costs associated with adverse drug events among older adults in the ambulatory setting. Med Care. 2005;43:1171–6. [PubMed]

21. Institute of Medicine. Preventing Medication Errors: Quality Chasm SeriesWashington, DC: National Academy Press; 2007.

22. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52:377–84. [PMC free article] [PubMed]

23. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–6. [PubMed]

24. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–21. [PMC free article] [PubMed]

25. Colpaert K, Claus B, Somers A, Vandewoude K, Robays H, Decruyenaere J. Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial. Crit Care. 2006;10:R21. [PMC free article] [PubMed]

26. Evans RS, Classen DC, Pestotnik SL, Clemmer TP, Weaver LK, Burke JP. A decision support tool for antibiotic therapy. Proceedings from the 19th Annual Symposium on Computer Applications in Medical Care. 1995:651–5.

27. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med. 1998;338:232–8. [PubMed]

28. Evans RS, Pestotnik SL, Classen DC, Horn SD, Bass SB, Burke JP. Preventing adverse drug events in hospitalized patients. Ann Pharmacother. 1994;28:523–7. [PubMed]

29. Mullett CJ, Evans RS, Christenson JC, Dean JM. Development and impact of a computerized pediatric antiinfective decision support program. Pediatrics. 2001;108:E75. [PubMed]

30. Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med. 2005;165:802–7. [PubMed]

31. Steele AW, Eisert S, Witter J, et al. The effect of automated alerts on provider ordering behavior in an outpatient setting. PLoS Med. 2005;2:e255. [PMC free article] [PubMed]

32. Upperman JS, Staley P, Friend K, et al. The impact of hospitalwide computerized physician order entry on medical errors in a pediatric hospital. J Pediatr Surg. 2005;40:57–9. [PubMed]

33. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223–38. [PubMed]

34. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339–46. [PubMed]

35. Donner A, Klar N. Design and Analysis of Cluster Randomization Trials in Health Research. New York: Oxford University Press; 2000.

36. Rochon PA, Field TS, Bates DW, et al. Clinical application of a computerized system for physician order entry with clinical decision support to prevent adverse drug events in long-term care. CMAJ. 2006;174:52–4. [PMC free article] [PubMed]

37. Judge J, Field TS, Deflorio M, et al. Prescribers’ responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc. 2006;13:385–90. [PMC free article] [PubMed]

38. Ganjavi H, Herrmann N, Rochon PA, et al. Adverse drug events in cognitively impaired elderly patients. Dement Geriatr Cogn Disord. 2007;23:395–400.

39. Naranjo CA, Busto U, Sellers EM, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30(2)239–45. [PubMed]

Articles from Journal of General Internal Medicine are provided here courtesy of
Society of General Internal Medicine