PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
 
J Am Med Inform Assoc. 2012 Jan-Feb; 19(1): 22–30.
Published online 2011 August 18. doi:  10.1136/amiajnl-2011-000304
PMCID: PMC3240758

The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials

Abstract

Objective

The US Agency for Healthcare Research and Quality funded an evidence report to address seven questions on multiple aspects of the effectiveness of medication management information technology (MMIT) and its components (prescribing, order communication, dispensing, administering, and monitoring).

Materials and Methods

Medline and 11 other databases without language or date limitations to mid-2010. Randomized controlled trials (RCTs) assessing integrated MMIT were selected by two independent reviewers. Reviewers assessed study quality and extracted data. Senior staff checked accuracy.

Results

Most of the 87 RCTs focused on clinical decision support and computerized provider order entry systems, were performed in hospitals and clinics, included primarily physicians and sometimes nurses but not other health professionals, and studied process changes related to prescribing and monitoring medication. Processes of care improved for prescribing and monitoring mostly in hospital settings, but the few studies measuring clinical outcomes showed small or no improvements. Studies were performed most frequently in the USA (n=63), Europe (n=16), and Canada (n=6).

Discussion

Many studies had limited description of systems, installations, institutions, and targets of the intervention. Problems with methods and analyses were also found. Few studies addressed order communication, dispensing, or administering, non-physician prescribers or pharmacists and their MMIT tools, or patients and caregivers. Other study methods are also needed to completely understand the effects of MMIT.

Conclusions

Almost half of MMIT interventions improved the process of care, but few studies measured clinical outcomes. This large body of literature, although instructive, is not uniformly distributed across settings, people, medication phases, or outcomes.

Keywords: Clinical decision support, clinical decision support systems, communication, decision support, drug therapy, eHealth, electronic prescribing, evidence-based medicine, health information technology, informatics education, information retrieval, library science, machine learning, medical informatics, medication systems, predictive modeling, privacy technology, randomized trials, research methodology, safety, son's name, statistical learning, systematic reviews

Medication management is a major component of the healthcare system. Currently, approximately 10% of the US healthcare budget is spent on prescription medication.1 Suboptimal prescribing, order communication, dispensing, administering, and monitoring of medications can lead to medication errors, therapeutic failures, adverse drug withdrawal events, and adverse drug events (ADEs).2 ADEs are associated with approximately 1.2 million or 3.1% of all hospitalizations in the USA. A meta-analysis of ADEs suggests that these events are between the fourth and sixth leading cause of death in the USA.3 Each ADE is estimated to increase the length of hospital stay by 2.2 days and to increase hospital costs by US$3244.4

Many US organizations endorse the use of medication management information technology (MMIT) to improve the medication management process and patient safety. These include Leapfrog, the Institute of Medicine, the Agency for Healthcare Research and Quality (AHRQ), and the Office of the National Coordinator for Health Information Technology. In addition to these groups, the recent US Health Information Technology for Economic and Clinical Health Act will authorize incentive payments through Medicaid and Medicare for implementing and meeting certain goals using electronic health records for patient care. Achievements in healthcare processes and outcomes will be tied to incentive payments starting in 2011. The goal of the US Health Information Technology for Economic and Clinical Health Act is not just based on the adoption, but also on the ‘meaningful use’ of electronic health records and their integration with other systems. Meaningful use is defined by a set of 25 core health information technology (HIT) objectives that constitute an essential starting point, as well as an additional menu of activities that providers and hospitals may choose to implement.5 Examples of these core measures related to medication management are computerized provider order entry (CPOE) for e-prescribing, drug–drug and drug–allergy checking, and medication reconciliation capabilities.

The AHRQ recognizes that HIT is an important and promising means of improving healthcare outcomes. They contracted with McMaster University to produce a comprehensive report summarizing the existing evidence of the potential for MMIT to improve healthcare processes and support patients needing prescription medication. Previous systematic reviews generally report the effect of MMIT on one of the medication management phases (prescribing, order communication, dispensing, administering, and monitoring). For example, Mollon and colleagues6 reviewed randomized controlled trials (RCTs) of clinical decision support systems (CDSSs) for prescribing and found 37 reports that showed changes in the behavior of healthcare providers, but only five of these noted improvements in patient outcomes. Similarly, Eslami and colleagues7 reviewed mostly observational studies of CPOE applications in outpatient prescribing. They showed that although CPOE and other information systems are often costly, some evidence supports medication safety benefits. However, they also found some studies showing unintended negative consequences including increased error rates (existing and new errors caused directly by the systems or indirectly by changes in workflow) and ADEs with CPOE.

The AHRQ contract specified summarization of the evidence on many aspects of MMIT. The report addressed seven broad questions related to medication management and HIT (effectiveness; gaps in knowledge and evidence; the value proposition of MMIT; system characteristics and their association with decisions to purchase, implement, or use MMIT; sustainability of MMIT; two-way electronic data interchange for order communication; and RCTs for CDSSs).8 The full report, available online, includes 378 studies across multiple qualitative and quantitative methods and many outcomes and encompasses more than 1000 pages. For this review, which is a subset of the full report, we describe the process of collecting and synthesizing only the findings from RCTs on the effectiveness and effects of MMIT on all phases of medication management. We report results for process changes, clinical outcomes, and other outcomes related to use, usability, knowledge, skills, and attitude. Other outcomes are provided in the full report.

Methods

Defining the process of medication management

Bell and colleagues9 model the medication management continuum in five phases, which we use in this review: prescribing or ordering the appropriate medication; order communication or transmitting and perfecting the prescription through the interaction between the prescriber and the pharmacist; dispensing the medication in its requested form and strength; administering the medication; and monitoring (ongoing oversight to review the benefits and potential adverse effects of the medication such as therapeutic failures, adverse drug withdrawal events, and ADEs). We also evaluated the evidence concerning the reconciliation of medication at transitions between settings of care and education. Both education or training for the health professionals associated with installation of MMIT applications and patient education related to medication management were part of our mandate.

Data sources and searches

We searched electronic databases: MEDLINE, EMBASE, CINAHL (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts, Compendex, INSPEC (which includes IEEE), Library and Information Science Abstracts, E-Prints in Library and Information Science, PsycINFO, Sociological Abstracts, and Business Source Complete. The search terms are included in the full online evidence report.8

Supplemental searches targeted gray literature: the New York Academy of Medicine databases, SIGLE, US Health and Human Services Health Information Technology, Health Technology Assessment reports and economic studies from the UK Centre for Reviews and Dissemination, ProQuest Dissertations, National Library for Health United Kingdom (includes Bandolier), ProceedingsFirst, PapersFirst, US National Technical Information Service, and Google. As part of our gray literature search, we reviewed the AHRQ Knowledge Library on e-prescribing, bar coding, and CPOE.

The literature search details are in the full report.8 When possible, we excluded letters, editorials, commentaries, and animal studies. No limits were placed on language, date, subject domain, or geography.

The references of review articles were screened for eligibility. Members of the AHRQ technical expert panel8 and writing team provided articles from their personal files. Our search was peer reviewed by a librarian following the PRESS checklist for systematic review searches.10

Study selection

Independent reviewers from a team of reviewers screened titles and abstracts in duplicate. This first screen was to detect articles that described the use of MMIT. We defined MMIT as electronic systems that collect, process, or exchange health information about the need for and use of medication for patients and their formal caregivers. We included articles only if the medication management system was integrated with at least one other HIT system such as an electronic medical record system. In addition, the MMIT systems had to integrate patient-specific information and provide processed data (eg, trending data for time in therapeutic range for patients taking warfarin), advice (eg, allergy alert for penicillin) or suggestions to healthcare providers or patients and families on medication issues related to health care and wellness (eg, lower lithium dose based on blood concentrations). We excluded stand-alone medical devices such as infusion pumps not linked into electronic medical records (ie, no integration) with the exception of personal digital assistants or handheld devices into which clinicians or patients entered patient-specific information to assist in medication management (eg, dosing calculations for pediatric patients).

For this systematic review, we selected only RCTs that assessed the effectiveness and effects of MMIT (two of the seven contract questions). The full AHRQ evidence report addresses the other five questions as well as qualitative articles and observational studies.8

Data extraction and quality assessment

Data from each article were abstracted by one reviewer and checked for accuracy by a second. The reviewers were not blinded to authors, institutions, or journal. Senior staff performed final accuracy checks. Extracted data included general study characteristics: study design, intervention, study population, setting, disease and drugs of interest, and a description of the medication management phase and MMIT application. We abstracted primary and secondary outcomes from each article. If no primary outcome measures were indicated, we focused on outcomes related to medication management and clinical outcomes.

We recorded whether the outcomes were positively changed by the intervention, negatively, or not changed (not significant). If more than one primary or secondary outcome was reported, the direction of the majority of outcomes was used to determine the overall effect. We used the framework suggested by Chaudry and colleagues11 to present and analyze our findings: process changes (eg, proportion of orders that were adherent to CDSS recommendations, prescription error reduction, improved rates of appropriate antibiotics), knowledge, skills and attitudes; satisfaction; clinical outcomes; and costs.

Experienced reviewers (CL, KAM, AMH, ST, DO'R) assessed the included studies for quality of reporting. Quantitative studies were assessed using the same criteria employed by Jimison et al12 in their AHRQ evidence report. RCT scoring was based on Delphi consensus work done by Verhagen and colleagues.13 Up to nine points were awarded for random allocation, allocation concealment, comparable groups at baseline, the presence of eligibility criteria, blinding (patients, caregivers and outcome assessors), measurement of the variability of outcomes, and intention to treat analyses. In addition, we scored studies on their use of clustered design, analytical adjustments for clustering, and at least 80% follow-up.

Data synthesis and analysis

We did not perform meta-analyses because of study differences in intervention, populations, technologies, and outcomes measured. Data were summarized by counts of articles that showed positive, negative or no difference in outcome measures.

Results

Literature retrieval

Our searches retrieved 32 785 articles that were screened at the abstract level (figure 1). Of those, 4356 were further assessed in full text, 87 of which were RCTs studying the effect of MMIT applications. The full AHRQ evidence report includes analysis of data from 378 articles (191 were not RCTs).

Figure 1
Information flow for literature searching for randomized controlled trials (RCT) in medication management information technologies (MMIT).

General study characteristics

The RCTs focused mainly on assessing the prescribing or monitoring medication management phases (table 1). Twenty-seven articles studied the effect of MMIT on both of these medication management phases. Order communication, dispensing, administering, education, or reconciliation were seldom studied. Supplementary appendix tables 1–6 (available online only) contain study information by setting (hospital and ambulatory) and by endpoints (process changes, clinical and other intermediate measures).

Table 1
List and number of articles studying the phases of medication management

Most of the 87 trials were performed in the USA (n=63, 72.4%), Europe (n=16, 18.4%), and Canada (n=6; 6.9%), and two each in Australia and Israel. Several pioneering US institutions were strongly represented: Harvard, Brigham Women's, and Partners in Massachusetts (n=14), Regenstreif and Wishard in Indiana (n=12), Kaiser Permanente (n=8), Veterans Affairs (n=5), and University of Washington (n=4). Most studies were published after 2000 (86%) (figure 2). Although the number of RCTs increased over time, the proportion of RCTs to all of the literature declined over the same period (data not shown).

Figure 2
Year of publication of the RCT of MMIT. Note that 2010 was not searched only up to June 2010.

Seventy-eight trials (90%) had a CDSS component: eight of these assessed CPOE plus CDSSs,14 26 27 37 53 58 76 82 two assessed e-prescribing systems plus CDSSs.68 83 To differentiate between the two, we tagged systems as being CPOE if they included medication ordering and often other orders in hospitals and communicated with inhospital pharmacies while e-prescribing systems were more often ambulatory based and allowed communication with community pharmacies. Two studies assessed pharmacy information systems with CDSS.21 87 Three studies looked at a CPOE system with minimal or no decision support.63 74 77 Of these, two were older with no or limited CDSS and one was based on order sets.100 Three evaluated a personal health record system16 90 94 and one was a medication reconciliation system.99

Quality and reporting

The articles rated an average score of 4.5 out of 9 (95% CI 4.1 to 4.8) on the Verhagan quality score. Fifty of the trials had a clustered design (57%); 44 of which indicated analysis and adjustment for the clustering, if needed. Sample sizes were five to 156 772 participants in 73 of the trials (mean 11 435, 95% CI 5320 to 17 551) and 10 to 778 healthcare providers in 31 studies (mean 148, 95% CI 84 to 212). Thirty trials reported greater than 80% follow-up, seven reported lower than 80%, and 50 failed to report follow-up.

Outcomes

Process changes

Most of the trials evaluated prescribing. Eighty trials measured one or more changes in process: 69 changes were primary (table 2) and 30 were secondary endpoints (table 3). Nineteen of the 24 trials set in hospitals measured process as the primary endpoint. Eight of these measured changes in prescribing behavior, all of which showed significant improvements (table 2). Ten trials measured adherence to prescribing and monitoring advice, eight of which showed improvement. Errors and time to perform tasks were seldom studied and no hospital trials measured workflow.

Table 2
Results of the trials that used process change measures as their primary outcome, by hospital or ambulatory setting, in 70 RCT given as positive or non-significant
Table 3
Results of trials that reported process changes as their secondary outcomes in hospital and ambulatory care settings across medication management phases

In the ambulatory setting, 46 of 58 trials assessed process as their primary measures (table 2). Most focused on prescribing or monitoring behavior changes and adherence to CDSS advice. Many of the interventions simultaneously influenced prescribing and monitoring. Seven of 10 trials found improvements in prescribing behaviors, 12 of 20 showed significant improvements in adherence to prescribing advice, and 10 of 16 showed improved adherence to monitoring advice. Both trials that measured time for activities showed reductions.63 76 Two trials found significant improvements in composite endpoints,61 89 and two found improvements in dispensing.21 22 The only study measuring the effect of MMIT on patient compliance detected no effect.64 Three trials measured contacts made by healthcare providers47 83 87 and two found no effect of MMIT.47 83 These three trials studied telephone consultation between patients and physicians around prescribing for antibiotics,47 pharmacist and physician telephone calls to perfect the prescription before dispensing,83 and pharmacist discussion with patients for prophylactic aspirin use in diabetes.87

Two trials took place in pharmacies with significant reductions in the rate at which pharmacists called physicians for clarification (callback rate)83 87 and increased patient encounters to prompt aspirin use for people with diabetes.83 87 One study in long-term care assessed prescribing changes as their primary endpoint (appropriateness of antidepressant orders) and found significant improvements.14

Nine hospital-based trials measured process changes as secondary outcomes (table 3). Five of these showed improvements,29 52 65 95 three showed no difference with the use of MMIT,23 66 99 and one showed an increase in time to write orders.77 Nineteen trials in ambulatory settings looked at secondary process changes and nine showed significant improvements (table 3).

Two trials in nursing homes assessed secondary process outcomes; one found significant improvements in prescribing behavior,14 the other found no difference in the number of potential ADEs with MMIT.15

Clinical outcomes

Twenty-three trials evaluated clinical endpoints as their main outcome measures (table 4) and 26 had secondary outcomes of clinical endpoints (table 5). Counting articles with multiple outcomes individually, 38 trials measured a clinical endpoint. Only three hospital-based trials assessed a clinical primary endpoint, of which two showed significant improvements 40 59 and one showed no difference.82

Table 4
Results of the studies that have clinically important outcome measures as their main outcome, by hospital or ambulatory setting, in 23 included RCT given as positive, negative or non-significant
Table 5
Studies that report secondary clinical endpoints across hospital and ambulatory settings.

Twenty ambulatory trials measured clinical endpoints; 12 included such physiological measures as blood pressure, cholesterol, or blood glucose levels. Overall, eight of these 20 trials showed clinical improvements (table 4).

Seventeen trials measured 27 secondary clinical endpoints (table 5). One trial showed significant improvements in both hospitalizations and mortality.84 Another found increased emergency department visits for intervention patients.94 All other trials showed no differences for these secondary outcomes (table 5).

In other settings, Gurwitz and colleagues15 found no difference in the rate of ADEs in nursing homes incorporating CDSSs and CPOE compared with homes with CPOE alone. In a pilot study of five patients using a hand-held insulin regimen optimizer, Holman and colleagues 88 reported improved hemoglobin A1c levels.

Other outcomes: primary endpoints

Three trials had costs as a main endpoint; all three showed improvement including reduced costs per practice with thiazide prescribing for hypertension (US$540 per practice),29 US$37.64 less per patient on antimicrobial agents with the addition of a CDSS,77 and a 13% reduction in costs with CPOE.25 Two trials measured knowledge, skills, or attitudes, one of which was positive,37 and the other not significant.94 One trial examined satisfaction, with no significant difference seen.68

Other outcomes: secondary endpoints

Costs (one not significant95 and one positive73) and perceptions (one not significant75 and one positive83) were measured as secondary outcomes in two articles and satisfaction in three trials (one positive,32 two not significant44 94). All primary cost trials reported cost reductions measured in multiple ways with the use of HIT. Satisfaction with systems and perceptions showed mixed results.

Discussion

A large number of RCTs of MMIT exist. They are not uniform in number or intensity of study (eg, length of study, complexity or extent of MMIT and its integration, number of healthcare providers involved) across medication management phases, settings, types of MMIT, personnel, or outcomes. Prescribing (n=74) and monitoring (n=38) phases were much more frequently studied in the 87 RTCs. We identified very few MMIT interventions that targeted order communication, dispensing, administering, continuing professional or patient education, or medication reconciliation. Most trials were conducted in US centers with a strong history of MMIT: Boston, Indianapolis, Washington state, Kaiser Permanente, and Veterans Affairs hospitals. Most trials were published after 1999. The trials concentrated on CDSSs and CPOE systems and few evaluated MMIT systems used by non-physicians. MMIT in long-term care settings such as nursing homes, pharmacies, homes, and community was also not well studied.

Overall, the quality of the articles was poor, with scores indicating that only half reported the use of methods generally accepted as minimizing bias. In addition, we found varying definitions and methods of measurement of outcomes and lack of consensus on reporting. An RCT may not be the only method necessary for a full assessment of the effects of HIT. Programatic analyses or assessment methods for complex interventions can also provide valuable insights into the effects of MMIT, but we did not identify these in our literature review. The full AHRQ report contains studies of various other designs that address important aspects of MMIT systems that are not usually identified using RCT methods.8 For example, qualitative studies identified substantial unintended consequences of MMIT, and case studies of implementations can provide valuable qualitative insights.

Consistent with other reviews of MMIT, most studies measured changes in process and the majority of these showed benefit. Few trials studied clinical outcomes. Despite using a broad definition of clinical outcomes that included physiological measurements (eg, blood pressure and blood glucose levels), very few studies showed improvement in primary outcomes (11 of 26), and even fewer studies that used clinical endpoints as secondary outcomes found benefit (two of 43 showed benefit and one showed harm).

The literature of MMIT is dominated by implementation and demonstration projects (see full report). These were not designed to evaluate and establish clinical benefit for patients in settings that used MMIT systems compared with those who received usual care without MMIT. Few studies measured clinical improvements for patients or the quality of health care provided. Potential harm to patients resulting from the use of MMIT was rarely evaluated. Future research should address these gaps. In particular, it is also important that the effect of MMIT on the consumer be considered.

The consistent application of a set of agreed-upon standards for the assessment, evaluation, and description of MMIT (analogous to CONSORT guidelines for RCT reporting and PRISMA guidelines for reviews) could improve the quality and generalizability of future research. Consistency and greater depth of reporting is also needed in published trials, especially with respect to MMIT itself and settings.

The findings of this review are consistent with those of the full AHRQ report, which included studies using multiple research designs.8 The AHRQ report also includes summaries of qualitative studies, sustainability of MMIT systems, the value proposition of MMIT, and feature sets associated with the likelihood of purchase, implementation and use.

Limitations

This review has several important limitations. First, the studies reported only limited data on systems, installations, institutions, and targets of the intervention making complex synthesis difficult. We also found problems with methods and analyses, a wide variation in the number of studies in certain areas, and a broad range of MMIT systems. We attempted to address this limitation by basing our work on well-defined analytical frameworks and by identifying not only the systems used but also their functional capabilities. Third, this review summarizes only the RCTs from the 378 studies included in the full report. Although we used only the studies with strong research methods for this overview of MMIT, we feel that they provided a very similar assessment of the effects and effectiveness provided across all articles. Our requirement that MMIT systems were integrated with other HIT may also have led to the exclusion of important evidence. In addition, missing in this review are the qualitative studies that portray a rich understanding of effects of MMIT on clinicians and patients.

Conclusion

Medication management is a complex and important component of health care. MMIT systems have been implemented in many organizations to improve the safe and appropriate use of medication. Many studies demonstrated benefits on process measures for MMIT focused on improving medication management. Little evidence of benefit is available for the effectiveness of the use of MMIT on clinical outcomes. The body of evidence from studies of MMIT is not uniform across domains, settings, phases, and geography, or held to the same standards as the pharmaceutical industry. Future research directions can be based on the findings and gaps of this report: phases of order communication, dispensing, administering, and reconciliation; settings (long-term care, communities, and homes); MMIT beyond CDSSs and CPOE and especially the MMIT systems used by non-physicians; and systems designed for patients and caregivers. Also, multiple study methods need to be used to produce a comprehensive analysis of the benefits, harms and costs of MMIT systems. Reporting standards for all HIT are also recommended for all studies.

Acknowledgments

The research team would like to thank those who helped with screening, abstracting and article retrievals: Danika Walden, Nathan McKibbon, Catherine Salmon, Jan Burke-Gafney, Connie Freeborn, Jamie O'Donnell, Rizwan Bhaloo, Bryan Cheeseman, Hafsa Jan Qureshi, and Pouyan Ahangar. The authors would like to offer special thanks to Mary Gauld, Maureen Rice, Julianna Beckett, and Roxanne Cheeseman for helping with project management and editorial perfection. They acknowledge the hard work of Nicholas Hobson, their computer programmer for creating the evolving systematic review management software. The authors accomplished this task with the expert guidance of Rebecca Roper of the Agency for Healthcare Research and Quality as their task order officer.

Footnotes

Funding: This project was funded under contract HHSA 290-2007-10060-I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services. McMaster EPC was tasked with the original report and it can be found at http://www.ahrq.gov/clinic/tp/medmgttp.htm. The authors of this review are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Truffer C, Keenan S, Smith S, et al. Health spending projections through 2019: the recession's impact continues. Health Aff (Millwood) 2010;29:522–9. [PubMed]
2. Handler SM, Wright RM, Ruby CM, et al. Epidemiology of medication-related adverse events in nursing homes. Am J Geriatr Pharmacother 2006;4:264–72. [PubMed]
3. Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998;279:1200–5. [PubMed]
4. Bates DW, Spell N, Cullen D, et al. The costs of adverse drug events in hospitalized patients. JAMA 1997;277:307–11. [PubMed]
5. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010;363:501–4. [PubMed]
6. Mollon B, Chong J, Jr, Holbrook AM, et al. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med Inform Decis Mak 2009;9:11. [PMC free article] [PubMed]
7. Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inf 2008;77:365–76. [PubMed]
8. McKibbon KA, Lokker C, Handler SM, et al. Enabling Medication Management Through Health Information Technology. Evidence Report/Technology Assessment No. 11-E008-EF. Prepared by the McMaster University Evidence-based Practice Center under Contract HHSA 290-2007-10060-I). AHRQ Publication no. 11-E008-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2011.
9. Bell DS, Cretin S, Marken RS, et al. A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities. JAMIA 2004;11:60–70. [PMC free article] [PubMed]
10. Sampson M. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62:944–52. [PubMed]
11. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144:742–52. [PubMed]
12. Jimison H, Gorman P, Woods S, et al. Barriers and Drivers of Health Information Technology Use for the Elderly, Chronically Ill, and Underserved. Evidence Report/Technology Assessment No. 175 (Prepared by the Oregon Evidence-basedPractice Center under Contract No. 290-02-0024). AHRQ Publication no. 09-E004. Rockville, MD: Agency for Healthcare Research and Quality, 2008;175.
13. Verhagen A, de Vet H, de Bi R, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998;51:1235–41. [PubMed]
14. Field TS, Rochon P, Lee M, et al. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. JAMIA 2009;16:480–5. [PMC free article] [PubMed]
15. Gurwitz JH, Field TS, Rochon P, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc 2008;56:2225–33. [PubMed]
16. Grant RW, Wald JS, Schnipper JL, et al. Practice-linked online personal health records for type 2 diabetes mellitus: a randomized controlled trial. Arch Intern Med 2008;168:1776–82. [PubMed]
17. Javitt JC, Rebitzer JB, Reisman L. Information technology and medical missteps: evidence from a randomized trial. J Health Econ 2008;27:585–602. [PubMed]
18. Hicks LS, Sequist TD, Ayanian JZ, et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med 2008;23:429–41. [PMC free article] [PubMed]
19. Martens JD, van der WT, Severens JL, et al. The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial. Int J Med Inf 2007;76:S403–16. [PubMed]
20. Peterson JF, Rosenbaum BP, Waitman LR, et al. Physicians' response to guided geriatric dosing: initial results from a randomized trial. Stud Health Technol Inform 2007;129:2–40. [PubMed]
21. Raebel MA, Charles J, Dugan J, et al. Randomized trial to improve prescribing safety in ambulatory elderly patients. J Am Geriatr Soc 2007;55:977–85. [PubMed]
22. Raebel MA, Carroll NM, Kelleher JA, et al. Randomized trial to improve prescribing safety during pregnancy. JAMIA 2007;14:440–50. [PMC free article] [PubMed]
23. Bailey TC, Noirot LA, Blickensderfer A, et al. An intervention to improve secondary prevention of coronary heart disease. Arch Intern Med 2007;167:586–90. [PubMed]
24. Paul M, Andreassen S, Tacconelli E, et al. Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial. J Antimicrob Chemother 2006;58:1238–45. [PubMed]
25. Fretheim A, Aaserud M, Oxman AD. Rational prescribing in primary care (RaPP): economic evaluation of an intervention to improve professional practice. PLoS Med 2006;3:e216. [PMC free article] [PubMed]
26. Kuilboer MM, van Wijk MA, Mosseveld M, et al. Computed critiquing integrated into daily clinical practice affects physicians' behavior—a randomized clinical trial with AsthmaCritic. Methods Inf Med 2006;45:447–54. [PubMed]
27. Colpaert K, Claus B, Somers A, et al. Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial. Crit Care 2006;10:R21. [PMC free article] [PubMed]
28. Roumie CL, Elasy TA, Greevy R, et al. Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial. Ann Intern Med 2006;145:165–75. [PubMed]
29. McGregor JC, Weekes E, Forrest GN, et al. Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use: a randomized controlled trial. JAMIA 2006;13:378–84. [PMC free article] [PubMed]
30. Palen TE, Raebel M, Lyons E, et al. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care 2006;12:389–95. [PubMed]
31. Feldstein AC, Smith DH, Perrin N, et al. Reducing warfarin medication interactions: an interrupted time series evaluation. Arch Intern Med 2006;166:1009–15. [PubMed]
32. Feldstein A, Elmer PJ, Smith DH, et al. Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial. J Am Geriatr Soc 2006;54:450–7. [PubMed]
33. Berner ES, Houston TK, Ray MN, et al. Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial. JAMIA 2006;13:171–9. [PMC free article] [PubMed]
34. Lester WT, Grant RW, Barnett GO, et al. Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease. J Gen Intern Med 2006;21:22–9. [PMC free article] [PubMed]
35. Raebel MA, Lyons EE, Chester EA, et al. Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial. Arch Intern Med 2005;165:2395–401. [PubMed]
36. Bloomfield HE, Nelson DB, van Ryn M, et al. A trial of education, prompts, and opinion leaders to improve prescription of lipid modifying therapy by primary care physicians for patients with ischemic heart disease. Qual Saf Health Care 2005;14:258–63. [PMC free article] [PubMed]
37. Rosenbloom ST, Geissbuhler AJ, Dupont WD, et al. Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care. JAMIA 2005;12:458–73. [PMC free article] [PubMed]
38. Rasmussen LM, Phanareth K, Nolte H, et al. Internet-based monitoring of asthma: a long-term, randomized clinical study of 300 asthmatic subjects. J Allergy Clin Immunol 2005;115:1137–42. [PubMed]
39. Tierney WM, Overhage JM, Murray MD, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res 2005;40:477–97. [PMC free article] [PubMed]
40. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005;352:969–77. [PubMed]
41. Tierney WM, Overhage JM, Murray MD, et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med 2003;18:967–76. [PMC free article] [PubMed]
42. Rollman BL, Hanusa BH, Lowe HJ, et al. A randomized trial using computerized decision support to improve treatment of major depression in primary care. J Gen Intern Med 2002;17:493–503. [PMC free article] [PubMed]
43. Krall MA, Traunweiser K, Towery W. Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Stud Health Technol Inform 2004;107:1–9. [PubMed]
44. Murray MD, Harris LE, Overhage JM, et al. Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial. Pharmacotherapy 2004;24:324–37. [PubMed]
45. Filippi A, Sabatini A, Badioli L, et al. Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care 2003;26:1497–500. [PubMed]
46. Zanetti G, Flanagan HL, Jr, Cohn LH, et al. Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room. Infect Control Hosp Epidemiol 2003;24:13–16. [PubMed]
47. Flottorp S, Oxman AD, Havelsrud K, et al. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ 2002;325:367–72. [PMC free article] [PubMed]
48. Dexter PR, Perkins S, Overhage JM, et al. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001;345:965–70. [PubMed]
49. Frances CD, Alperin P, Adler JS, et al. Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial. West J Med 2001;175:165–6. [PMC free article] [PubMed]
50. Christakis DA, Zimmerman FJ, Wright JA, et al. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001;107:e15. [PubMed]
51. Montgomery AA, Fahey T, Peters TJ, et al. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ 2000;320:686–90. [PMC free article] [PubMed]
52. Shojania KG, Yokoe D, Platt R, et al. Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. JAMIA 1998;5:554–62. [PMC free article] [PubMed]
53. Evans RS, Classen DC, Pestotnik SL, et al. Improving empiric antibiotic selection using computer decision support. Arch Intern Med 1994;154:878–84. [PubMed]
54. Cobos A, Cobos A, Vilaseca J, et al. Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: report of a cluster-randomized trial. Dis Manag Health Outcome 2005;13:421–32.
55. Davis RL, Wright J, Chalmers F, et al. A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics. PLoS Clin Trials 2007;2:e25. [PMC free article] [PubMed]
56. Dexter PR, Maharry K, Jones K, et al. Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA 2004;292:2366–71. [PubMed]
57. Fiks AG, Hunter K, Localio R, et al. Impact of electronic health record-based alerts on influenza vaccination for children with asthma. Pediatrics 2009;124:159–69. [PubMed]
58. Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice: evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Health Care 1999;17:35–40. [PubMed]
59. Rood E, Bosman R, van der Spoel J, et al. Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation. JAMIA 2005;12:172–80. [PMC free article] [PubMed]
60. Rosser WW, Hutchison B, McDowell I, et al. Use of reminders to increase compliance with tetanus booster vaccination. Can Med Assoc J 1992;146:911–17. [PMC free article] [PubMed]
61. Sequist TD, Gandhi T, Karson A, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. JAMIA 2005;12:431–7. [PMC free article] [PubMed]
62. Van Wyk JT, van Wijk M, Sturkenboom M, et al. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation 2008;117:371–8. [PubMed]
63. Overhage JM, Perkins S, Tierney WM, et al. Controlled trial of direct physician order entry: effects on physicians' time utilization in ambulatory primary care internal medicine practices. JAMIA 2001;8:361–9. [PMC free article] [PubMed]
64. Persell SD, Denecke-Dattalo TA, Dunham DP, et al. Evidence-based medicine. Patient-directed intervention versus clinician reminders alone to improve aspirin use in diabetes: a cluster randomized trial. Jt Comm J Qual Patient Saf 2008;34:98–105. [PubMed]
65. Terrell KM, Perkins AJ, Dexter PR, et al. Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial. J Am Geriatr Soc 2009;57:1388–94. [PubMed]
66. Weir CJ, Lees KR, MacWalter RS, et al. Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke. QJM 2003;96:143–53. [PubMed]
67. Tamblyn R, Huang A, Perreault R, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. Can Med Assoc J 2003;169:549–56. [PMC free article] [PubMed]
68. Owens DK, McDonald TW, Brown BW, et al. A randomized controlled trial of a computer-based physician workstation in an outpatient setting. Implementation barriers to outcome evaluation. JAMIA 1996;3:340–8. [PMC free article] [PubMed]
69. Bertoni A, Bonds D, Chen H, et al. Impact of a multifaceted intervention on cholesterol management in primary care practices. Arch Intern Med 2009;169:678–86. [PMC free article] [PubMed]
70. DEmakis J, Beauchamp C, Cull W, et al. Improving residents' compliance with standards of ambulatory care: results from the VA cooperative study on computerized reminders. JAMA 2000;284:1411–16. [PubMed]
71. Fortuna R, Zhang F, Ross-Degnan D, et al. Reducing the prescribing of heavily marketed medications: a randomized controlled trial. J Gen Intern Med 2009;24:897–903. [PMC free article] [PubMed]
72. Frank O, Litt J, Beilby J. Opportunistic electronic reminders: improving performance of preventive care in general practice. Aust Fam Physician 2004;33:87–90. [PubMed]
73. Javitt J, Steinberg G, Locke T, et al. Using a claims data-based sentinel system to improve compliance with clinical guidelines: results of a randomized prospective study. Am J Manag Care 2005;11:93–102. [PubMed]
74. Musser R, Tcheng J. Quantitative and qualitative comparison of text-based and graphical user interfaces for computerized provider order entry. AMIA Annu Symp Proc 2006:1041. [PMC free article] [PubMed]
75. Overhage J, Tierney W, McDonald C. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996;156:1551–6. [PubMed]
76. Safran C, Rind D, Davis R, et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet 1995;346:341–6. [PubMed]
77. Tierney W, Miller M, Overhage J, et al. Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA 1993;269:379–83. [PubMed]
78. Quinn C, Clough S, Minor J, et al. WellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther 2008;10:160–8. [PubMed]
79. Plaza V, Cobas A, Ignacio-Garcia J, et al. Cost-effectiveness of an intervention based on the recommendations of the Global Initiative for Asthma (GINA) by a computerized clinical decision support: a trial with randomization of physicians. Med Clin (Barc) 2005;124:201–6. [PubMed]
80. Linder JA, Schnipper JL, Tsurikova R, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Inform Prim Care 2009;17:231–40. [PubMed]
81. Bell LM, Grundmeier R, Localio R, et al. Electronic health record-based decision support to improve asthma care: a cluster-randomized trial. Pediatrics 2010;125:e770–7. [PubMed]
82. Graumlich JF, Novotny NL, Stephen NG, et al. Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial. J Hosp Med 2009;4:E11–19. [PubMed]
83. Johnson KB, Ho YX, Cala CM, et al. Showing your work: impact of annotating electronic prescriptions with decision support results. J Biomed Inform 2010;43:321–5. [PubMed]
84. Gilutz H, Novack L, Shvartzman P, et al. Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Isr Med Assoc J 2009;11:23–9. [PubMed]
85. Tamblyn R, Reidel K, Huang A, et al. Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial. Med Decis Making 2010;30:176–88. [PubMed]
86. Gill JM, Chen YX, Glutting JJ, et al. Impact of decision support in electronic medical records on lipid management in primary care. Popul Health Manage 2009;12:221–6. [PubMed]
87. Reeve JF, Tenni PC, Peterson GM. An electronic prompt in dispensing software to promote clinical interventions by community pharmacists: a randomized controlled trial. Br J Clin Pharmacol 2008;65:377–85. [PubMed]
88. Holman RR, Smale AD, Pemberton E, et al. Randomized controlled pilot trial of a hand-held patient-oriented, insulin regimen optimizer. Med Inform (Lond) 1996;21:317–26. [PubMed]
89. Holbrook A, Thabane L, Keshavjee K, et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. Can Med Assoc J 2009;181:37–44. [PMC free article] [PubMed]
90. Ralston JD, Hirsch IB, Hoath J, et al. Web-based collaborative care for type 2 diabetes: a pilot randomized trial. Diabetes Care 2009;32:234–9. [PMC free article] [PubMed]
91. Lo HG, Matheny ME, Seger DL, et al. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. JAMIA 2009;16:66–71. [PMC free article] [PubMed]
92. Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. JAMIA 2008;15:424–9. [PMC free article] [PubMed]
93. Feldstein AC, Smith DH, Perrin N, et al. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med 2006;166:1848–54. [PubMed]
94. Ross SE, Moore LA, Earnest MA, et al. Providing a web-based online medical record with electronic communication capabilities to patients with congestive heart failure: randomized trial. J Med Internet Res 2004;6:e12. [PMC free article] [PubMed]
95. Overhage JM, Tierney WM, Zhou XH, et al. A randomized trial of “corollary orders” to prevent errors of omission. JAMIA 1997;4:364–75. [PMC free article] [PubMed]
96. Meigs J, Cagliero E, Dubey A, et al. A controlled trial of web-based diabetes disease management. Diabetes Care 2003;26:750–7. [PubMed]
97. White K, Lindsay A, Pryor T, et al. Application of a computerized medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol 1984;4:571–6. [PubMed]
98. McDonald C. Use of a computer to detect and respond to clinical events: its effect on clinical behavior. Ann Intern Med 1976;84:162–7. [PubMed]
99. Schnipper JL, Hamann C, Ndumele CD, et al. Effect of an electronic medication reconciliation application and process redesign on potential adverse drug events: a cluster-randomized trial. Arch Intern Med 2009;169:771–80. [PubMed]
100. Clark AS, Shea S. Developing a controlled vocabulary for the Columbia–Presbyterian Medical Center outpatient clinical information system. Proc Annu Symp Comput Appl Med Care 1990:205–9.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association