A Conceptual Framework for Evaluating Outpatient Electronic Prescribing Systems Based on Their Functional Capabilities

doi:10.1197/jamia.M1374

J Am Med Inform Assoc. 2004 Jan-Feb; 11(1): 60–70.

PMCID: PMC305459

A Conceptual Framework for Evaluating Outpatient Electronic Prescribing Systems Based on Their Functional Capabilities

Douglas S. Bell, MD, PhD, Shan Cretin, PhD, Richard S. Marken, PhD, and Adam B. Landman, MS, MIS

Affiliations of the authors: RAND Health, Santa Monica, CA (DSB, SC, RSM, ABL); Division of General Internal Medicine and Health Services Research, Department of Medicine, University of California, Los Angeles, School of Medicine, Los Angeles, CA (DSB). Dr. Cretin is currently Regional Director, American Friends Service Committee, Pasadena, CA; Mr. Landman is currently a medical student, UMDNJ-Robert Wood Johnson Medical School, New Brunswick, NJ.

Correspondence and reprints: Douglas S. Bell, MD, PhD, UCLA Division of General Internal Medicine and Health Services Research, 911 Broxton Plaza, Room 314, Los Angeles, CA 90095-1736; e-mail: dbell/at/ucla.edu.

Received March 31, 2003; Accepted September 19, 2003.

This article has been cited by other articles in PMC.

Abstract

Objective: Electronic prescribing (e-prescribing) may substantially improve health care quality and efficiency, but the available systems are complex and their heterogeneity makes comparing and evaluating them a challenge. The authors aimed to develop a conceptual framework for anticipating the effects of alternative designs for outpatient e-prescribing systems.

Design: Based on a literature review and on telephone interviews with e-prescribing vendors, the authors identified distinct e-prescribing functional capabilities and developed a conceptual framework for evaluating e-prescribing systems' potential effects based on their capabilities. Analyses of two commercial e-prescribing systems are presented as examples of applying the conceptual framework.

Measurements: Major e-prescribing functional capabilities identified and the availability of evidence to support their specific effects.

Results: The proposed framework for evaluating e-prescribing systems is organized using a process model of medication management. Fourteen e-prescribing functional capabilities are identified within the model. Evidence is identified to support eight specific effects for six of the functional capabilities. The evidence also shows that a functional capability with generally positive effects can be implemented in a way that creates unintended hazards. Applying the framework involves identifying an e-prescribing system's functional capabilities within the process model and then assessing the effects that could be expected from each capability in the proposed clinical environment.

Conclusion: The proposed conceptual framework supports the integration of available evidence in considering the full range of effects from e-prescribing design alternatives. More research is needed into the effects of specific e-prescribing functional alternatives. Until more is known, e-prescribing initiatives should include provisions to monitor for unintended hazards.

Prescription medications are central to health care, accounting for 13% of health care expenditures and being used by 65% of the U.S. public in a given year.¹ Errors in the use of prescription medications are common, and they often result in patient injuries.²^,³^,⁴ Many of these injuries would be preventable with better information management.⁵ Electronic prescribing (e-prescribing), which we define as clinicians' computerized ordering of specific medication regimens for individual patients, offers the potential to substantially reduce medication errors and also to improve health care efficiency. However, some e-prescribing efforts have met unexpected challenges,⁶^,⁷ and faced with the uncertainties, few provider organizations have adopted e-prescribing.⁸^,⁹ Nonetheless, inpatient e-prescribing is now being encouraged by a coalition of large employers,¹⁰ and provisions for outpatient e-prescribing are likely to be part of a new Medicare prescription drug benefit.¹¹^,¹² Provider organizations that choose to adopt e-prescribing can select from a wide variety of commercial e-prescribing options, but they have no coherent framework to guide their choices.

A few studies have found benefits from e-prescribing, but so far these evaluations have involved only “home-grown” systems implemented by “pioneers” at academic medical centers.¹³ New studies will likely produce additional evidence for effects of commercial systems, but even large investments in controlled trials would be unlikely to produce direct evidence for every type of prescribing system in every type of health care organization.

We sought to develop a conceptual framework for comparing the potential benefits and risks of e-prescribing systems based on their component functional capabilities. This effort is intended as a step toward a design theory¹⁴ for e-prescribing that would consist of empirically grounded principles for predicting the effects of design alternatives within specific clinical environments. This approach is consistent with guidance on the evaluation of complex health interventions, which calls for using a conceptual framework to model how the components of an intervention contribute to its overall effects.¹⁵^,¹⁶ Though the primary focus of our conceptual framework is on evaluating outpatient e-prescribing systems, many of the same principles should also apply to inpatient systems. In identifying specific functional capabilities, we focused in particular on features that would influence the prescribing step of medication management.¹⁷

Background

Failures of Current Medication Management Processes

Recognizing that medications can cause harm as well as benefits, federal law requires many medications to be dispensed only with a prescription from a licensed practitioner.¹⁸^,¹⁹ However, even with this basic safety mechanism in place, patients remain at significant risk of injury from the erroneous use of medications. Injuries from medications in general are termed “adverse drug events” (ADEs), and the subset that are due to errors are termed “preventable ADEs.” The health benefits of an e-prescribing system would be a function of its ability to reduce preventable ADEs and increase the appropriate use of medications in comparison with the existing medication management processes that the system would replace.

Net Risk of Preventable ADEs

shows preventable ADE rates found in studies of traditional inpatient and outpatient settings. The table also highlights methodological differences that might contribute to the variation observed among these estimates. An early study, which followed both inpatient and outpatient general internal medicine patients, found that 4.3% of patients experienced ADEs, 83% of which resulted from outpatient prescriptions. The majority of ADEs were preventable and at least serious in severity (

, first row).²⁰ Other studies, which have focused on inpatients or outpatients only, have found preventable ADE risks ranging from 0.4% of hospital admissions²¹ to 3.0% of outpatients within three months of receiving a prescription.²² As shown in the table, studies that found higher ADE rates also tended to include less-serious events. Other studies that reviewed emergency department charts found potential drug interactions or allergies in 4% to 21% of visits.²³^,²⁴^,²⁵ These studies are not included in

because they did not assess how often errors led to ADEs, but they suggest that the preventable ADE risk is also high in emergency departments.

Table 1

Net Risks of Preventable Adverse Drug Events (ADEs) among Studies Having a Defined Base Population

An additional group of studies has used emergency department visits or hospital admissions as sentinel events to identify possible outpatient ADEs. A study of 62,216 emergency department visits found that 1.7% resulted from outpatient ADEs.²⁶ A meta-analysis of 36 studies concluded that 5% of hospital admissions resulted from outpatient ADEs, with 23% of these due to patient errors.²⁷ Because about 7% of the U.S. population is hospitalized annually,²⁸ this study suggests that the population risk of being hospitalized due to an outpatient ADE is on the order of 0.35%. Since only 70% of the population receives outpatient care annually,²⁸ an outpatient's risk of an ADE requiring hospitalization would be about 0.5%, an estimate consistent with the serious ADE rates found in other studies.

In addition to their health impacts, ADEs increase work and resource consumption. In a study of inpatient ADEs, patients who suffered preventable ADEs had 4.6 additional hospital days and $5,857 in additional costs, compared with matched control patients on the same unit.²⁹ The study's authors projected that for a 700-bed teaching hospital, preventable ADEs led to additional direct costs of $2.8 million per year, exclusive of any malpractice losses or additional costs to patients. Another analysis found that few patient-level risk factors for ADEs could be identified, indicating that improved medication management is needed for all patients rather than select subgroups.³⁰

Nature and Causes of Preventable ADEs

For e-prescribing systems to optimally reduce ADEs, their design elements should be matched with the types of errors made when manual prescribing processes are used.¹⁴ A few studies have provided data on the nature and causes of preventable ADEs, though at varying levels of detail.

shows how errors that contributed to ADEs were distributed across stages of the medication management process, among studies that provided these data. Overall, prescribing, administration, and monitoring appear to present much larger risks than dispensing. An additional study has reported that 45% of preventable ADEs were related to errors in medication choice, with another 20% being due to errors in dose or frequency.²²

Table 2

Distribution of Errors Associated with Preventable Adverse Drug Events (ADEs) across Stages of Medication Management

The root causes of medication errors have been cataloged by two studies, both set in the inpatient environment. Though one study included only pharmacist-intercepted medication errors³¹ and the other used multiple methods to identify both intercepted and non-intercepted errors,⁵ the most common root causes were similar, as shown in

. In addition to the causes identified in the studies, illegible handwriting is also often cited anecdotally as a cause of medication errors. Though neither study found handwriting to be a frequent root cause, a 1994 report by the American Medical Association found that misinterpreted prescriptions were the second most prevalent and expensive malpractice claim category.³² Two hospital-based studies found legibility problems with 20% to 33% of orders.³³^,³⁴ Pharmacists usually avert harm from illegible prescriptions by calling the prescriber, but when illegible prescriptions are simply misread, the wrong medication is dispensed without a callback. In one fatal case, a prescription for Isordil (a long-acting nitrate for angina) was misread and dispensed as Plendil (an antihypertensive).³⁵ Drug names that look or sound like those of other commonly used drugs are the most susceptible to misinterpretation.³⁶

Table 3

Root Causes of Medication Errors and the Resulting Error Types in Two Inpatient Studies

Effects of Electronic Prescribing Systems

E-prescribing is one form of computer-based physician order entry (CPOE), which also includes computer-based orders for tests and other medical interventions. CPOE has been available in some hospital systems since the 1970s,³⁷ but it is seldom used by physicians.⁸ Evidence for the effects of CPOE is limited,¹³ but the available studies suggest that CPOE can improve patient safety and reduce hospital costs. The Regenstrief Medical Record System, at Wishard Memorial Hospital in Indianapolis, includes inpatient CPOE with drug-interaction and allergy warnings.³⁸ In a randomized controlled trial, inpatient medical teams assigned to CPOE generated 12.7% lower charges and a 0.9-day shorter length of stay than teams using handwritten orders.³⁹ The HELP system,⁴⁰ at LDS Hospital in Salt Lake City, has been associated with reductions in allergy, dosing, and antibiotic selection errors,⁴¹ and with reduced errors of omission for preoperative antibiotics.⁴² An inpatient CPOE system at Brigham and Women's Hospital in Boston included alerts for drug interactions, allergies, redundant medications, and relevant laboratory results. Following its implementation, the rates of serious medication errors (preventable ADEs + potential ADEs) fell by 55%.⁴³ After subsequent improvements, the rate of non-intercepted serious medication errors fell further, to a rate that was 86% below the baseline.⁴⁴ In contrast, a few distinctly negative experiences with CPOE have also been described. The University of Virginia Medical Center in Charlottesville experienced a houseofficer work-action in response to a new CPOE system, resulting in the system's being taken offline until user demands were addressed.⁶ Recently, a CPOE system at Cedars-Sinai Medical Center in Los Angeles was taken offline due to physician complaints.⁷

Evidence for the effects of outpatient e-prescribing is much more limited. PRODIGY, an outpatient system developed in Britain, involves prescribing from a list of medication/dose combinations recommended for the diagnoses entered on an encounter note. Preliminary studies have shown modest improvements in adherence to guidelines.⁴⁵ In a small study of outpatient ADEs, use of an elementary e-prescribing system, which did not include standardized medication or dose selection, was not associated with differences in preventable ADE rates.⁴⁶

Model Formulation Process

Literature Review

To systematically identify published literature related to the potential effects of e-prescribing, we searched the Medline database, including articles from 1966 through 2001, using combinations of the following terms: Medication errors; Drug therapy, computer assisted; Drug interactions; (Prescriptions, drug OR pharmaceutical preparations); (Reminder systems OR point of care systems); (Computer^* OR Internet OR software OR online systems); (Clinical pharmacy information systems OR medication systems). Unique citations were combined into a single data set (n = 2,591). The title and journal name of each citation in the data set were manually reviewed, and 1,609 citations were classified as “not relevant” to understanding the effects of electronic prescribing, leaving 982 potentially relevant articles. We reviewed abstracts from this set and retrieved full-text articles that might contribute to hypotheses about the effects of e-prescribing systems.

The literature review identified two models that have been used for analyzing failures of medication management processes. A model used by the Joint Commission for Accreditation of Healthcare Organizations considered five steps in medication management: prescribing, dispensing, administering, monitoring, and systems control.¹⁷ A model used by the Adverse Drug Event Prevention Study (ADEPS) considered four steps in medication management: ordering, transcription, dispensing, and administration.⁵ We formed a process flow model that integrated the steps of these two models, recasting the “transcription” step from the ADEPS model as the “transmission” step in our model in order to encompass either handwritten or electronic prescribing.

Review of Existing Systems' Functional Capabilities

After viewing live demonstrations of three commercial e-prescribing systems, we designed a structured telephone interview to elicit from vendor representatives the important functional capabilities offered in their e-prescribing product. A list of 127 companies offering electronic prescribing was abstracted from a published resource guide.⁴⁷ Based on an examination of each company's Website by at least two reviewers, we determined that 57 of these companies were resellers or otherwise not in the business of producing an original e-prescribing product. We telephoned the remaining companies and conducted a structured interview with a company representative who was qualified to answer technical questions about their e-prescribing products. The interview script (Appendix, available as an online data supplement at www.jamia.org) focused primarily on eliciting features of the prescribing step. Vendors that were difficult to reach by telephone were also contacted using a structured e-mail request for information. The project was approved by institutional review boards at RAND and UCLA.

The vendor interviews identified 56 distinct e-prescribing products from 49 companies. Results of the interviews were compiled into a list of major functional capabilities, which was then organized using our process model of medication management. Evidence for the effects of these functional capabilities was sought in the literature review results. Additional hypotheses were generated for possible effects of functional capabilities.

Model Description

Process Model of Medication Management

shows the process model of medication management that we propose for organizing prescribing system evaluations. The model is expressed at a level intended to make it applicable for both handwritten and electronic prescribing, across patient settings. Central to the model are the five major activities involved in medication management: prescribe, transmit, dispense, administer, and monitor. For each activity, the diagram shows inputs, outputs, resources that may be occupied (which are drivers of the activity's direct costs), and information that may regulate the activity (which largely influences its quality).

Figure 1.

A function model of medication management. The major activities involved in medication management are shown as boxes. For each box, arrows on the left show the activity's inputs, those on the right show its outputs, those above show information that may (more ...)

The “prescribe” activity is defined, in part, by the involvement of a prescribing clinician. The other major resources that may be used include paper records and e-prescribing systems. The activity's mandatory inputs are a clinician's assessments about the need for prescription medications. The patient's preferences regarding medications are also a desirable input. The “prescribe” activity may be informed by drug information, patient data (such as known allergies), and drug formulary restrictions, which may be available from print or electronic resources. The activity's mandatory output is a completed prescription. The prescribing step may also output information that helps the patient adhere to the prescribed regimen.

The “transmit” activity delivers a completed prescription for fulfillment. In traditional outpatient environments, patients would often complete this step without assistance, but clinicians or office staff may also participate, for example by telephoning prescriptions. The “dispense” activity involves a pharmacist except when medications are dispensed in the clinician's office. Pharmacists may also use information systems, and they may access the same types of information used in the prescribing step. Problematic prescriptions may require a call to the clinician; as a result, prescriptions may be changed or cancelled rather than being dispensed. The “administer” activity always involves the patient, at least as recipient, and it could involve a range of health care providers in some settings. “Monitoring” as we define it always involves the patient, at least as the subject of observation, and the clinician, whose assessments feed back into prescription changes.

E-prescribing Functional Capabilities

An e-prescribing system would influence the quality and efficiency of prescribing through functional capabilities intended to alter aspects of each medication management activity.

summarizes potential effects from 14 individual e-prescribing functional capabilities. We identified seven of these in existing outpatient e-prescribing systems; seven others were identified only in the literature review. This section reviews the potential effects of each functional capability.

Table 4

Potential Effects of Individual Electronic Prescribing Functional Capabilities

Patient selection, often from a menu of patients in the clinician's practice, is usually the first e-prescribing step. The literature review did not identify specific evidence for the effects of this feature. However, since users often slip⁴⁸ when selecting from menus, the wrong patient may be inadvertently selected. Unless the system makes these slips easy to detect and correct, patient selection menus could increase the likelihood of wrong-patient errors. In the outpatient environment, wrong-patient prescriptions transmitted to a mail-order pharmacy could cause injury for patients who aren't vigilant. Patient selection could also increase or decrease clinicians' labor, depending on its workflow integration.

Diagnosis selection, when it is used to initiate prescribing, enables reminders that can increase adherence to practice guidelines, as demonstrated by the PRODIGY system⁴⁵ and others.⁴⁹ However, if clinicians are forced to enter a diagnosis for each prescription, they may enter incorrect or embellished diagnoses to access the medications they want to prescribe. This inaccurate information could then interfere with the system's subsequent performance. Furthermore, the diagnostic codes used for billing often fail to express clinically important distinctions⁵⁰; thus, relying on the same codes to guide billing and prescribing may compound inaccuracies. Requiring diagnosis selection would also add a step to the prescribing process, possibly increasing the clinician's labor unless this step also contributes to documentation and billing requirements. Users of the PRODIGY system, which combines diagnosis documentation with prescribing, perceived that the system made office visits longer, but the measured duration of visits was no different.⁴⁵

Selecting medication regimens using menus decreases wrong-dose errors by disallowing invalid combinations. The structured prescription data obtained also enable other safety measures such as alerts. With the institution of menu-based CPOE at Brigham and Women's Hospital, the proportion of orders with doses above the recommended maximum fell from 2.1% to 0.6%.⁵¹ However, errors in intravenous potassium orders markedly increased, returning to pre-CPOE levels only after potassium-ordering screens were modified.⁴⁴ This finding demonstrates that flawed prescribing menus can introduce new errors. Look-alike medication errors⁵²^,⁵³ might potentially increase if clinicians select an adjacent name on an alphabetized list and complete the prescription without recognizing the error. The resulting well-formed prescriptions would be very difficult for pharmacists to recognize as an error and intercept. Finally, menu-based prescription writing may increase physician labor. At Brigham and Women's, CPOE took substantially longer than handwriting, and the extra time was offset only modestly by time saved in looking for charts and responding to pharmacy calls.⁵⁴ CPOE with the Regenstrief System initially took twice as long as handwritten ordering,⁵⁵ but ordering speed improved substantially with experience.⁵⁶

Safety alerts may reduce the risks of preventable ADEs considerably. Following the institution of CPOE at Brigham and Women's Hospital, 50 to 80 orders per day were changed because of allergy alerts.⁴⁴ Alerts from the HELP system also dramatically reduced allergy and drug selection errors.⁴¹ However, drug interaction programs vary in their sensitivity and specificity,⁵⁷ and the thresholds set for triggering alerts could significantly influence their safety impact and costs. Highly sensitive systems that generate many “false alarms” could lead users to disregard future alerts; signs of this extinguishing effect have been observed.⁵⁸ Overly specific alerting could also degrade performance if prescribers expect their errors to be detected and review their prescriptions less carefully. Incomplete data on the patient's current regimen would also lead to alerting failures, even by systems set for high sensitivity. Thus, the adequacy of alerting could be particularly dependent on the prescribing system's integration with electronic medical records, not only within the prescribing clinician's practice but also with records from other clinicians who care for the patient.

Formulary alerts can also influence prescribing significantly. At Brigham and Women's, adherence to the formulary for intravenous H2 blockers increased from 14% to 88% with the implementation of a formulary alert.⁵¹ In most outpatient settings, formulary alerts would primarily reduce the work that results when payment is rejected at the pharmacy. To the extent that patients pay out of pocket rather than waiting for a prescriber to respond to a formulary rejection, formulary alerts could save money and reduce hassle for patients. Finally, formulary alerts may or may not reduce overall health care costs, since restrictive formularies can result in expenditures' being shifted from drugs to other health care services.⁵⁹

Computer-assisted dose calculations, which may rely on patients' body size, age, renal function, and other metabolic parameters, have increased prescribing accuracy in several studies.⁴⁹^,⁶⁰ The addition of automated renal function monitoring to the HELP system was associated with reductions in excessive antibiotic doses and in antibiotic-related ADEs.⁶¹ However, appropriate calculations may require e-prescribing systems to have access to medical record data such as age, body weight, and laboratory test results that reflect renal and hepatic function. If these data are only intermittently available, the resulting intermittent failure of dosing calculations could potentially increase dosing errors.

Transmitting data electronically from prescribing systems to pharmacies should eliminate human transcription errors, generally improving safety and efficiency. However, errors or clinician work might increase if transmissions are unreliable or if prescribing data must be re-entered manually at the pharmacy because of incompatible systems. Though this step is the least frequent source of preventable ADEs (

), developing the required linkages with pharmacy systems may facilitate systems integration, resulting in more complete patient data being available to drive alerts and reminders.

Dispensing from physicians' offices is possible with some e-prescribing systems, potentially eliminating community pharmacists from the outpatient medication management process. If the prescribing system is less capable than pharmacists are at intercepting medication errors or educating patients, or if physicians are induced to prescribe from an in-office inventory, then drug-choice and patient-administration errors might increase.

The administration step is a frequent source of errors and might also be improved by e-prescribing functions. Well-designed educational materials should reduce outpatients' self-administration errors. Systems that facilitate physician, nurse, and pharmacist collaboration in patient education could further augment adherence.⁶² However, poorly designed patient educational materials may result in inconsistent instructions or misunderstandings that actually increase errors. Systems that help schedule and track drug administration should also reduce administration errors. Currently, inpatient systems usually feature drug administration tracking, but outpatient systems could also foster tracking, for example, by providing patients with printed daily administration schedules.

Features that enhance monitoring may be particularly important for outpatient prescribing, given the frequency of errors found at this step.⁶³ Automatic orders for laboratory monitoring tests more than doubled physicians' adherence to recommended testing.⁶⁴ Prescribing systems with access to pharmacy data could inform the clinician when patients fail to fill prescriptions on time, allowing clinicians to address patient nonadherence, a task that physicians currently perform poorly.⁶⁵ Prescribing systems might also improve monitoring by administering patient questionnaires that detect important adverse reactions and trigger more timely appropriate action.⁶² Although clinicians might need to devote additional time to monitoring tasks, their time might be saved overall if this information led to better patient communication. However, in most clinical settings, each of these monitoring capabilities would require extensive integration with external information systems, thus making them possibly less achievable in the near term.

Applying the Conceptual Framework

To apply the proposed conceptual framework in evaluating specific e-prescribing systems, an evaluator would first use the process model (

) to assess prescribing in the intended clinical environment at baseline, before the implementation of any new systems. Then, for each activity in the process model, the evaluator would list the functional capabilities of the proposed e-prescribing system that might influence the activity's outputs. These capabilities might include those shown in

as well as other features that may be novel among e-prescribing systems. Finally, each functional capability's potential effects would be hypothesized, using evidence from the literature where it is available.

Validation by Example

As an example of applying this conceptual framework, we assess the potential effects of two specific, unnamed e-prescribing systems in a community general internal medicine office.

summarizes the comparison of each system with the baseline features of medication management in the practice before e-prescribing.

Table 5

Summary of Results from Applying the Conceptual Framework to Evaluate Two Alternative Outpatient Electronic Prescribing (E-Rx) Systems

In the baseline environment of this office, a practice management system produces stickers that physicians use to identify the patient on paper prescriptions. Patient history information (e.g., allergies) may be available directly from the patient or in handwritten medical records, though the medical record is sometimes not available due to filing problems. Medications and dosages are then handwritten, based on drug information in the clinician's memory and in reference books available in the examination room. Drug formulary information is not readily accessible. Explicit drug safety checking is performed occasionally, using the available reference information. Clinicians' labor is the primary resource used. Transmitting prescriptions to the pharmacy relies largely on patients' labor, but prescription refills are commonly handled by telephone. Errors are rarely introduced at this step. Dispensing takes place at a variety of community pharmacies, where third-party formulary adherence is enforced. Formulary mismatches are common for new prescriptions, resulting in additional labor for pharmacists and physicians. Pharmacy computer systems also perform checks for drug interactions, but interactions are rarely discovered. Patients self-administer medications based largely on verbal instructions from the pharmacist and the physician. Monitoring relies on physicians' or patients' initiative. Overall, medication management in this environment likely leads to preventable ADEs at rates similar to those summarized in

for outpatient prescribing.

One e-prescribing alternative is a standalone system that uses a handheld computing platform. Prescribing begins with clinicians' selection of a patient from their schedule, which can be loaded daily into the e-prescribing system from a practice management system. The patient selection process is quick, but the selected patient's name is not displayed through subsequent prescribing screens. Thus, the system misses an opportunity to help clinicians intercept wrong-patient errors. The system does not support diagnosis selection or diagnosis-based suggestions, removing the potential positive and negative effects of that feature from consideration. The system does support medication selection menus that enforce appropriate doses, thus likely decreasing wrong-dose errors. The system also indicates the formulary status of medications within the menus, likely reducing the clinician's time spent on pharmacy callbacks. Safety alerts and dosage calculations are not supported, thus omitting a large category of potential safety benefits. The system can transmit prescriptions to local or mail-order pharmacies via electronic fax, possibly saving clinicians and staff time on the telephone, at the cost of extra labor in choosing a pharmacy and dealing with failed transmissions. At the pharmacy, faxed prescriptions would still need to be manually entered into an information system; thus the rate of errors at this step may be unchanged. The system has none of the capabilities shown in

that could support the dispensing, administration, or monitoring steps. In summary, this system offers some potential benefits in decreasing wrong-dose errors and decreasing time spent on formulary-related telephone communications, but it might increase the risk of wrong-patient errors and it omits many possibly beneficial features, thereby missing many potential benefits as well as some risks.

The other e-prescribing alternative is an integrated module of an electronic medical record system. The system is intended for use during the clinical encounter, replacing the paper medical record. The patient's identity is selected from a list of patients in the practice, not from the clinician's schedule, since the system does not interface with the office's practice management system. Thus, patient selection might be slightly less efficient for the prescriber; however, the patient's identity is displayed in the interface throughout, likely decreasing wrong-patient errors. Diagnosis selection is performed as part of documenting the current encounter, and the system provides diagnosis-based reminders, likely improving guideline adherence. However, clinicians may prescribe without a diagnosis, and they are not limited to the medications recommended for the diagnosis. Medication selection is generally quick, but clinician labor may be slightly increased by the extra step involved in linking documentation with prescribing. Medications and doses are selected from menus presenting only valid combinations, likely decreasing wrong-dose errors. The system provides safety alerts based on allergies and drug–drug interactions, but not body size, age, drug–disease, or drug–laboratory interactions. Current internist users estimate that about 10% of prescriptions generate an alert and that 10% of those are useful, suggesting that the specificity of alerts may be relatively low. Nonetheless, these alerts would likely prevent some proportion of drug-choice errors, at a cost of increased clinician labor. The system provides no formulary information, thus missing an opportunity to prevent pharmacy calls. For the remaining steps of medication management, the system is no different from the first system reviewed—completed prescriptions can be transmitted to pharmacies by electronic fax, and the system has no capabilities to support dispensing, administration, or monitoring. In summary, this system's integration with an electronic medical record enables features that would likely reduce drug-choice errors and omission errors, compared with the current environment or with the first e-prescribing system reviewed. However, this system may have fewer benefits for office efficiency, given the apparent low specificity of alerts and the lack of formulary information that could prevent calls from pharmacies.

Discussion

Health care organizations have compelling reasons for interest in outpatient e-prescribing. The evidence summarized in

and

indicates that a primary care provider with 3,000 patients could expect 45–90 preventable ADEs per year among his or her patients, with more than half of these resulting from a prescribing error. In academic hospitals, CPOE with e-prescribing has been associated with dramatic improvements in medication errors and in guideline adherence. However, the generalizability of these findings to commercial systems and community outpatient settings remains uncertain.¹³ Moreover, the structural and cultural variance among health care organizations combined with the variance in functional capabilities among e-prescribing systems will likely make any evidence about overall e-prescribing effects difficult to generalize.

This paper describes a conceptual framework for evaluating e-prescribing that accounts for variation among clinical settings and among e-prescribing systems. Using our approach, evaluators first construct a version of the medication management process model (

) that is localized for the intended patient care setting. Next, they identify the e-prescribing functional capabilities that might influence each activity in the process (

). Finally, they consider the effects from each functional capability, integrating specific evidence where it is available. Since e-prescribing systems can introduce errors that did not exist with handwritten prescribing,⁴⁴ the framework specifically guides users to consider both the positive and the negative effects of each system feature.

Our framework organizes potential e-prescribing features using a process model of the activities in medication management. We believe that this organization of the framework provides a comprehensive view of a system's potential effects within its environment, using a sequence of activities that are familiar to most clinicians and administrators. Other investigators have used a similar process model to simulate the results that could be expected from different degrees of ADE reduction within each medication management activity.⁶⁶^,⁶⁷ Our approach differs in that it opens the “black box” of each activity to consider how system design features would alter the activity.

One limitation of the proposed conceptual framework is the relative paucity of evidence available to support judgements about the effects from different e-prescribing design features. Although we found some evidence to support 8 specific effects for six of the 14 functional capabilities we catalogued (five effects related to safety, one to formulary adherence, and two to clinician labor), all of this evidence needs to be considered preliminary given the limited number of studies. Overall, the evidence for functional capabilities is not yet sufficient to support quantitative modeling of effects. Thus, the statements resulting from an application of our framework to individual vendor systems need to be considered hypotheses—of varying strength—rather than conclusions.

Users of our model should also be aware that the 14 functional capabilities we identified do not represent an exhaustive list. In identifying functional capabilities, we focused primarily on the “prescribing” step of medication management. Capabilities that we have not explicitly considered, especially involving later steps of the process model, may also be important in determining the effects of e-prescribing systems. However, the proposed evaluation framework has the virtue of being able to accommodate additional functional capabilities that we have not considered.

Though the current evidence is incomplete, it nonetheless supports a set of e-prescribing design priorities. Because errors in medication selection are most common, functional capabilities such as medication selection menus and safety alerts may be the most important for preventing ADEs. Because the administration and monitoring steps are also common sources of outpatient preventable ADEs, adequate support of outpatient education and follow-up monitoring may also reduce health risks significantly. Because the transmission and dispensing steps accounted for fewer than 5% of preventable ADEs, electronic transmission of prescriptions may have a lesser impact on patient safety than it does on process efficiency. When evidence is incomplete, a Delphi panel process can help to codify expert opinion about best practices.⁶⁸ We are currently conducting an expert panel process to develop quality standards for e-prescribing.

Another limitation in the evidence base is the lack of information about factors that can lead to unintended hazards. The experience with potassium ordering screens at Brigham and Women's Hospital⁴⁴ demonstrated that occult hazards can exist in e-prescribing systems, but the specific factors that caused this hazard were not identified. Analysis of the Therac-25 radiation therapy accidents found that rare, fatal overdoses arose from interactions between software and hardware flaws coupled with frequent, cryptic error messages to which users became insensitive.⁶⁹ Some unintended hazards may be prevented by better software engineering and evaluation,⁷⁰ but identifying rare hazards may still depend on user vigilance.⁶⁹ Future work integrating the functional capability approach with cognitive models of human errors⁷¹^,⁷² may prove valuable for predicting user interface hazards. Until then, our framework may still help evaluators to systematically consider possible hazards by applying basic principles of user interface design⁷³ when examining each system capability. More importantly, implementations of e-prescribing should be accompanied by specific efforts to monitor for and respond to unexpected quality and efficiency problems.

Clearly, more research is needed in e-prescribing effects. Given the high costs of controlled trials, some experts have argued that less-rigorous evidence must be accepted for safety-related information technology (IT).⁷⁴ However, others have countered that the harms potentially arising from IT interventions justify significant investments in evaluation before the nation commits to the high costs of implementation.⁷⁵ We posit that experimentation might focus most productively on alternative functional capabilities within different clinical environments rather than on e-prescribing vs. traditional prescribing. As more complete evidence is accumulated, our framework may serve as a starting point for quantitative models. Future work might also expand the framework to deal with other aspects of CPOE, such as computerized test ordering and results retrieval.

Finally, little information is available about the costs of creating and maintaining e-prescribing systems. These costs would likely be greatest for institutions starting with less-developed informatics infrastructures. Whether e-prescribing systems will be cost-saving or whether their potential health benefits will require additional net investments in health care remains unknown. Future studies of e-prescribing systems will be most useful if they can capture implementation costs.

Conclusions

We propose a conceptual framework for evaluation of e-prescribing that facilitates the consideration of systems' functional capabilities in projecting their expected effects. The framework uses a process model of medication management to comprehensively identify the relevant functional capabilities. The 14 functional capabilities we have identified through our review of the literature and of existing systems can serve as an initial list of features to look for. However, rather than stopping with a simple checklist, the proposed framework guides the integration of existing evidence in judging the potential effects of each feature within a specific health care environment.

This proposed methodology of focusing assessments on functional capabilities may have utility for assessing a broader range of information systems. Future research should focus on collecting evidence for the effects of functional capabilities and also on the factors that may lead to unintended hazards from e-prescribing. The collection of evidence at the level of functional capabilities may provide a way to generalize results from one system evaluation to another.

Supplementary Material

Appendix

Click here to view.

Notes

Supported by a grant from Pfizer, Inc.

References

1. Agency for Healthcare Research and Quality. MEPS Highlights #11: Distribution of Health Care Expenses, 1999 [web page]. 2001; available at: http://www.meps.ahrq.gov/papers/hl11_00-0024/hl11.htm. Accessed Sept 19, 2002.

2. Melmon KL. Preventable drug reactions—causes and cures. N Engl J Med. 1971;284:1361–8. [PubMed]

3. Steel K, Gertman PM, Crescenzi C, Anderson J. Iatrogenic illness on a general medical service at a university hospital. N Engl J Med. 1981;304:638–42. [PubMed]

4. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA. 1995;274:29–34. [PubMed]

5. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA. 1995;274:35–43. [PubMed]

6. Massaro TA. Introducing physician order entry at a major academic medical center: I. Impact on organizational culture and behavior. Acad Med. 1993;68:20–5. [PubMed]

7. Ornstein C. Hospital Heeds Doctors, Suspends Use of Software; Cedars–Sinai physicians entered prescriptions and other orders in it, but called it unsafe. The Los Angeles Times. Jan 22, 2003; Sect B:1.

8. Ash JS, Gorman PN, Hersh WR. Physician order entry in U.S. hospitals. Proc AMIA Symp. 1998:235–9. [PMC free article] [PubMed]

9. The Leapfrog Group. Hospital Survey Results Summary [web page]. Jan 17, 2002; available at: http://www.leapfroggroup.org/Briefing/ResultsSummary011702.pdf. Accessed Mar 10, 2003.

10. The Leapfrog Group. Patient Safety: Setting Standards: An Rx for Rx [web page]. Available at: http://www.leapfroggroup.org/safety1.htm# CPOE. Accessed Oct 6, 2002.

11. U.S. Senate. 108th Congress. S. 1, Prescription Drug and Medicare Improvement Act of 2003. Available at: http://thomas.loc.gov/cgi-bin/bdquery/z?d108:s.00001 [passed: Jun 27, 2003].

12. U. S. House. 108th Congress. H.R. 1, Medicare Prescription Drug and Modernization Act of 2003. Available at: http://thomas.loc.gov/cgi-bin/bdquery/z?d108:HR00001 [passed: Jun 27, 2003].

13. Kaushal R, Bates DW. Computerized Physician Order Entry (CPOE) with Clinical Decision Support Systems (CDSSs). Shojania KG, Duncan BW, McDonald KM, Wachter RM, eds. Making Health Care Safer: A Critical Analysis of Patient Safety Practices. Washington, DC: Agency for Healthcare Research and Quality, 2001; AHRQ Publication 01-E058. AHRQ Evidence Report/Technology Assessment: Number 43.

14. Simon HA. The science of design: creating the artificial. In: Simon HA. The Sciences of the Artificial. Cambridge, MA: MIT Press; 1996.

15. Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–6. [PMC free article] [PubMed]

16. MRC Health Services and Public Health Research Board. A framework for the development and evaluation of complex interventions to improve health [web page]. April 2000; available at: http://www.mrc.ac.uk/pdf-mrc_cpr.pdf. Accessed Mar 31, 2002.

17. Nadzam DM. Development of medication-use indicators by the Joint Commission on Accreditation of Healthcare Organizations. Am J Hosp Pharm. 1991;48:1925–30. [PubMed]

18. Federal Food, Drug, and Cosmetic Act. 21 U.S.C. 9: 353(b).

19. Wax PM. Elixirs, diluents, and the passage of the 1938 Federal Food, Drug and Cosmetic Act. Ann Intern Med. 1995;122:456–61. [PubMed]

20. Burnum JF. Preventability of adverse drug reactions [letter]. Ann Intern Med. 1976;85:80–81. [PubMed]

21. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard-Medical Practice Study II. N Engl J Med. 1991;324:377–84. [PubMed]

22. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003;348:1556–64. [PubMed]

23. Beers MH, Storrie M, Lee G. Potential adverse drug interactions in the emergency room. An issue in the quality of care. Ann Intern Med. 1990;112:61–4. [PubMed]

24. Goldberg RM, Mabee J, Chan L, Wong S. Drug–drug and drug–disease interactions in the ED: analysis of a high-risk population. Am J Emerg Med. 1996;14:447–50. [PubMed]

25. Langdorf MI, Fox JC, Marwah RS, Montague BJ, Hart MM. Physician versus computer knowledge of potential drug interactions in the emergency department. Acad Emerg Med. 2000;7:1321–9. [PubMed]

26. Schneitman-McIntire O, Farnen TA, Gordon N, Chan J, Toy WA. Medication misadventures resulting in emergency department visits at an HMO medical center. Am J Health Syst Pharm. 1996;53:1416–22. [PubMed]

27. Einarson TR. Drug-related hospital admissions. Ann Pharmacother. 1993;27:832–40. [PubMed]

28. Agency for Healthcare Research and Quality. 1998 Medical Expenditure Panel Survey–Table Compendium [web page]. Jun 14, 2002; available at: http://www.meps.ahrq.gov/compendiumtables/98ch1/compendiumtables.htm. Accessed Sep 7, 2002.

29. Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA. 1997;277:307–11. [PubMed]

30. Bates DW, Miller EB, Cullen DJ, et al. Patient risk factors for adverse drug events in hospitalized patients. ADE Prevention Study Group. Arch Intern Med. 1999;159:2553–60. [PubMed]

31. Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA. 1997;277:312–7. [PubMed]

32. Cabral JD. Poor physician penmanship. JAMA. 1997;278:1116–7. [PubMed]

33. Anonymous A study of physicians' handwriting as a time waster. JAMA. 1979;242:2429–30. [PubMed]

34. Winslow EH, Nestor VA, Davidoff SK, Thompson PG, Borum JC. Legibility and completeness of physicians' handwritten medication orders. Heart Lung. 1997;26:158–64. [PubMed]

35. Charatan F. Compensation awarded for death after illegible prescription. West J Med. 2000;172(2):80. [PMC free article] [PubMed]

36. Boring D, Di Domizio G, Cohen MR. The role of pharmaceutical trademarks in medication errors. In: Cohen MR, ed. Medication Errors: Causes, Prevention, and Risk Management. Sudbury, MA: Jones and Bartlett; 2000.

37. Sittig DF, Stead WW. Computer-based physician order entry: the state of the art. J Am Med Inform Assoc. 1994;1:108–23. [PMC free article] [PubMed]

38. McDonald CJ, Overhage JM, Tierney WM, et al. The Regenstrief Medical Record System: a quarter century experience. Int J Med Inf. 1999;54:225–53. [PubMed]

39. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA. 1993;269:379–83. [PubMed]

40. Gardner RM, Pryor TA, Warner HR. The HELP hospital information system: update 1998. Int J Med Inf. 1999;54:169–82. [PubMed]

41. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med. 1998;338:232–8. [PubMed]

42. Pestotnik SL, Classen DC, Evans RS, Burke JP. Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes. Ann Intern Med. 1996;124:884–90. [PubMed]

43. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–6. [PubMed]

44. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–21. [PMC free article] [PubMed]

45. Purves IN. PRODIGY: implementing clinical guidance using computers. Br J Gen Pract. 1998;48:1552–3. [PMC free article] [PubMed]

46. Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003;348:1556–64. [PubMed]

47. 2001 resource guide Healthcare IT products and services. Healthc Inform. 2000;17(12):17–172. [PubMed]

48. Reason J. Human Error. New York: Cambridge University Press, 1990.

49. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994;120:135–42. [PubMed]

50. Chute CG. Clinical classification and terminology: some history and current observations. J Am Med Inform Assoc. 2000;7:298–303. [PMC free article] [PubMed]

51. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized physician order entry on prescribing practices. Arch Intern Med. 2000;160:2741–7. [PubMed]

52. Lambert BL, Lin SJ, Chang KY, Gandhi SK. Similarity as a risk factor in drug-name confusion errors: the look-alike (orthographic) and sound-alike (phonetic) model. Med Care. 1999;37:1214–25. [PubMed]

53. Lambert BL, Chang KY, Lin SJ. Effect of orthographic and phonological similarity on false recognition of drug names. Soc Sci Med. 2001;52:1843–57. [PubMed]

54. Bates DW, Boyle DL, Teich JM. Impact of computerized physician order entry on physician time. Proc Annu Symp Comput Appl Med Care. 996;1994:. [PMC free article] [PubMed]

55. Tierney WM, Overhage JM, McDonald CJ, Wolinsky FD. Medical students' and housestaff's opinions of computerized order-writing. Acad Med. 1994;69:386–9. [PubMed]

56. Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled trial of direct physician order entry: effects on physicians' time utilization in ambulatory primary care internal medicine practices. J Am Med Inform Assoc. 2001;8:361–71. [PMC free article] [PubMed]

57. Poirier TI, Giudici R. Evaluation of drug interaction microcomputer software: an updated comparison. Hosp Pharm. 1995;30:888–90, 893–4. [PubMed]

58. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp. 2000:2–6. [PMC free article] [PubMed]

59. Lexchin J. Effects of restrictive formularies in the ambulatory care setting. Am J Manag Care. 2002;8(1):69–76. [PubMed]

60. White RH, Hong R, Venook AP, et al. Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing. J Gen Intern Med. 1987;2:141–8. [PubMed]

61. Evans RS, Pestotnik SL, Classen DC, Burke JP. Evaluation of a computer-assisted antibiotic-dose monitor. Ann Pharmacother. 1999;33:1026–31. [PubMed]

62. Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA. 1998;279:1024–9. [PubMed]

63. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289:1107–16. [PubMed]

64. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc. 1997;4:364–75. [PMC free article] [PubMed]

65. Miller LG, Liu H, Hays RD, et al. How well do clinicians estimate patients' adherence to combination antiretroviral therapy?. J Gen Intern Med. 2002;17:1–11. [PMC free article] [PubMed]

66. Anderson JG, Jay SJ, Anderson M, Hunt TJ. Evaluating the potential effectiveness of using computerized information systems to prevent adverse drug events. Proc AMIA Annu Fall Symp. 1997:228–32. [PMC free article] [PubMed]

67. Anderson JG, Jay SJ, Anderson M, Hunt TJ. Evaluating the capability of information technology to prevent adverse drug events: a computer simulation approach. J Am Med Inform Assoc. 2002;9:479–90. [PMC free article] [PubMed]

68. Brook RH, Chassin MR, Fink A, et al. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2:53–63. [PubMed]

69. Leveson NG, Turner CS. An investigation of the Therac-25 accidents. IEEE Computer. 1993;26(4):18–41.

70. McDaniel JG. Improving system quality through software evaluation. Comput Biol Med. 2002;32:127–40. [PubMed]

71. Reason J. Human error: models and management. BMJ. 2000;320:768–70. [PMC free article] [PubMed]

72. Zhang J, Patel VL, Johnson TR, Shortliffe EH. Toward a cognitive taxonomy of medical errors. Proc AMIA Symp. 2002:934–8. [PMC free article] [PubMed]

73. Nielsen J. Usability engineering. Boston: AP Professional, 1994.

74. Leape LL, Berwick DM, Bates DW. What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA. 2002;288:501–7. [PubMed]

75. Shojania KG, Duncan BW, McDonald KM, Wachter RM. Safe but sound: patient safety meets evidence-based medicine. JAMA. 2002;288:508–13. [PubMed]

76. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA. 2001;285:2114–20. [PubMed]

77. Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001;8:254–66. [PMC free article] [PubMed]

78. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA. 1995;274:35–43. [PubMed]

79. U.S. Department of Commerce National Institute of Standards and Technology. Integration Definition for Function Modeling (IDEF0). 1993. (Federal Information Processing Standards Publications (FIPS PUBS); vol 183).

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association