|Home | About | Journals | Submit | Contact Us | Français|
Prescribing alerts generated by computerized drug decision support (CDDS) may prevent drug-related morbidity. However, the vast majority of alerts are ignored because of clinical irrelevance. The ability to customize commercial alert systems should improve physician acceptance because the physician can select the circumstances and types of drug alerts that are viewed. We tested the effectiveness of two approaches to medication alert customization to reduce prevalence of prescribing problems: on-physician-demand versus computer-triggered decision support. Physicians in each study condition were able to preset levels that triggered alerts.
This was a cluster trial with 28 primary care physicians randomized to either automated or on-demand CDDS in the MOXXI drug management system for 3,449 of their patients seen over the next 6 months.
The CDDS generated alerts for prescribing problems that could be customized by severity level. Prescribing problems included dosing errors, drug–drug, age, allergy, and disease interactions. Physicians randomized to on-demand activated the drug review when they considered it clinically relevant, whereas physicians randomized to computer-triggered decision support viewed all alerts for electronic prescriptions in accordance with the severity level they selected for both prevalent and incident problems. Data from administrative claims and MOXXI were used to measure the difference in the prevalence of prescribing problems at the end of follow-up.
During follow-up, 50% of the physicians receiving computer-triggered alerts modified the alert threshold (n = 7), and 21% of the physicians in the alert-on-demand group modified the alert level (n = 3). In the on-demand group 4,445 prescribing problems were identified, 41 (0.9%) were seen by requested drug review, and in 31 problems (75.6%) the prescription was revised. In comparison, 668 (10.3%) of the 6,505 prescribing problems in the computer-triggered group were seen, and 81 (12.1%) were revised. The majority of alerts were ignored because the benefit was judged greater than the risk, the interaction was known, or the interaction was considered clinically not important (computer-triggered: 75.8% of 585 ignored alerts; on-demand: 90% of 10 ignored alerts). At the end of follow-up, there was a significant reduction in therapeutic duplication problems in the computer-triggered group (odds ratio 0.55; p = 0.02) but no difference in the overall prevalence of prescribing problems.
Customization of computer-triggered alert systems is more useful in detecting and resolving prescribing problems than on-demand review, but neither approach was effective in reducing prescribing problems. New strategies are needed to maximize the use of drug decision support systems to reduce drug-related morbidity.
At least 2% to 3% of ambulatory patients are treated each year for preventable adverse drug events, 58% of which are related to prescribing errors. 1–4 Drug-related illness accounts for 5% to 23% of hospital admissions, 5–8 and is now claimed to be the sixth leading cause of mortality. 9 Computerized decision support (CDS) is considered to be a critical safety feature needed to reduce the risk of preventable adverse drug-related events. 10,11 This is because dosing errors as well as drug–allergy, drug–drug, and drug–disease interactions are responsible for an important share of preventable adverse events, 12 and integrated CDS can be designed to alert physicians at the point of prescribing about potential problems before a prescription is generated. 2 However, experience has shown that physicians override 49% to 96% of alerts for drug, allergy, and disease contraindications, 13–20 substantially reducing any potential value that CDS may have in preventing prescribing errors and adverse events.
The factors that lead physicians to override drug alerts are complex. Commercial vendors aim for comprehensiveness and leave the judgment of relevance to individual clinicians. As a result, physicians receive many alerts; a substantial proportion of which are considered clinically irrelevant. 18,19,21,22 Physicians report that the sheer volume of alerts interferes with workflow, increases the likelihood that they will fail to respond to critical prescribing problems, and creates substantial barriers to the use of electronic prescribing systems altogether. 15,19 Systems developed in-house that reduce the volume of alerts by targeting a limited set of drug problems have achieved the best success in altering prescribing practices. 17,23,24 This suggests that the capacity to customize commercial drug alert systems to the local context may improve physician acceptance of electronic prescribing and drug management systems. 11
Two approaches that can be used to customize drug decision support and alerts within integrated systems are to: (1) provide decision support only when a physician considers it relevant to request this information (on-demand decision support) or (2) provide automated computer-triggered decision support that can be modified by the physician to exclude alerts of a severity level that are considered not relevant (customizable computer-triggered decision support). The on-demand approach places a greater emphasis on workflow congruence by closely matching the way in which physicians use resources such as specialist consultants and drug interaction databases in usual practice. Physicians seek advice when they believe it is needed, and do so when it fits within their usual workflow. By optimizing workflow congruence, there should be substantially greater acceptance by physicians and a higher rate of response to alerts for prescribing problems. Yet there are important limitations of this approach. Even physicians who are confident in their ability to identify clinically relevant prescribing problems can only identify 51% of relevant drug problems correctly. 21 Further, on-demand systems for expert assistance provide no safety net for forgetting, inadvertent oversights or data entry errors—problems that occur frequently in complex health care systems. 25 The customizable computer-triggered decision support places a greater emphasis on patient safety, even at the risk of substantial workflow disruption. The customizable approach takes advantage of severity ratings for drug and disease interactions that are included in many commercial knowledge bases to allow individual physicians to filter the alerts they view. Within this context, physicians can customize an automated surveillance system by selecting severity ratings for the level of alerts to be viewed as well as actively filtering out clinically irrelevant alerts at the alert or patient level. In theory, the capacity to customize automated computer-triggered alerts systems should provide the optimal approach to patient safety by providing physicians with the tools to create clinically relevant alerts, combined with the safety net to identify clinically relevant prescribing problems.
We undertook this study to determine whether there would be a greater reduction in potential prescribing problems and a lower rate of alert overrides with a customizable computer-triggered drug decision support system compared to a physician on-demand decision support system. We tested this hypothesis in a cluster-randomized controlled trial of primary care physicians and their patients.
A single-blind, cluster randomized controlled trial was conducted to assess the benefits of customizable computer-triggered versus on-demand drug decision support in reducing the prevalence of prescribing problems. The study was conducted in a fixed cohort of primary care physicians and an open cohort of patients seen by study physicians in the 6-month follow-up period after randomization (February 1, 2004, to September 30, 2004). Physicians were blinded to the study outcome but not their intervention assignment. Patients, clustered within physicians, were the unit of analysis. The benefit of the intervention was assessed at the end of follow-up by comparing the prevalence of prescribing problems among patients whose physicians received decision support by automated surveillance with patients of physicians who received on-demand support.
Physicians were eligible for inclusion if they were general practitioners or family physicians in full-time (≥4 days/wk), fee-for-service practice in Montreal. Only fee-for-service physicians were included (representing 86% of all Quebec physicians 26 ) because demographic data from the province-wide health insurance agency (RAMQ) were needed to identify all potentially eligible patients and to produce accurate denominator counts of all patients who visited the physician during the baseline and follow-up periods. There was no information available on practice size, visit rates, or patient characteristics for salaried physicians, and for this reason, they were excluded.
All patients in the practice who consented to participate, had at least one prescription written by the study physician, and visited the study physician during the follow-up period were included in the study. The RAMQ medical services claims database was used to provide the list of all patients seen in the last year by each study physician (first name, last name, health insurance number) to prepopulate the patient list on each physician's electronic health record. Written patient consents to participate in the research program and permission to access all medication data from the RAMQ were obtained by the physician or by office staff. Ethics approval was granted by the McGill Faculty of Medicine Institutional Review Board.
Physicians were stratified by clinic, and an equivalent number of physicians within each stratum were randomly allocated to the computer-triggered versus on-demand drug decision support management system. Randomization was carried out on January 29, 2004, after physicians were actively using the electronic drug management system for 12 months, and 14,130 patients had been recruited to participate.
The MOXXI electronic prescribing and integrated drug management system, described in detail elsewhere, 27 was used to test the two approaches of providing drug decision support. Physicians wrote prescriptions electronically using a personal digital assistant that was connected by wireless networks to a central server. Based on medical services claims files, the drug management module provided a drug profile that listed all currently active medications, color-coded by prescribing physician, as well as drug costs and dates of emergency room and inpatient hospital visits. Patient information was retrieved by real-time integration with the beneficiary, prescription, and medical services claims files of the RAMQ, and with private pharmacy computer networks. A one-step re-prescribing feature from the list of current medications reduced the time for multiple repeat prescriptions. A health problem list was generated from recorded treatment indications for drugs prescribed, diagnostic codes (ICD9) from all medical services claims, single-indication drugs, and manual entry by physicians. Physicians were provided with wireless connections to laser printers so prescriptions could be printed for the patient and the medical chart.
The drug knowledge database (MentoR) (MentoR, Vigilance Santé, Montreal, Quebec) provided customizable levels of alerts for all major types of prescribing problems: excess dose, drug–allergy, drug–drug, drug–disease, drug–age contraindications, and therapeutic duplication (www.vigilance.ca). Information from the patient's current medication list and new written prescriptions was used to screen for both prevalent and incident prescribing problems. For drug–disease contraindications, disease information was acquired from mandatory documentation of treatment indications for each prescription and from the health problem list. For drug–age contraindications, the patient's age was retrieved from the RAMQ beneficiary database at the time of enrollment, and these data were combined with the list of current and newly prescribed medications for identification of potential problems. Allergy information was entered by the physician and linked to a particular drug. All drugs with the same chemical ingredients were identified, and documented allergies and current and new prescriptions were used to identify potential drug–allergy prescribing problems.
The sensitivity of alerts in the drug knowledge base is classified by level of clinical importance: level 1: definite and serious adverse effects: should be avoided in all patients; level 2: likely adverse effects: should be avoided in most patients; level 3: possible adverse effects: monitor or avoid if possible. The system was initially set to monitor level 1 and level 2 problems. Physicians could change the default setting to show all alerts (levels 1, 2, and 3), or only level 1 alerts. Customization of the alert level was specified in physician preference options. In addition, each physician had the option of suppressing individual drug alerts that he or she disagreed with at the time the alert was generated. When a physician decided to override an alert, he or she was required to indicate the main reason for doing so from a menu of the following choices: benefit greater than risk, interaction already known, drug/disease information incorrect, need to consult with prescribing physician, no time at this visit, not clinically important, patient resistant to change.
The computer-triggered decision support program functioned in the background of the prescription drug management system. It would assess prescribing problems and generate automated alerts, in accordance with the severity level selected by the physician, at two points in the drug management process. First, when a chart was opened a review of all drugs, disease, and allergies was conducted to assess prevalent problems for a given patient. Prescribing problems were identified by an exclamation mark (!) beside the respective drug in the drug profile. Drugs involved in the alert were highlighted by color code, red reflecting the most severe prescribing problems/alerts (level 1), with orange and green being used for levels 2 and 3, respectively. If the physician clicked on the drug, the details of the alert were shown. Second, prescribing problems were assessed at the time new prescriptions and refill prescriptions for existing drugs were sent electronically or printed, and alerts for problems with both refilled and new prescriptions were shown. The physician could respond to an alert by using the stop and change order function to modify the medication or dose, and to send a message to the pharmacist to stop all future refills of the medication. For each alert that was ignored, the physician was required to document a reason.
The on-demand decision support system could be accessed by a physician, for a given patient, at any time during the prescribing process. To do so, the physician clicked on drug review in the MOXXI system menu. The system reviewed all current medication, allergies, and health problems, as well as any new medication prescribed, and alerted the physician if any problem was detected. The customization options, alert screens, and requirement for documenting reasons for overridden alerts were identical to those that would be produced in computer-triggered decision support, the only difference being that the physician was required to request decision support to initiate the process.
The RAMQ administrative and MOXXI databases were used to assess service use and patient characteristics in the 12 months preceding the start of the intervention as well as in the follow-up period. The RAMQ beneficiary demographic database provided data on individual age, gender, and mortality, and census data on income and education. 28 The prescription claims database and retail pharmacy data provided information on each drug dispensed including the drug name, quantity, date, and duration for each prescription; the prescribing physician; and the dispensing pharmacy. The medical services claims database provided information on the beneficiary, date, type, provider, and location of service delivery (e.g., inpatient, emergency, clinic) for all medical services remunerated on a fee-for-service basis. The MOXXI database provided data on prescriptions written, treatment indications linked to each prescription, potential and verified health problems, and audit trails reflecting use of the system and response to drug alerts. Data were linked by RAMQ number, a unique identifier for each Quebec resident.
The primary outcome was the prevalence of prescribing problems at the end of the follow-up period. Prescribing problems included drug–disease, drug–drug, drug–allergy, drug–age contraindication, excess dose, or therapeutic duplication alerts identified by the drug knowledge database decision support system based on records of prescribed and dispensed medication for each patient from the RAMQ, retail pharmacies, and MOXXI databases. Only prescribing problems that were attributable to drugs prescribed by the study physician alone or in conjunction with other physicians were included in the assessment. Prevalence was assessed by determining the number of patients with one or more prescribing problem in the last month of follow-up divided by the number of patients who made a visit to the study physician, and were prescribed and dispensed at least one prescription medication in the follow-up period. Secondary outcomes and descriptive information that were also measured included the proportion of alerts that were viewed and overridden, the reasons for overriding alerts, the extent to which physicians used the customization features of the application to filter alerts, and the effect of the intervention on rates of prescribing problems by level of severity and type of prescribing problem.
Baseline assessment of the mean monthly number and type prescribing problems in the 12 months prior to randomization was performed for each patient. Age, gender, income, and education 28 were measured using data from the RAMQ beneficiary file. Comorbidity was assessed using the Charlson comorbidity index. 29,30 The Charlson comorbidity index was measured using diagnostic codes recorded in medical service claims in the 12 months prior to the start of the intervention for each patient, information that has been validated by the investigators with Quebec data. 31 Health care service use included assessment of the number of emergency room visits and number of hospitalizations for all causes in the 12 months prior to the start of the intervention for each patient, as well as the continuity of care, reflecting the proportion of visits to the study physician relative to all other physicians. 32 Physician age, gender, intention to use MOXXI, and speed in writing electronic prescriptions was measured using standardized questionnaires and skill tests. 27 Practice size, volume, and location(s) were measured using medical claims data for each physician.
Descriptive statistics were used to evaluate differences in the baseline characteristics of participating physicians and patients in the two arms of the trial. Study hypotheses were tested among: (1) all patients who made a visit, and (2) the subset of patients for whom the physician had used the electronic drug management system at least once during the follow-up period using multivariate logistic regression within a generalized estimating equation framework, a nested modeling approach. Patient was the unit of analysis, and physician was the clustering factor. An exchangeable correlation structure was used to account for dependence between observations. All multivariate models were adjusted for baseline differences in the number of prescribing problems, as well potential differences in other relevant patient characteristics (age, gender, income and education, comorbidity, health care use, and continuity of care) between intervention groups. The estimated sample size of 2,359 patients for this study was designed to have 80% power to detect an absolute change in the prevalence of prescribing problems of 10%, based on a baseline prevalence of 30%, a type 1 error of 5%, and a within-physician intracluster correlation of 0.02.
Physicians randomized to on-demand drug decision support had lower intentions to use the drug management system, were slower in writing electronic prescriptions, and used the MOXXI system less frequently in the baseline year. Physicians in both on-demand and computer-triggered groups saw, on average, 30 patients per day and worked in 1.5 to 1.8 different settings. The average age of patients in the practice population was 67.3 in the on-demand group and 66.9 in the computer-triggered group, and 61% were female. In both groups, over 50% of visits were to the study physicians, and 74% to 78.9% of prescriptions were written by them ().
In the 6 months prior to randomization, the mean proportion of patients in each month who had at least one prescribing problem was 29.3% in the on-demand group and 34.3% in the computer-triggered group ().
On average, patients had 3.1 prescribing problems in the on-demand group and 3.5 in the computer-triggered group, of which 71.3% to 75.2% respectively were attributable to the study physicians' prescriptions alone. Among the 15,669 prescribing problems identified, the majority were for drug–disease contraindications (on-demand: 36.4%; computer-triggered: 35.6%), drug interactions (on-demand: 23.3%; computer-triggered: 24.5%), and therapeutic duplications (on-demand: 16.1%; computer-triggered: 14.4%) (). Over half of all prescribing problems identified were in the lowest severity category—use with caution, whereas 9.6% (on-demand) to 10.1% (computer-triggered) were considered absolutely contraindicated.
During the follow-up period, 50% of physicians in the computer-triggered group modified the level of alerts, in comparison to 21% of on-demand physicians; 35.7% of computer-triggered physicians modified the default setting of level 1 and 2 alerts to seeing only the most serious level 1 alerts, in comparison to 14.3% of on-demand physicians ().
Overall 4,445 problems were identified in patients seen by on-demand physicians, 2,524 (56.5%) were not seen because of alert setting, 1,320 (29.6%) were not seen because the physician did not use the MOXXI system, and 41 (0.9%) were seen by study physicians by requesting a patient drug review. Of the 41 problems seen by on-demand physicians, 31 (75.6%) were eliminated by changing the medication or revising the drug dose. In contrast, among the 6,505 prescribing problems identified for patients seen by computer-triggered physicians, 4,402 (67.7%) were not seen because of alert setting, and 1,435 (22.1%) were not seen because the physician did not use the MOXXI system. Of the 668 (10.3%) alerts that were seen by computer-triggered physicians, only 81 (12.1%) were addressed by changing medication(s) or modifying the dose. Among the 585 prescribing problems that were ignored by physicians in the computer-triggered group, the most frequent reason for deciding not to revise a patient's drug profile was that the benefit was considered to be greater than the risk, the interaction was already known, or it was not clinically important (75.8% of reasons for ignoring alerts). These were also the only reasons for choosing to ignore alerts by on-demand physicians.
Among all alerts that were seen by study physicians, modifications in treatment were more likely to be made for alerts related to dosing errors, drug interactions, drug–disease contraindications, and therapy duplications for both computer-triggered and on-demand physicians ().
The drug–disease contraindication alert that was most frequently revised was cardiovascular disease and fast-acting beta-agonist, and for therapy duplications, it was the prescription of multiple beta-blockers. The alerts that were most frequently ignored in these categories were hypothyroidism and oral sulfonylurea, and multiple antidepressants. Surprisingly, prescribing problems that were considered to be absolutely contraindicated were not more likely to be revised than prescribing problems that should be avoided (level 2) or used with caution (level 3) by either computer-triggered or on-demand physicians.
After adjustment for patient characteristics and baseline differences in the prescribing problems, there was no significant difference in the prevalence of prescribing problems at the end of the follow-up period ().
These findings prevailed for the 1,485 patients for whom physicians used the MOXXI system as well as for all 3,422 patients who made a visit during the follow-up period. The only exception was for therapeutic duplications (more than one drug from the same therapy class), for which there was a significant 57% (odds ratio: 1.43; p < 0.0001) reduction in prevalence in the computer-triggered alert group, a finding that remains significant even after applying a Bonferroni 33 adjustment for multiple comparisons.
We found that providing physicians with control over drug alerts had a significant impact on the alerts they viewed and the proportion of alerts they ignored. Physicians in the computer-triggered group saw more alerts than the on-demand group, made more changes to the level of alerts they would see, but ignored 87.8% of the problems identified by the computer system. In contrast, on-demand physicians rarely requested drug review and thus saw <1% of the prescribing problems identified by the drug knowledge system, but ignored only 24.4% of problems identified through their requests for advice. Although a greater absolute number of alerts were seen and revised by physicians in the computer-triggered alert group, both groups underused the drug decision support system. As a result, there was no significant reduction in the overall prevalence of prescribing problems by the end of the follow-up period.
We required physicians to document the reason for ignoring an alert, and in so doing we gained insight into potential avenues to improve vendor-based decision support systems. We found that the most common reasons for ignoring an alert was that the problem was already known, the benefit was judged to be greater than the risk, or that it was not clinically important—reasons that are similar to those reported by physicians in prior research. 13–17,19,21,22 Indeed, in a recent survey of physicians' responses to alerts for drug interactions, therapy duplications and allergy checks in a computerized order-entry system, only 11% of alerts were considered clinically relevant. 19 Most commercial vendors use a common set of standard references in the development of their knowledge bases. 34–37 These references may not provide sufficient empirical information to accurately establish the clinical relevance and severity of problems identified.
There are several approaches that could improve the clinical relevance of alerts. First, current drug alert systems are not designed to take the patient's historical medication profile into account—a drug that is used for 20 years is treated in the same manner as a drug that is a new prescription. Yet the greatest risk of adverse drug effects is in starting or stopping medication. 38 Long-term users have already demonstrated their capacity to tolerate the medication. This may be why physicians are more likely to respond to an alert when it is a new drug for a patient than a refill of existing therapy, 14,16 and why physicians responded to a small proportion of alerts in our study, as the majority of alerts would have been for prevalent problems related to refills of medication for chronic disease management. To address this problem, drug alert systems could be designed to provide a two-step review process. Automated surveillance and computer-triggered alerts could be restricted to new drugs, allergies, or diseases. A complete review of all existing drugs, allergies, and diseases could be provided as an optional or periodic review requirement for all patients. As chronic therapy accounts for the majority of prescriptions, 39 this approach could dramatically reduce clinically irrelevant alerts, yet provide physicians with the option of reviewing drug-related problems when relevant.
Second, drug review systems could produce more clinically relevant alerts if patient characteristics that influence the absolute risk of an adverse drug event such as age, number of medications, renal function, and comorbidity could be taken into account. 38,40 Although empirical estimates of the actual risk of an adverse drug-related event are generally not available for the majority of drug alerts, risks have been estimated in relationship to other patient characteristics, 38,40 and could be used to target patients at greater absolute risk of adverse events, for whom modification of treatment may have the greatest potential to reduce preventable adverse events. Redesign of drug alert systems to consider clinical characteristics will also be needed in future to incorporate patient-specific pharmacogenomic risk profiles. If combined, strategies to restrict computer-triggered alerts to new prescriptions and patients with high-risk profiles could produce a greater yield on clinically relevant alerts.
Our study has a number of limitations that need to be considered in interpreting the results, namely a lack of robust evidence to support the clinical relevance of drug alerts and study outcomes, a limited number of physicians, and the assessment of only two approaches to customization. Although the knowledge base we selected had many of the features of advanced commercial systems, including severity classification, and the capacity to suppress alerts at the individual and group level, 41 many of the computer-triggered alerts were overridden because of clinical irrelevance, a problem that has been noted in other studies. 15 It is possible that there may be greater utilization of computer-triggered decision support if a more clinically relevant subset of problems that pose a greater threat to patient safety were assessed. The challenges of developing homegrown drug databases or customizing commercial knowledge systems to include only clinically relevant alerts appear to be considerable. 41 Even when an alert severity classification is available, such as in this study, it did not produce the expected result of having fewer alert overrides for the most serious, absolutely contraindicated problems. There seem to be two problems in the implementation of filters for clinical relevance. First, there is limited empirical evidence to judge the risk relative to benefit for the vast majority of drug alerts. Until such evidence is more broadly available, clinically relevant alerts will be selected on the basis of expert opinion, a lower level of evidence that may not yield predictable benefits for patient safety. Second, similar to the results reported by Shah, 17 we found that data errors were responsible for many false-positive alerts. For example, excess dose errors for alendronate was the most common absolutely contraindicated false-positive drug alert in our study, a problem attributable to systematic errors in recording treatment duration by the dispensing pharmacist.
As a small number of physicians were included in this study, we had insufficient power to use multilevel modeling approaches that would allow us to investigate physician characteristics that may modify response to drug alerts. This is an important area for future research because response to alerts may vary by learning style, and alert systems could be customized to provide different approaches depending on physician preference. We only assessed two approaches for customizing alerts. Other approaches, such as using physician feedback to modify the central knowledge system, or tailored feedback reports of prescribing alerts, may be effective and should be assessed in future research.
In summary, we found that on-demand and computer-triggered drug review systems had a substantial effect on the drug alerts that were viewed by physicians as well as their response to drug alerts, but not on the overall prevalence of prescribing problems except for therapeutic duplication errors. Both groups underused drug decision support, a problem that likely exists for both community and hospital-based physicians. New approaches to produce more clinically relevant drug alerts in commercial systems are needed. Focusing drug decision support on new prescriptions and high-risk patients may be more effective in reducing drug-related morbidity in the short term.
The authors thank Mr. Christian Savard and Ms. France Bourque of the Régie de l'assurance maladie du Québec, and Mr. Jimmy Fragos, Mr. Alphonse Estruch, Mr. Yannick Lazzari, Mr. David Marques, Mr. Jonathan Richard, and Ms. Irena Sesartic of the MOXXI Development Team, for their assistance.
Supported by Health Canada, Canadian Institutes of Health Research.