|Home | About | Journals | Submit | Contact Us | Français|
Medication errors constitute a major problem in all hospitals. Between 20% and 46% of prescriptions requiring dosage adjustments based on renal function are inappropriate. This study aimed to determine whether implementing alerts at the time of ordering medication integrated into the computerized physician order entry decreases the proportion of inappropriate prescriptions based on the renal function of inpatients.
Six alternating 2-month control and intervention periods were conducted between August 2006 and August 2007 in two medical departments of a teaching hospital in France. A total of 603 patients and 38 physicians were included. During the intervention periods, alerts were triggered if a patient with renal impairment was prescribed one of the 24 targeted drugs that required adjustment according to estimated glomerular filtration rate (eGFR).
The main outcome measure was the proportion of inappropriate first prescriptions, according to recommendation.
A total of 1,122 alerts were triggered. The rate of inappropriate first prescriptions did not differ significantly between intervention and control periods (19.9% vs. 21.3%; p = 0.63). The effect of intervention differed significantly between residents and senior physicians (p = 0.03). Residents tended to make fewer errors in intervention versus control periods (Odds ratio 0.69; 95% confidence interval 0.41 to 1.15), whereas senior physicians tended to make more inappropriate prescriptions in intervention periods (odds ratio 1.88; 95% confidence interval 0.91 to 3.89).
Alert activation was not followed by a significant decrease in inappropriate prescriptions in our study. Thus, it is still necessary to evaluate the impact of these systems if newly implemented in other settings thanks to studies also designed to watch for possible unanticipated effects of decision supports and their underlying causes.
Adverse events constitute a major problem in all hospitals, and the most common are attributed to drugs. In the Harvard Medical Study Practice II, 1 adverse drug events (ADEs) were estimated to account for 19% of iatrogenic injuries. Adverse drug events are associated with increases in the duration of hospital stay, additional costs, and mortality. 2 Medication errors, which account for a large proportion (28%) of ADEs, are inherently preventable. 3 More than half of these errors occur at the ordering stage. 3,4 More specifically, dosages are inappropriate for 20% to 46% of prescriptions requiring dosage adjustments based on renal function. 5–7
Computerized physician order entry (CPOE) 2,8 and the review of all prescriptions by pharmacists 9 have been put forward for reducing prescription errors. However, a recent study performed in our hospital showed that pharmacy validation produced only a moderate short-term impact on decreasing potential prescription errors. 10 Only 26% of alerts targeted to the prescribers resulted in a modification of the prescription. It was suggested that development of prescription aids for drug-dose adjustment might prevent some errors before the intervention of pharmacists, allowing them to concentrate on the most relevant interventions.
Indeed, computerized clinical decision support systems (CDSS) that automatically prompt users are often associated with increased practitioner performance. Garg et al. 11 showed that 19 of 29 drug-dosing or prescribing systems improved practitioner performance. However, in a recent systematic review, Chaudhry et al., 12 evaluating efficacy of health information technology in improving quality, pointed out that of the 257 studies analyzed, 25% were concentrated in four benchmark institutions. Thus, whether the results reported would be similar for all institutions is questionable and additional studies in other settings are needed.
We developed and implemented a system for drug dosage adjustment integrated to the CPOE system in the two general medical departments of our institution. The alert guided physicians to adjust drug doses for patients suffering from impaired renal function at the time of prescription. The main objective of this study was to evaluate the effect of this alert on the rate of inappropriate prescriptions. The second objective was methodological, i.e., to check whether an evaluation based on routine data automatically retrieved from the electronic health record (EHR) was feasible and reliable. We suggested that the complete automation and computerization of both the intervention and its evaluation would be a condition for the long-term success of such a system.
Our study was conducted in two medical departments of the Georges Pompidou European Hospital, an 827-acute-bed teaching hospital of the Assistance Publique-Hôpitaux de Paris (metropolitan area public hospital network in Paris). The hospital information system integrates an EHR (DxCare®, MEDASYS) with CPOE. The EHR currently collates orders and results of clinical tests and imaging procedures. The DxCare EHR is at the center of care delivery. It is integrated with other applications to allow the circulation of information among wards, laboratories, and the pharmacy. 13 Orders for laboratory tests are transmitted to the Laboratory Information Management System, which returns the results to the EHR. Drug prescriptions are transmitted to the Phedra® program, which is used by the pharmacy to manage prescriptions.
All prescriptions are reviewed for interactive validation by ward pharmacists. Four pharmacists (two senior pharmacists and two residents), assisted by two part-time pharmacy students, perform these validations. The results of the pharmacy validation can be accessed by the prescribers and/or the nurses (depending on the pharmacist's choice) via a symbol inserted in front of a given prescription order line: accepted (if the pharmacist agrees with the prescription), refused (if the pharmacist has identified a potentially severe prescribing error), or availability problem (if the drug should be changed due to a problem of availability). The physician may click on the symbol to view the pharmacist's comment, but is not obliged to take that comment into account. 10
This study was conducted between August 2006 and August 2007, and was based on an alternating time-series design, with three 2-month control periods and three 2-month intervention periods. All patients hospitalized in the Internal Medicine Department or in the Geriatric Department and prescribed one or more medications targeted by the alert system were included in this study; this also included all physicians working in these departments.
An expert panel, including nephrologists and pharmacists, determined which renally cleared and/or nephrotoxic medications were the most ordered in our institution. Using the Summary of Product Characteristics and expert opinion, the panel determined the dosage adjustments according to the estimated glomerular filtration rate (eGFR) for 24 drugs (). The eGFR was estimated using the revised-4 component of the Modification of Diet in Renal Disease (MDRD) study equation, 14 except that we did not take into account the race component because it is not validated in the European context. 15 Required dosage adjustments were based on level of kidney function impairment, which was divided into three categories: mild (eGFR 60 to 79 mL/min/1.73 m2), moderate (15 to 59 mL/min/1.73 m2) and advanced (<15 mL/min/1.73 m2). The expert panel determined the dose range and frequency for each of the medications in each of these categories. During intervention periods, the alert system was developed to compute these recommendations and to automatically display the corresponding alert if a targeted drug was selected for prescription. Throughout DxCare, the alert system collected the data transmitted from the Laboratory Information Management System and required for computer-assisted decision making: result of the last eGFR and its date. If the last eGFR was <60 mL/min/1.73 m2 (or <80 mL/min/1.73 m2 for the metformin) and was measured <5 days before the prescription was written, a reminder displayed the last result of eGFR and its date, the dosage adjustment table, and the reference on the prescription screen (dose adjust alert, ). The reminder appeared just after drug selection and before dosage selection. If no eGFR was stored or if it was performed 5 days or more before the prescription, the prescriber was alerted to measure the GFR before prescribing the drug (need for eGFR alert). The physician was not required to acknowledge or justify why he chose to ignore the electronic recommendation.
During control periods, no alert was displayed between drug selection and drug completion. In control periods, as well as in intervention periods, all prescriptions for drugs targeted in the study were validated by the ward pharmacist, with physicians being able to access his recommendations.
A log was kept each time a reminder was displayed (in intervention periods) or should have been displayed (in control periods) to support the adjudication of dose adjust alerts. The reviewer pharmacist (G.B.) judged each order independently of the ward pharmacist by examining the dose and frequency in combination with the recommendation that was displayed. There were three possible results: appropriate dose if the dose and interval did not exceed the recommendations displayed or if the drug was cancelled before completion after being recommended (metformin was indeed contraindicated when eGFR was <30 mL/min/m2), inappropriate dose if the dose or interval exceeded the recommendations, and not applicable if the patient's weight was needed to calculate the adequate dose (for some antibiotics) and was unavailable in the database. The reviewer pharmacist took no account of patient information, except weight if necessary. To ensure consistency of the reviewer pharmacist's adjudication, a subset of 204 logs chosen at random among the 700 first logs was blindly rated by a physician (E.S.). The physician was unaware of the pharmacist's adjudication and the study period. Cases of discordance were analyzed and consensus ratings were derived by the two raters. The inter-rater reliability was reported using the concordance rate and Kappa statistics.
To support the adjudication of the “need for eGFR” alerts, a log was kept for all instances in which a targeted drug was ordered without an eGFR measurement taken within the previous 5 days. In this case, a log was considered inappropriate if it was followed by the completion of the prescription and appropriate if the prescription was cancelled.
Our primary outcome assessed the proportion of inappropriate prescriptions among the first prescriptions that required a dosage adjustment. A prescription was considered a first prescription if it was the first order of a given medication during a patient's stay in the department, as opposed to renewals or updates for the same medication during his stay. The first prescription could not be influenced by the ward pharmacist's recommendations, in contrast to subsequent prescriptions. Thus, by including only the first prescription, we were able to estimate the effect of alerts prior to input by ward pharmacists.
We also investigated secondary outcomes. First, we assessed the proportion of inappropriate prescriptions among all prescriptions that required a dosage adjustment (excluding prescriptions adjusted on plasmatic concentration). In this way, we evaluated the effect of alerts in addition to recommendations of the ward pharmacist. We then evaluated the proportion of prescriptions for which no eGFR was available, which were followed by the cancellation of the order (rather than its completion).
In parallel, we also conducted a more automated process for data extraction and validation. First, we directly selected eligible prescriptions from the DxCare database, without using recorded logs. Second, we ignored all prescriptions that did not result in the drug being administered to the patient (this method allowed us to provide the proportion of drug dosage errors affecting patients). Third, we performed an electronic validation using the recommendations set forth by the expert panel. We finally assessed the proportion of inappropriate first prescriptions and the overall proportion of inappropriate prescriptions among drugs effectively administered to the patient.
Comparison of characteristics of patients and physicians during intervention and control periods were tested using the Chi square test and the Wilcoxon rank sum test when appropriate. The nominal significance level for the outcomes was 0.05 (two-sided formulation). We assessed the effectiveness of intervention on the rate of inappropriate prescriptions using a logistic regression model. We estimated the odds ratio (OR) for a prescription to be classified as inappropriate and its associated 95% confidence interval (CI) in intervention periods compared with control periods. Prescriptions were not assumed to be independent due to the hierarchical structure of data. Indeed, prescriptions (level 1) were nested within patient and/or within physician clusters (level 2). Patient and physician clusters were crossed at level 2: several physicians could have ordered a prescription for one patient and a physician could have ordered prescriptions for several patients. The departments (level 3) were accounted for by including them as a dummy variable in the model (i.e., fixed effects), because only two departments were involved. For each outcome, we determined the patient effect on how appropriate the prescriptions were by testing the significance of variance between patients. We repeated the procedure for the physician effect. The patient effect was sometimes significant; however, this was never the case for the physician effect. Thus, if appropriate, we used a mixed logistic model accounting for patients as a random effect. In this case, explanatory variables (drug category, patient gender, patient age, eGFR, prescriber gender, physician's level and department) were tested as fixed effects. Parameters of the model were estimated by a full maximum likelihood method with adaptative quadrature. 16,17 In other cases, if the patient effect was not significant, we used an ordinary multivariable logistic model.
We repeated analyses using the data retrieved from DxCare and the automatic validation to assess how robust the results were and to determine any meaningful effects of the selected outcome and validation procedure.
All analyses were performed using STATA V9.0 (StataCorp, College Station, Texas).
A total of 603 patients were included in the study, 321 in control periods and 282 in intervention periods. Among them, 294 (47.7 %) were ordered two or more targeted drugs. A total of 38 physicians were involved (23 residents and 15 senior physicians), of which 29 made prescriptions in both intervention and control periods, 3 only in intervention periods, and 6 only in control periods. Senior physicians were defined as all qualified physicians, including academic physicians (professors of internal medicine and assistant professors) and full-time non-academic physicians. No significant differences were observed for patients' and physicians' characteristics between the 2 groups ().
During the study period, 1,122 logs were analyzed (). A total of 955 logs corresponded to prescriptions for targeted drugs that were ordered for patients with renal insufficiency and that subsequently needed a dosage adjustment. Among them, 707 were considered as first prescriptions. The number and distribution by drug category are presented in . The remaining 167 logs corresponded to orders of targeted drugs without a recent eGFR measurement.
The concordance rate was estimated at 90% (95% CI 0.86 to 0.94) and kappa at 0.72 (95% CI 0.61 to 0.83). Thirteen instances of discordance were due to errors by the second rater (E.S.). Nine were induced by the revalidation process, which precludes the blinded physician from obtaining the relevant data from the EHR required for validation. Seven discordant validations were due to errors by the reviewer pharmacist. The three drugs for which the pharmacist made an error were systematically verified throughout the study and corrected if necessary.
Seven of the 707 first prescriptions evaluated by the reviewer pharmacist were rated as not applicable (0.99%) and were dropped for subsequent analyses. During intervention periods, 19.9% of orders (59 of 297) were inappropriate versus 21.3% (86 of 403) in control periods (unadjusted odds 0.91, 95% CI 0.63 to 1.32, p = 0.63). Results according to period are shown in .
The rate of inappropriate prescriptions did not differ significantly between residents (62 of 288, 21.5%) and senior physicians (24 of 115, 20.9%) in control periods (p = 0.88). However, senior physicians made more errors (24 of 82, 29.3%) than residents (35 of 215, 16.3%) in intervention periods (p = 0.01).
In the multivariable analysis (, Model 1), the interaction between intervention and prescriber level was significant, resulting in a different odds ratio for inappropriate prescriptions between intervention and control periods for residents (0.69, 95% CI 0.41 to 1.15) and for senior physicians (1.88, 95% CI 0.91 to 3.89). The odds ratio of inappropriate prescriptions for senior physicians compared with residents during intervention periods was 2.35 (95% CI 1.53 to 4.60, p = 0.03).
Drug category was strongly associated with whether a prescription was inappropriate or not. Angiotensin-converting enzyme inhibitors were less prone to errors (6.7%). Antibiotics accounted for most inappropriate prescriptions (54.5%), as they were ordered often (26% of first prescriptions) and they were classified as inappropriate in 43.4% of cases. B-adrenergic blocking agents, antigout drugs, and digoxin were inappropriate in 11.9%, 15.5%, and 17.3% of cases, respectively. Eleven of the 13 first prescriptions for metformin (84.6%) were inappropriate. Senior physicians and residents did not order different drug classes (p = 0.36). The effect of the intervention was not associated with the drug class (p = 0.44).
There were no differences in the intervention effect between the two departments and patient's gender. Finally, an increase in the age of the patient was associated with a decrease in inappropriate prescriptions.
Eight of the 955 prescriptions were rated as not applicable, and five others were adjusted on plasmatic concentration. These 13 prescriptions (1.4%) were subsequently excluded for analyses. During control periods, 20.4% of orders were inappropriate (106 of 520), whereas 18.5% (78 of 422) were inappropriate in intervention periods (unadjusted OR 0.81; 95% CI 0.51 to 1.28, p = 0.37). In multivariable analysis (, Model 2), the results were similar as previously outlined, except for the interaction between the physician's level and intervention, which was not significant.
Models 3 and 4 in present the results obtained from the analyses of data retrieved exclusively from the CPOE and whose validation was automated. Only prescriptions that were actually administered to patients were included. The results were similar to those obtained previously and, as for models with the validation carried out by the pharmacist (Models 1 and 2), the patient effect was significant only if all prescriptions, not only the first one, were included for analyses.
In control periods, 20 of 64 logs were followed by cancellation of the prescription (31.3%), in contrast to 36 of 103 logs (35.0%) in the intervention periods. The unadjusted odds of having an appropriate order was 1.18 (95% CI 0.61 to 2.30, p = 0.62) in intervention periods compared with control periods.
Considerable attention has been drawn to electronic health record systems as a means of reducing medication errors. We implemented alerts for drug dosing adjustment into the daily routine use of a CPOE, through a design for intervention shown to be effective in previous studies. 7,18,19 However, the intervention was not associated with a significant decrease in the number of inappropriate prescriptions for inpatients with renal insufficiency. The design we chose was the best alternative to a randomized controlled trial due to the particular features of our study. Indeed, it was not possible to randomize physicians to intervention or control groups due to a contamination effect. Also, it was not appropriate to randomize units, because there were only two units with a different recruitment of patients, which would have led to a baseline imbalance between the two groups. This alternative design enabled us to avoid selection bias. Furthermore, we made substantial efforts to control for possible confounding factors, and our analyses accounted for the clustering of the data (multilevel models).
Moreover, we confirmed our results by various methods of data retrieval and evaluation. We were able to retrieve all of the relevant data from the hospital information system and conduct an electronic validation of the prescriptions. The overall results were similar for whichever method we used. Although concordance and discordance cases for various methods were not subtly evaluated, our results suggest that it is possible to obtain a long-term indicator of physician performance or that of the intervention effect without additional resources (i.e., log recording, manual validation). Consequently, despite the negative results of our study, it is conceivable that in the future the effect of these alerts may be evaluated in other departments of our hospital.
In our study, the rate of inappropriate prescriptions during control periods was 21.3%. This good performance may represent a ceiling effect, which may have limited the impact of our intervention. Some factors may contribute to the quality of prescriptions in our hospital: these include access, through the CPOE, to the Vidal electronic database and the role of the ward pharmacists. Moreover, the purpose of our alerts was to avoid excessive doses regardless of the renal function only, and the error rates could have been greater if we had taken into consideration the indication of each order (reducing consequently the recommended dosing range for specific indication).
We developed our alert to incorporate evidence-based features 20 associated with the success of decision support: we used computers to deliver support, offering specific recommendations and not just an assessment. Our application generated alerts automatically as part of the clinical workflow, at the time and the place that the decision was made. Indeed, if a system providing recommendations is dependent on the initiative of the physicians for use, they seldom make the required effort. 21 However, the manner chosen to provide overdosing alerts in our study may partly explain the negative results. Our alert was possibly not directive enough for the physician. Indeed, in the study by Chertow et al., 7 the investigators implemented and evaluated a system for adjusting drug doses for inpatients with renal insufficiency using a similar design to that reported here. During intervention periods, the adjusted dose list, default dose amount, and default interval were displayed to the prescribers. The intervention resulted in a significant decrease in inappropriate dosages, from 67% to 54%. Similarly, Oppenheim et al. 22 implemented an alert displaying the correct dosage of a drug only when a wrong dosage was selected by the physician. Half of the orders generating alerts were adjusted in response to alerts. In these two previous studies, the exact dosage was suggested to the prescriber for each prescription, whereas we provided an adjustment table as rationale for advice. Also, the patients included in our study were older than those in other studies that were investigated with similar interventions. 7,18,19
As in a previous study, 23 we observed a significant difference of the intervention effect between residents and senior physicians: residents tended to improve their performance on receipt of alerts, whereas senior physicians tended to make more errors during similar periods. Although the difference of inappropriate prescriptions was not significantly higher in intervention than in control periods among senior physicians, it is worth trying to explain this unanticipated effect. First, in French academic institutions, residents write most prescriptions for hospitalized patients. Senior physicians are likely to write prescriptions when doing the weekly or bi-weekly review of all patients in the unit; this review is carried out more frequently for more complicated patients, which residents are unable to cope with, especially young residents. Even if potentially there are situations in which residents and senior physicians wrote different prescriptions, and our multivariate analysis were unable to adjust for these situations, it is unlikely that they occurred differently in intervention and control periods. Second, senior physicians may have judged that the benefit of a chosen drug-dosing regimen outweighed the risk of an excessive dose, or they might have disagreed with the expert panel's recommendations. It is also likely that senior physicians preferred disregarding the alerts in favor of their own established practice, whereas younger physicians were keener to improve and were more receptive to new information. Although these factors may explain the absence of an effect of intervention among senior physicians, they fail to explain an increase in errors. Third, we cannot exclude that the senior physicians may have misinterpreted information relating to the dose, which may have been unclear. Indeed, unclear dose-related information within the context of alerts and an absence of time could lead to the misinterpretation of the alerts. 24 Finally, the intervention phases may have induced a false sense of security among senior physicians, because they knew these alerts were part of a research study.
Our study was subject to several limitations. First, the intervention was conducted in only one hospital with its own CPOE and health information system. Second, only two departments were included, and thus a small number of physicians. However, they were the most relevant departments in relation to this study, because they gathered the oldest patients, who consequently are the most likely to suffer from renal insufficiency. Third, our study may have lacked power for detecting a difference between the two groups. However, it is obvious that the alerts had no meaningful clinical impact. Fourth, eGFR was calculated thanks to the Modification of Diet in Renal Disease (MDRD) formula, which is imprecise for elderly patients, but is more accurate than the Cockroft-Gault equation. 15 There is also little evidence on drug dose recommendations in elderly patients, because the randomized controlled trials often exclude elderly patients. Finally, the MDRD formula does not accurately reflect renal function in unsteady-state conditions (such as acute increase or decrease of creatinine), so the alert may have delivered inappropriate recommendations in these cases. However, patients with these unstable conditions are more likely to be hospitalized in intensive care or in a renal unit in our institution. If this was not the case, the ward pharmacist could have alerted the clinician.
We assessed the effect of the intervention as a result of process measures rather than patient outcomes measures because process measures are the most suitable tool for judging quality of care. 25
Our results contrast with those of other recent reviews, 11,12 which conclude the effectiveness of CDSS for improving physician performance. However, these reviews can be biased by the fact that negative studies are less published than positive ones. 26 Moreover, Chaudhry et al. 12 suggested caution in interpreting the results of identified studies of CDSS because of flaws in their design and analysis. Furthermore, positive studies have been carried out for the most part by the developers of these systems. Finally, the results of these reviews contradict other studies, which identify high rates of alerts being overridden by physicians. 22,23 The implementation of effective CDSS remains a challenging task 27 and is not yet the miracle drug for improving the performance of physicians. It is still necessary to evaluate their impact if newly implemented in other settings and to monitor for unanticipated effects and their underlying causes.
The authors thank Jérôme Rosser, MD (nephrologist), Bruno Cassard and Alexandre Stoehr (pharmacists) for developing dosage adjustment tables, and Florence Gillaizeau (statistician) for comments during statistical analyses.
Dr. Sellier was supported by a grant from the LEEM/Fédération Hospitalière de France. J. Nies was a PhD student funded by a grant from ANRT (CIFRE convention number: 498/2004).