Considerable attention has been drawn to electronic health record systems as a means of reducing medication errors. We implemented alerts for drug dosing adjustment into the daily routine use of a CPOE, through a design for intervention shown to be effective in previous studies. 7,18,19
However, the intervention was not associated with a significant decrease in the number of inappropriate prescriptions for inpatients with renal insufficiency. The design we chose was the best alternative to a randomized controlled trial due to the particular features of our study. Indeed, it was not possible to randomize physicians to intervention or control groups due to a contamination effect. Also, it was not appropriate to randomize units, because there were only two units with a different recruitment of patients, which would have led to a baseline imbalance between the two groups. This alternative design enabled us to avoid selection bias. Furthermore, we made substantial efforts to control for possible confounding factors, and our analyses accounted for the clustering of the data (multilevel models).
Moreover, we confirmed our results by various methods of data retrieval and evaluation. We were able to retrieve all of the relevant data from the hospital information system and conduct an electronic validation of the prescriptions. The overall results were similar for whichever method we used. Although concordance and discordance cases for various methods were not subtly evaluated, our results suggest that it is possible to obtain a long-term indicator of physician performance or that of the intervention effect without additional resources (i.e., log recording, manual validation). Consequently, despite the negative results of our study, it is conceivable that in the future the effect of these alerts may be evaluated in other departments of our hospital.
In our study, the rate of inappropriate prescriptions during control periods was 21.3%. This good performance may represent a ceiling effect, which may have limited the impact of our intervention. Some factors may contribute to the quality of prescriptions in our hospital: these include access, through the CPOE, to the Vidal electronic database and the role of the ward pharmacists. Moreover, the purpose of our alerts was to avoid excessive doses regardless of the renal function only, and the error rates could have been greater if we had taken into consideration the indication of each order (reducing consequently the recommended dosing range for specific indication).
We developed our alert to incorporate evidence-based features 20
associated with the success of decision support: we used computers to deliver support, offering specific recommendations and not just an assessment. Our application generated alerts automatically as part of the clinical workflow, at the time and the place that the decision was made. Indeed, if a system providing recommendations is dependent on the initiative of the physicians for use, they seldom make the required effort. 21
However, the manner chosen to provide overdosing alerts in our study may partly explain the negative results. Our alert was possibly not directive enough for the physician. Indeed, in the study by Chertow et al., 7
the investigators implemented and evaluated a system for adjusting drug doses for inpatients with renal insufficiency using a similar design to that reported here. During intervention periods, the adjusted dose list, default dose amount, and default interval were displayed to the prescribers. The intervention resulted in a significant decrease in inappropriate dosages, from 67% to 54%. Similarly, Oppenheim et al. 22
implemented an alert displaying the correct dosage of a drug only when a wrong dosage was selected by the physician. Half of the orders generating alerts were adjusted in response to alerts. In these two previous studies, the exact dosage was suggested to the prescriber for each prescription, whereas we provided an adjustment table as rationale for advice. Also, the patients included in our study were older than those in other studies that were investigated with similar interventions. 7,18,19
As in a previous study, 23
we observed a significant difference of the intervention effect between residents and senior physicians: residents tended to improve their performance on receipt of alerts, whereas senior physicians tended to make more errors during similar periods. Although the difference of inappropriate prescriptions was not significantly higher in intervention than in control periods among senior physicians, it is worth trying to explain this unanticipated effect. First, in French academic institutions, residents write most prescriptions for hospitalized patients. Senior physicians are likely to write prescriptions when doing the weekly or bi-weekly review of all patients in the unit; this review is carried out more frequently for more complicated patients, which residents are unable to cope with, especially young residents. Even if potentially there are situations in which residents and senior physicians wrote different prescriptions, and our multivariate analysis were unable to adjust for these situations, it is unlikely that they occurred differently in intervention and control periods. Second, senior physicians may have judged that the benefit of a chosen drug-dosing regimen outweighed the risk of an excessive dose, or they might have disagreed with the expert panel's recommendations. It is also likely that senior physicians preferred disregarding the alerts in favor of their own established practice, whereas younger physicians were keener to improve and were more receptive to new information. Although these factors may explain the absence of an effect of intervention among senior physicians, they fail to explain an increase in errors. Third, we cannot exclude that the senior physicians may have misinterpreted information relating to the dose, which may have been unclear. Indeed, unclear dose-related information within the context of alerts and an absence of time could lead to the misinterpretation of the alerts. 24
Finally, the intervention phases may have induced a false sense of security among senior physicians, because they knew these alerts were part of a research study.
Our study was subject to several limitations. First, the intervention was conducted in only one hospital with its own CPOE and health information system. Second, only two departments were included, and thus a small number of physicians. However, they were the most relevant departments in relation to this study, because they gathered the oldest patients, who consequently are the most likely to suffer from renal insufficiency. Third, our study may have lacked power for detecting a difference between the two groups. However, it is obvious that the alerts had no meaningful clinical impact. Fourth, eGFR was calculated thanks to the Modification of Diet in Renal Disease (MDRD) formula, which is imprecise for elderly patients, but is more accurate than the Cockroft-Gault equation. 15
There is also little evidence on drug dose recommendations in elderly patients, because the randomized controlled trials often exclude elderly patients. Finally, the MDRD formula does not accurately reflect renal function in unsteady-state conditions (such as acute increase or decrease of creatinine), so the alert may have delivered inappropriate recommendations in these cases. However, patients with these unstable conditions are more likely to be hospitalized in intensive care or in a renal unit in our institution. If this was not the case, the ward pharmacist could have alerted the clinician.
We assessed the effect of the intervention as a result of process measures rather than patient outcomes measures because process measures are the most suitable tool for judging quality of care. 25
Our results contrast with those of other recent reviews, 11,12
which conclude the effectiveness of CDSS for improving physician performance. However, these reviews can be biased by the fact that negative studies are less published than positive ones. 26
Moreover, Chaudhry et al. 12
suggested caution in interpreting the results of identified studies of CDSS because of flaws in their design and analysis. Furthermore, positive studies have been carried out for the most part by the developers of these systems. Finally, the results of these reviews contradict other studies, which identify high rates of alerts being overridden by physicians. 22,23
The implementation of effective CDSS remains a challenging task 27
and is not yet the miracle drug for improving the performance of physicians. It is still necessary to evaluate their impact if newly implemented in other settings and to monitor for unanticipated effects and their underlying causes.