We found that our non-interruptive medication laboratory alerts did not impact provider ordering of baseline labs for new prescriptions, even when analyzed by medication class or laboratory test type. We felt that our warnings merited action from providers in most instances. The attempt, however, to decrease alert fatigue through non-interruptive alerts did not translate into changes in clinician behavior for this set of alerts. Unlike the Shah et al. study which demonstrated that alert compliance increased with a limited set of interruptive, severity-tiered alerts, failure to demand action from the user may cause alerts to be ineffective.
Many hypotheses exist for the end-result. Assuming that providers have every intention of delivering an accepted standard of care, lack of alert compliance implies a deliberate choice to disagree with the recommendations or a latency that includes unawareness, information deficiencies, and human error. If the decision to ignore the alerts is active, then users are reading the warnings. They may, however, have felt that our knowledge base for the non-interruptive alerts had too low a threshold for firing medication laboratory alerts and/or may not have agreed with the recommendations. We did, however, expect a high clinical acceptance from our providers, especially since the knowledge base used in this study was reviewed by a team within our institution.
Latency, or the tendency to let the status quo “slide,” however, is more concerning. The goal of a decision support system is to adjust for human knowledge gaps and the nature to err, especially when overwhelmed with the tremendous load of information in the delivery of health care. Keeping in mind Bates et al.'s suggestion that “physicians will strongly resist stopping” we had designed our alerts to be minimally intrusive, occupying a restricted box within the medication order entry screen. We wanted neither to resort to cluttering pop-ups nor to sacrifice on-screen real estate belonging to clinical order entry, yet it is possible the warnings were simply too easy to ignore. Despite being displayed in the color red, users may require additional stimuli such as alternate color schemes, animations, or screen arrangements to capture their attention. 23
Alternately, providers could possess an inherent bias against non-interruptive recommendations, mentally linking suggestions that are neither life-threatening nor action-dependent to the assumption that the information is not very important. If the total number of alerts crossed a clinician's subjective threshold of “too much” or “too distracting,” the user could have adopted the “lesser evil” of accepting the higher-tiered, interruptive warnings and bypassing the less critical, non-interruptive warnings. One critical systems aspect is that we did not have an electronic link between computerized physician medication order entry, our automated alerts, and laboratory requisitions at the time of the study. Although the alert may have fired when a request for medications was initiated, providers could submit the recommended blood tests only by hand. The involvement of two mediums, computer and paper, may simply have been too inconvenient and disruptive to permit a reasonable level of alert compliance. Ultimately, the outcome is likely not dominated by one reason but rather an amalgamation of these many factors.
Our findings do not mitigate the overall utility of tiered alerts. As noted by Shah et al. in a study which utilized the identical Partners knowledge base of medication warnings, high acceptance of interruptive, moderate-high acuity alerts depended on a tiered alert system to limit alert burden. 20
By maintaining less critical recommendations in a non-interruptive state, users were less prone to workflow disruption and possibly more amenable to heeding an alert that warranted interruptive decision-making. 23,24
These findings reinforce that the significant value of a non-interruptive alert lies in its minimally intrusive nature. Promoting non-interruptive alerts of low importance to the next tier of interruptive alerts would diminish the overall acceptance rates for interruptive alerts and make it more likely that more important alerts would be overridden.
Encouraging provider awareness toward or compliance with embedded electronic alerts remains a formidable task. The range of responses to alerts has been varied, and while studies have shown some compliance with recommendations, the vast majorities have demonstrated minimal to modest changes in behavior. 25–28
One example would be the Weingart et al. study which found that 91% of drug-allergy and 89% of high-severity drug-drug interaction automated alerts were overridden by physicians in ambulatory care. 29
In some cases, the reason was justified—when the benefits of the drug outweighed the disadvantages or the possible adverse events, the drug selection was limited, or existing patient information was incorrect. However, ignoring or misinterpreting multiple or all alerts as “unjustified overrides” poses a patient safety issue. 18
In fact, addressing the need to intercept enough potential adverse drug events while avoiding fatigue caused by clinically insignificant alerts, technological problems, unnecessary workflow interruptions, and lack of time that contribute to alert overrides identifies the critical balance between the sensitivity and specificity of drug alerts. 30
The study has a number of limitations. As we mentioned earlier, it is the unjustified overriding of alerts that poses a notable patient safety concern, but without reasons justifying why providers elected not to order the recommended baseline laboratory tests, we cannot determine the proportion of alerts that were inappropriately disregarded. In addition, we recognize that our results are influenced by the on-screen interface specific to our system, but without further feedback from users we are unable to identify any contribution of system design to human behavior that led to the final outcome. Over-testing may also have gone undetected; it was outside the scope of this study but has been examined in our other work. 31,32
Finally, these practices were all affiliated with large academic medical centers, and while they represented a diverse group, the results may not be generalizable to small, independent practices.
Future studies should include an assessment of interruptive alerts with linked lab order entry to facilitate provider ordering of the recommended baseline labs. Another study could track the eye movements of providers using a tool to register what on-screen elements are effective in capturing the user's initial attention and to what temporal extent users are paying heed to the non-interruptive alerts. Finally, we suggest that the urgency of the interruptive alerts may, in part, be defined by the existence of a lower limit provided by the less critical set of recommendations in non-interruptive alerts; however, we have yet to determine the extent non-interruptive alerts impact behavior toward the two more critical alert levels. A future study would examine the provider medication and lab ordering behavior of critical interruptive alerts with and without the presence of less urgent, non-interruptive alerts.
In conclusion, the delicate art of implementing clinically useful alerts in ambulatory care requires the consideration of many factors including user demands, knowledge base content, and systems design. The clinical importance of non-interruptive medication laboratory rules alone are an insufficient drive toward full compliance with alerts, yet safe and high quality patient care is contingent upon baseline laboratory values that should be ordered in conjunction with new prescriptions. As prescribers often have limited time and a strong tendency to maintain an existing workflow, efforts should still be directed at maximizing the utility of non-interruptive alerting. Future studies should examine if linked order entry or changes in screen design might significantly influence these results.