Clinical decision support has the potential to improve care, but a key limitation of the success of CDSS has been that they often have low user acceptance rates, which have been described for different settings and types of alerts. In this study, we quantified the impact of nine variables which appear to act as modulators of providers' alert acceptance.
Many previous studies have evaluated the impact of clinical decision support.15
Some of the factors affecting whether or not decision support is accepted appear to include workflow issues, the intrusiveness and importance of the alert, who it was displayed to, and factors about the system itself. However, many studies have focused more on whether decision support was expected, and have not explored in detail factors related to the reasons for this.
In this study, we attempted to quantify the impact of some of the major human factors issues which may affect alert acceptance by specification of three variables—the display of the alert, the textual information, and the prioritization of the alerts, all of which summarize specific covariates. According to the number of covariates integrated in the CDSS, the corresponding human factors variable may be classified as poor, moderate, or excellent. In our model, the alert display most strongly correlated with alert acceptance, reinforcing that the presentation of the alerts at the user interface is an important determinant of alert acceptance. Hence, to optimize current and future systems, knowledge of human factors and esthetics as well as the psychological aspects of human–computer interaction should be included in the development process. The textual information did not influence the frequency of alert acceptance; however, it did affect whether the provider would actually cancel an interacting drug or continue the prescription and modify it in order to account for the DDI. We were able to assess the influence of distinct covariates and found that providers were less likely to cancel a prescription but to keep and modify it if providers received detailed instructions on how to manage the interaction.
The factor with the largest impact on alert acceptance, which we therefore could not include in the logistic regression models, is whether an acknowledgment of the alert is mandatory or not, that is, whether the provider is forced to interact with the system or not. Of course, all the alerts in the category in which no acknowledgment was required had been put in that category because they were felt to be relatively less important than the remainder. Making interaction with the system mandatory for a specific alert will increase the likelihood that a specific alert will be accepted—but it also creates risks: (1) if the provider disagrees with the presented knowledge but is forced to accept warnings, overall acceptance and user satisfaction might decrease; (2) users might demand all alert functions be turned off; or (3) users may over-rely on the alert and the presented information.27
Indeed, in the current study, alert acceptance of inappropriate alerts was lower than those for appropriate alerts but still, one in 10 alerts was accepted (compared to three in 10, if the alert was appropriate). This underscores the importance of providing accurate and correct knowledge considering a maximum of clinical information of the individual patient context, in particular if alert acknowledgment is mandatory.
Previous studies have suggested that alerts are more frequently overridden if they are repeatedly presented. However, we found that alerts were more often accepted, especially by keeping the prescription but modifying it, if they were presented more frequently per user of the system. This may correspond to findings from psychology which indicate that we are better able to handle information we already know.
The current approach has several limitations. We assessed only one specific domain, DDIs, and other issues may be found for other domains. Moreover, the overall quality of the studied DDI knowledge base was high, and only a small number of DDI alerts were classified as inappropriate. Thus, the impact of quality of knowledge on alert acceptance might be stronger if knowledge bases with a larger proportion of poor alerts are assessed. Moreover, in such a case, a more detailed assessment of the clinical impact of the alerts might be required and patient individual clinical context should be considered in order to determine the appropriateness of the alert on a patient individual basis. Providers might also have altered their decisions later after the initial interaction based on consultation with another provider or another reference, but we could not evaluate this. We included only CDSS for adult patients and did not assess pediatric populations. We also performed studies within only one integrated delivery system, although we studied several different applications. This approach should be tested in other systems and with other vendors. Although we included a large amount of data from three different sites with different CDSS applications (even though an identical DDI knowledge base was used) and patient populations, the data were still not diverse enough to assess each variable or covariate. In particular, the variable prioritization was not considered in the logistic regression model because multicollinearity was found. On the other hand, because of the large sample size, some results are statistically significant when the differences observed are probably not clinically significant. Moreover, except for the covariates of textual information, there were not enough combinations of different specifications of the covariates in order to model the influence of each covariate. In order to conclusively quantify the impact of the covariates as well as the interplay of the display characteristics and the alert prioritization, further studies with different CDSS applications must be conducted. At that point the classification of distinct covariates may also be changed from binary (present/absence) to a gradual scale.
We conclude that specific modulators may affect the likelihood that decision support will be effective and hence can be useful for improving patient safety. If validated in other settings, the model we developed might help predict the acceptance of CDSS along with factors characterizing the setting, the system itself, and the presented knowledge. If healthcare is to be improved with CDSS, it will be important to have approaches for predicting whether or not specific alerts, warnings, and suggestions are likely to be successful.