The use of EHR-based CTAs has been demonstrated to increase participant recruitment rates to clinical trials, and is a promising approach for overcoming the major problem of inadequate and slow participant recruitment.7–9
Because such an approach will necessarily be employed in the context of complex and varied clinical care environments, information on the performance characteristics and response patterns among different groups of potential end-users is needed to inform its application and use.
These findings add to our understanding of how such alerts for clinical trials operate in real-world implementations by demonstrating empirically how and to what extent the rates of responses to such alerts decline across a variety of settings and end-users. Notably, they reveal, as hypothesized, that responses to point-of-care CTAs decline over time, although not as severely as anticipated, at least with regard to response rates.
Indeed, overall response rates to this series of alerts was initially high at 50% and remained reasonably high at 35% even after 36 months of exposure, compared to CDS alerts which tend to have 4%–51% response rates.12
While the fall in response rate suggests alert fatigue over time, the fact that a substantial proportion of the alerts were still being responded to at 36 weeks suggests that such a duration of use may still provide benefit. However, the finding that referral rates declined more quickly and more precipitously over time than response rates suggests there might be a point after which use of a CTA might not be worth even the minimal disruption they cause.10
In addition, the differences seen among community-based versus university-based physicians suggest that future CTA deployments should be tailored to a particular setting (ie, shorter in community-based settings and longer in university-based settings) in order to maximize benefit while avoiding excess fatigue. Additionally, as noted with some CDS alerts, tailoring of the alert's operating characteristics (eg, increasing specificity such that they trigger less often) might also affect response patterns and ultimately effectiveness, particularly in practice settings or specialties where response rates fall more rapidly.13
While the design of this study does not allow for definitive determination of the reasons for the declines noted, the difference between the response rate decline (2.7% per time period) and the referral rate decline (4.9% per time period) might reflect the fact that the act of CTA referral requires more effort than a simple response, and therefore causes more fatigability over time. However, this difference could also suggest the presence of other factors such as the possibility that declines in referrals reflect a drop in the available pool of eligible or interested candidates rather than alert fatigue. However, the population of potentially eligible participants (ie, patients with a recent stroke) remained relatively constant during the study, making this less likely. Nevertheless, it is probable that the reasons for the declines were multi-factorial, reflecting the combined influence of alert fatigue and other factors. Additional studies, including qualitative studies to assess physician-user perceptions, are ongoing and should help clarify other reasons for the declines noted.
Comparison of physician response patterns over time and apparent alert fatigue with those when similar CDS approaches are employed for clinical use would be useful. Unfortunately, data on such changes over time
in CDS response rates appear to be lacking in the published literature. As noted above, plentiful circumstantial evidence of this aspect of alert fatigue in many studies reveals less than ideal average
rates of response to CDS interventions,11
with some studies commenting on the common behavior of overriding alerts,20
and still others addressing changes that can increase average response rates by improving the usability or appropriateness of alerts.13
However, although this form of alert fatigue over time undoubtedly exists, there has been surprisingly little empirical evidence of it, or data to characterize the nature of the phenomenon. Our study appears to be among the first to empirically demonstrate this aspect of alert fatigue by tracking changes in clinician response to alerts over time. Therefore, we believe it has implications beyond recruitment using CTAs, and that such an approach to measuring responses over time can help advance understanding of alert fatigue in general. We also believe that the methodology employed here could be used to evaluate and refine the design and application of decision support alerts in the future.
Although the randomized study design and multi-user, multi-environment setting strengthen these findings and advance our understanding of CTA usage, this study has some limitations. These findings were derived from a single study of CTAs employed in a single trial of patients with recent stroke. Whether these findings would differ if the CTA were applied to another type of trial or in different settings remains to be determined. Also, while the CTA approach has been demonstrated to be effective using multiple EHR platforms,7–9
this study employed a single EHR and these findings might differ with the use of another EHR. Furthermore, this alert was employed in a setting where other alerts were rarely triggered. Another factor possibly impacting response rates over time is the threshold setting (ie, sensitivity vs specificity) for a given alert. Whether the findings of this study would differ if there were multiple or more frequent alerts is not known but is possible given that multiple simultaneous alerts are a commonly cited factor leading to alert fatigue as noted above, and should be studied.