The increasing availability of electronic health records (EHRs) presents an opportunity to address the issue of inadequate recruitment for clinical trials by leveraging the information and decision support resources often built into such systems. Indeed, recent studies of EHR-based, point-of-care, clinical trial alerts (CTA) have demonstrated they have the potential to improve recruitment rates when applied to clinical trials.^7–9 Despite their promise and the fact that they have been well tolerated in recent studies, CTAs like any point-of-care alert do have the potential for misuse, and further study is needed to better understand and inform their appropriate, widespread use.¹⁰ One important but poorly understood aspect relates to the performance characteristics of such alerts, particularly the issue of clinician responsiveness to alerts over time, and the implications of such phenomena on alert design and deployment decisions.

It is well recognized that when clinicians are exposed to too many clinical decision support (CDS) alerts they may eventually stop responding to them. This phenomenon is often called alert fatigue.^{11 12} While definitions vary and empirical evidence as to its cause is limited, alert fatigue is generally thought to result from one or more distinct but closely related factors. One such factor is declining clinician responsiveness to alerts as the number of simultaneous alerts increases. This is thought to be related to issues such as alert irrelevance and cognitive overload, and has also been referred to as ‘alert overload.’^{13 14} A second factor relates to declining clinician responsiveness to a particular type of alert as the clinician is repeatedly exposed to that alert over a period of time, gradually becoming ‘fatigued’ or desensitized to it.¹⁵ Few studies have explored this latter issue of ‘fatigue’ due to repeated exposure to alerts over time,¹⁶ and it is this latter aspect of alert fatigue that motivated and was the focus of the current study.

Subjects involved in this 36-week study of the CTA intervention in 2009 (the first, pre-cross-over phase of a larger study on the impact of this CTA intervention) included 178 physicians who were randomly divided into equal groups within their specialties (ie, neurologists, n=26; family medicine physicians, n=35; general internists, n=46; internal medicine-pediatrics specialists, n=8; and internal medicine house staff, n=63). All neurologists were university-based practitioners, while the other generalist physicians practiced either in university-based or community-based settings. Prior to CTA activation, all physicians were encouraged via traditional means (eg, discussion at staff meetings, email blasts, flyers) to recruit patients to the trial.

Upon CTA activation, intervention physicians seeing eligible patients were presented with on-screen CTAs that suggested they consider and discuss trial recruitment with the patient, and click an on-screen button to send a secure referral message to the trial coordinator if appropriate (figure 1). The physicians involved in this study did not receive any incentives for their participation in these recruitment efforts. It is worth noting that the results of the phase of the underlying CTA intervention study during which this analysis of responses took place revealed a significant 20-fold increase in referrals (p<0.0002) and a ninefold increase in enrollments (p<0.006).¹⁷ The design of the CTA intervention itself has also previously been reported and involved a minimal amount of novel programming as the built-in tools and resources of the particular EHR system were used.¹⁸ This was a similar approach to that employed with another popular EHR system upon which we have previously reported.¹⁹

Additional data were also collected during the same period via direct query of the EHR-fed enterprise data warehouse for the two types of events in which we were interested: (1) responses that indicated interaction with the CTA (ie, taking action by responding to at least one of the questions posed in the alert); and (2) flags (referrals) sent to the study coordinator via positive responses to both questions and then processing of the CTA. In addition, data on the total number of alerts triggered were extracted as the denominator value for determination of a ratio of (1) response rate and (2) referral rate per physician, per time unit.

During the initial time period, the response rate to CTAs was about 50% among all users but dropped significantly over time by 2.7% for each advancing 2-week period, and that trend was significantly different from zero (flat) (p<0.0001; figure 2). Notably, there was still a 35% response rate at the 36th week of exposure.

Subgroup analyses of response rate changes over time reveal that there are no significant differences between subspecialists (neurologists in this case) and generalists, with both groups trending downward to a similar extent, whether or not community generalists are included in the analyses. However, there is a greater response rate drop-off among all community-based providers compared with all university-based physicians (p=0.0489). Further, that drop-off is stronger when university-based subspecialists (neurologists) are removed and the analyses are restricted to generalists in both types of settings (p=0.0146).

Referrals rates started at about 33% and, although they fluctuated, declined to about 9% by the end of the study period. Of note, generalists made fewer referrals than subspecialists (p<0.0001), and community-based physicians made fewer referrals than university-based physicians (p=0.006). The decline in referrals rates over time was more pronounced than the decline in response rates noted above. Specifically, there was a significant 4.9% decrease in referral rates per time period (p=0.0294) (figure 3).

These findings add to our understanding of how such alerts for clinical trials operate in real-world implementations by demonstrating empirically how and to what extent the rates of responses to such alerts decline across a variety of settings and end-users. Notably, they reveal, as hypothesized, that responses to point-of-care CTAs decline over time, although not as severely as anticipated, at least with regard to response rates.

Indeed, overall response rates to this series of alerts was initially high at 50% and remained reasonably high at 35% even after 36 months of exposure, compared to CDS alerts which tend to have 4%–51% response rates.¹² While the fall in response rate suggests alert fatigue over time, the fact that a substantial proportion of the alerts were still being responded to at 36 weeks suggests that such a duration of use may still provide benefit. However, the finding that referral rates declined more quickly and more precipitously over time than response rates suggests there might be a point after which use of a CTA might not be worth even the minimal disruption they cause.¹⁰ In addition, the differences seen among community-based versus university-based physicians suggest that future CTA deployments should be tailored to a particular setting (ie, shorter in community-based settings and longer in university-based settings) in order to maximize benefit while avoiding excess fatigue. Additionally, as noted with some CDS alerts, tailoring of the alert's operating characteristics (eg, increasing specificity such that they trigger less often) might also affect response patterns and ultimately effectiveness, particularly in practice settings or specialties where response rates fall more rapidly.^{13 20}

While the design of this study does not allow for definitive determination of the reasons for the declines noted, the difference between the response rate decline (2.7% per time period) and the referral rate decline (4.9% per time period) might reflect the fact that the act of CTA referral requires more effort than a simple response, and therefore causes more fatigability over time. However, this difference could also suggest the presence of other factors such as the possibility that declines in referrals reflect a drop in the available pool of eligible or interested candidates rather than alert fatigue. However, the population of potentially eligible participants (ie, patients with a recent stroke) remained relatively constant during the study, making this less likely. Nevertheless, it is probable that the reasons for the declines were multi-factorial, reflecting the combined influence of alert fatigue and other factors. Additional studies, including qualitative studies to assess physician-user perceptions, are ongoing and should help clarify other reasons for the declines noted.

Comparison of physician response patterns over time and apparent alert fatigue with those when similar CDS approaches are employed for clinical use would be useful. Unfortunately, data on such changes over time in CDS response rates appear to be lacking in the published literature. As noted above, plentiful circumstantial evidence of this aspect of alert fatigue in many studies reveals less than ideal average rates of response to CDS interventions,^{11 12} with some studies commenting on the common behavior of overriding alerts,²⁰ and still others addressing changes that can increase average response rates by improving the usability or appropriateness of alerts.¹³ However, although this form of alert fatigue over time undoubtedly exists, there has been surprisingly little empirical evidence of it, or data to characterize the nature of the phenomenon. Our study appears to be among the first to empirically demonstrate this aspect of alert fatigue by tracking changes in clinician response to alerts over time. Therefore, we believe it has implications beyond recruitment using CTAs, and that such an approach to measuring responses over time can help advance understanding of alert fatigue in general. We also believe that the methodology employed here could be used to evaluate and refine the design and application of decision support alerts in the future.