Figure presents the barriers, facilitators, and suggestions for improvement that were most frequently raised by participants according to their groundedness; all themes presented were mentioned by multiple participants. The most commonly cited barriers overlapped considerably across sites and focus groups, despite differences in site characteristics and focus group composition. Furthermore, these themes were raised by multiple participants across the focus groups, suggesting they were not simply an artifact of a single, dominant participant.
Most commonly cited barriers, facilitators, and suggestions for improvement, by site.
Number of Alerts Received
The most frequently raised barrier was the number of alerts received by providers. In addition to test result alerts providers received many other types of notifications, which complicated the task of reviewing test results and providing timely follow-up care. Participants expressed concern about both the total number of alerts and the proportion of notifications perceived as unnecessary. Providers in all focus groups reported that their already heavy clinical workloads left very little time for the task of alert management:
"On an average it takes about two to three minutes per alert. And we get sixty to seventy alerts per day. So, there's no time allowed for alerts... I've just finished seeing patients. I have to go back and handle all the alerts. Some people actually come in on weekends. So, yeah, time is definitely a factor." -PCP, Site A
"One of the issues is just the sheer volume of alerts, and there are a number of alerts that in all honesty [you] really don't have any business seeing." -PCP, site B
"I counted 150 alerts one day just to see how many were coming in that normal day, and this is a fairly regular day, 150 alerts. That's a lot of time spent trying to go through that while you're seeing patients, while there's no in between time to get caught up." -PCP, Site B
Types of alerts perceived as unnecessary varied across the sites, but at both sites providers discussed situations when they were needlessly notified of events they deemed as strictly "for your information." These included, for example, overly detailed status updates of services performed outside primary care:
"The surgeon needs to take care of his own alerts. I don't need to be a backup for him. I mean, you know, he's licensed, right? He holds a license. He needs to worry about his license. He needs to take care of his stuff. And if every department did that, I mean, that would cut down our workload by fifty percent. That's where the problem is, everybody expects us to be the backup, and there's really no need." PCP, Site A
"You could have half a dozen notifications on a given consult which really are unimportant. The only thing I really need to know about is if it was actually scheduled and what the date was in case it's something I want to have done soon, and this is way too far away, or if it's canceled altogether. I really don't care about any of the other notifications, all these notes that they pass back and forth about whether or not they contacted the patient, whether or not he had transportation." PCP, Site B
To further explore the problem of large numbers of alerts, we conducted a co-occurrence analysis to examine common themes in participants' proposed solutions. We identified all passages in which suggestions for improvement co-occurred with any of the three quantity-related barriers most heavily discussed in the focus groups: too many alerts, unnecessary alerts, and an overly heavy patient-related workload created by the alerts. As seen in Figure , the three barriers co-occurred with a total of 17 suggestions for improvement. Fourteen were associated with overcoming the workload barrier; these suggestions involved both changes to CPRS (e.g., ability to categorize alerts, bundling alerts together) and changes to workflow (e.g., allocating protected time to manage alerts). Ten of seventeen suggestions applied to multiple barriers, suggesting that these barriers are interrelated. Interestingly, only three suggestions were uniquely associated with the two barriers about number of alerts. This analysis suggests that workload created by alerts is a complex barrier needing multidimensional solutions.
Co-occurrence analysis of participants' proposed solutions to volume-related barriers.
Tracking and Categorizing Relevant Clinical Information
Another salient theme was providers' desire for a mechanism within CPRS to organize, track, and retrieve alerts so that providers remember to follow up on needed care. As the CPRS View Alerts system was designed to alert providers so they could take action at the time of the alert, no functions for longitudinal tracking currently exist in CPRS. Therefore, at both sites, better EHR capabilities to help visualize, organize, and track alerts ranked among the most frequently cited suggestions.
I always wished to see a way to see or a way to know because then I can know how they're performing. There is no way you can tell me how many consults you've placed to GI in the month of November. There's no place I can click... . (PCP, Site A)
Provider Knowledge of EHR Features
Notably, providers often requested to add functionality to CPRS that already existed (e.g., sorting, ability to delete multiple alerts as a batch), suggesting they lacked knowledge about features within CPRS to manage the large number of incoming alerts:
"Oh, another thing, I learned yesterday that you can do it [sort] by patient also, so mine were all mixed up. So, I just learned that today before coming to this meeting that you can [sort], did you know that? ... That's something which I learned today after eight years of being at the VA."
-PCP, Site A
Along similar lines, participants strongly advocated improvements in CPRS training to highlight existing EHR features specific to alerts citing the existing training sessions they received as "pretty lackluster" (PCP, site B).
Improvements to Alert Content
Finally, providers at both sites also suggested improvements to alert content, such as minimizing inpatient alerts to receiving only the discharge summary, bundling related alerts to avoid overlap and redundancy):
I just want to be alerted, if there's any abnormal I don't care if it's CBC or Chem 7 just say "abnormal lab" and this is the name of the patient, you know, abnormal lab. IF the CBC came in later, click me again- abnormal lab. But if they come in all at the same time don't give me three alerts for the same patient! It clutters my view alert. It's so difficult for me to get into what is important and what is not. (PCP, Site B)