In this study, the highest numbers of ADEs were found through electronic text searching and computer-generated signals with provider reports, hospital discharges, and emergency department notes identifying much lower numbers. A low percentage of ADEs were reported by providers; only 11% of all ADEs and 6% of the preventable ADEs found during the year were identified through provider reports. This finding parallels previous studies of ADEs among inpatients for whom the proportion of ADEs reported by providers has averaged approximately 5%.5
During the study period, repeated attempts were made to enhance provider reports through in-service training sessions and the development of multiple reporting options including a Web-based mechanism based on the medical group's Intranet. Previous research has demonstrated a correlation between the intensity of prompting and the rate of provider reports of adverse events,21
suggesting that these activities may have been responsible for the slightly higher rate of ADEs reported by providers in this study. Provider reports did have the highest PPV for preventable ADEs compared with other methodologies. However, overall, these data suggest that electronic strategies identify many more ADEs than other sources.
Comparing the productivity of signals to that of undirected chart review in an ambulatory population is complex. The one year of follow-up in this study included 334,045 person-months for Medicare + Choice enrollees and 47,039 outpatient visits by older adults who were not enrolled in that program. Undirected chart reviews would have required assessment of all the visit notes, laboratory test results, medication prescriptions, and reports from hospitalizations and emergency department visits for approximately 381,084 person-months. Based on this estimate and the 1,523 total ADEs identified in this study, the PPV for a person-month of chart review would be 0.4%.
One previous study of ADEs in the ambulatory setting employed incident detection methods similar to the computer-based approaches in our study.8
Although the earlier study included the full age-range of patients seen by a group practice and the current study was limited to elders, there are a number of similarities in the findings. Notably, the rate of ADEs detected was similar (55 per 1,000 patients with visits during one year in the earlier study vs. 48 per 1,000 patient-years in the current study) as was the overall PPV across the computer-based signal sources (7.5% in the earlier study vs. 8.8% in the current study). The earlier study generated many fewer computer signals from laboratory test results and drug prescribing, probably due to the use of higher thresholds and requirements for combinations of information as well as a difference in the age of the patient population; the earlier study found a lower PPV for these signals (3% in the earlier study vs. 6.8% in the current study). In contrast, free-text reviews of electronic notes in the earlier study were much more expansive, identifying 22,792 incidents for review compared with only 5,048 in the current study. Here again, the earlier study found a lower PPV (7.5% in the earlier study vs. 12% in the current study). Although the current study expanded the sources of signals to include manual reviews of hospital discharge summaries and emergency department reports as well as reports from providers, the majority of ADEs were identified through computer-based sources.
We also found very little overlap among the ADEs identified in the various sources that we employed; only 5% of ADEs were found in more than one source. Previous studies have also found low rates of overlap,1,3,22
although not as low as in this setting. This low overlap suggests that every one of the sources that we investigated has low sensitivity in the ambulatory setting. In the current study, there were notable differences in the types of events captured by different sources. Not surprisingly, ADEs for which the major effects are symptomatic were found most often in the electronic notes whereas those that result in out-of-range laboratory tests appeared in the computer-generated signals. Only one type of event, ADEs that consisted of drug-related falls, were found almost exclusively through review of hospitalizations and emergency department visits. Aside from these exceptions, event types were identified across all sources despite the lack of overlap.
In summary, our results suggest that all these sources contribute important, independent information about the occurrence of ADEs in a population treated in the ambulatory setting. Investigators designing studies in similar settings and those attempting quality improvement projects aimed at reducing ADEs should include multiple sources if the aim is to approximate the true underlying rate. Recent work from the Institute for Healthcare Improvement based in hospital settings has suggested that health care systems without easy access to computerized information may be able to replicate some of these signals through the use of “triggers” in paper-based medical records,23
and this may be an option in ambulatory settings as well. Where specific sources are omitted, the details presented here will assist developers in understanding the events that they are likely to miss. Overall, the low PPVs suggest that extensive investigator time will be required to use any of the sources investigated.
Our findings suggest several ways in which the search for ADEs could be made more efficient and less labor-intensive, primarily through enhancements in the use of automated clinical data. In particular, the systems accessed in this study could not easily track changes in laboratory values or drug dispensing over time and did not include entries of new allergies. Our use of electronically recorded clinical notes was based on simple searches for specific keywords and phrases among patients using various drug classes. Natural language processing could improve the PPVs of these indicators through pattern matching and rule-based techniques,24,25,26
and algorithms for detection of ADEs could be developed through machine learning. The initial review of hospital discharge summaries and emergency department visits was a major component of pharmacist investigators' time in our study. For settings in which these reports are captured in electronic form, a similar process could be used to automatically search for indications of drug-related incidents.27
As electronic medical record systems are increasingly used in ambulatory care settings, more sophisticated approaches may be possible. For example, output from computer-generated signals could include clinical notes from the relevant time period, eliminating the need to obtain and search paper medical records.
Our study has several limitations. ADEs were identified only through the methods described so that negative predictive values could not be measured. We did not perform time-motion studies of the pharmacist investigators so that the relative efficiency of the various strategies employed could not be assessed. The study was conducted in the context of a single multispecialty group practice providing care to elderly persons residing in a single geographic area, and the vast proportion of the study population was composed of Medicare + Choice enrollees. This particular setting is ideal for such research because automated data on medications, laboratory results, and electronic clinic notes are readily available. However, the patterns of ADEs and their identification in various sources are likely to differ at other sites. The extent to which possible drug-related incidents could be determined during the medical record reviews is dependent on the quality and extent of record keeping. This medical group used integrated medical records with extensive documentation in a well-organized record. This clarity may not be available at other ambulatory sites, which would reduce the ability to identify ADEs.