Objective: To review methodologies of detecting adverse events using information technology, reports of studies that used these techniques to detect adverse events, and study results for specific types of adverse events.

Design: Structured review.

Methodology: English-language studies that reported using information technology to detect adverse events were identified using standard techniques. Only studies that contained original data were included.

Main Outcome Measures: Adverse events, with specific focus on nosocomial infections, adverse drug events, and injurious falls.

Results: Tools such as event monitoring and natural language processing can inexpensively detect certain types of adverse events in clinical databases. These approaches already work well for some types of adverse events, including adverse drug events and nosocomial infections, and are in routine use in a few hospitals. In addition, it appears likely that these techniques will be adaptable in ways that allow detection of a broad array of adverse events, especially as more medical information becomes computerized.

Although the “To Err is Human” report brought patient safety into the public eye, the principal research demonstrating this major problem was reported years ago, with much of the data coming from the 1991 Harvard Medical Practice Study.^3,4 The most frequent types of adverse events affecting hospitalized patients were adverse drug events, nosocomial infections, and surgical complications.⁴ Earlier studies identified similar issues,^5,6 although their methodology was less rigorous.

Hospitals routinely underreport the number of events with potential or actual adverse impact on patient safety. The main reason is that hospitals historically have relied on spontaneous reporting to detect adverse events. This approach systematically underestimates the frequency of adverse events, typically by a factor of about 20.^7–9 Although manual chart review is effective in identifying adverse events in the research setting,¹⁰ it is too costly for routine use.

Another approach to finding events in general and adverse events in particular is computerized detection. This method generally uses computerized data to identify a signal that suggests the possible presence of an adverse event, which can then be investigated by human intervention. Although this approach still typically involves going to the chart to verify the event, it is much less costly than review of unscreened charts,¹¹ because only a small proportion of charts need to be reviewed and the review can be highly focused.

This paper reviews the evidence regarding the use of electronic tools to detect adverse events, first based on the type of data, including ICD-9 codes, drug and laboratory data, and free text, and then on the type of tool, including keyword and term searches and natural language processing. We then discuss the evidence regarding the use of these tools to identify nosocomial infections, adverse drug events in both the inpatient and outpatient setting, falls, and other types of adverse events. The focus of this discussion is to detect the events after they occur, although such tools can also be used to prevent or ameliorate many events.

The data source most often applied to patient safety work is the administrative coding of diagnoses and procedures, usually in the form of ICD-9-CM and CPT codes. This coding represents one of the few ubiquitous sources of clinically relevant data. The usefulness of this coding—if it is accurate and timely—is clear. The codes provide direct and indirect evidence of the clinical state of the patient, comorbid conditions, and the progress of the patient during the hospitalization or visit. For example, administrative data have been used to screen for complications that occur during the course of hospitalization.^12,13

However, because administrative coding is generated for reimbursement and legal documentation rather than for clinical care, its accuracy and appropriateness for clinical studies are variable at best. The coding suffers from errors, lack of temporal information, lack of clinical content,¹⁵ and “code creep”—a bias toward higher-paying diagnosis-related groups (DRGs).¹⁶ Coding is usually done after discharge or completion of the visit; thus its use in real-time intervention is limited. Adverse events are poorly represented in the ICD-9-CM coding scheme, although some events are present (for example, 39.41 “control of hemorrhage following vascular surgery”). Unfortunately, the adverse event codes are rarely used in practice.¹⁷

Despite these limitations, administrative data are useful in detecting adverse events. Such events may often be inferred from conflicts in the record. For example, a patient whose primary discharge diagnosis is myocardial infarction but whose admission diagnosis is not related to cardiac disease (e.g., urinary tract infection) may have suffered an adverse event.

Pharmacy data and clinical laboratory data represent two other common sources of coded data. These sources supply direct evidence for medication and laboratory adverse events (e.g., dosing errors, clinical values out of range). For example, applications have screened for adverse drug reactions by finding all of the orders for medications that are used to rescue or treat adverse drug reactions—such as epinephrine, steroids, and antihistamines.^18–20 Anticoagulation studies can utilize activated partial thromboplastin times, a laboratory test reflecting adequacy of anticoagulation. In addition, these sources supply information about the patient’s clinical state (a medication or laboratory value may imply a particular disease), corroborating or even superseding the administrative coding. Unlike administrative coding, pharmacy and laboratory data are available in real time, making it possible to intervene in the care of the patient.

With increasing frequency, hospitals and practices are installing workflow-based systems such as inpatient order entry systems and ambulatory care systems. These systems supply clinically rich data, often in coded form, which can support sophisticated detection of adverse events. If providers use the systems in real time, it becomes possible to intervene and prevent or ameliorate patient harm.

The detailed clinical history, the evolution of the clinical plan, and the rationale for the diagnosis are critical to identifying adverse events and to sorting out their causes. Yet this information is rarely available in coded form, even with the growing popularity of workflow-based systems. Visit notes, admission notes, progress notes, consultation notes, and nursing notes contain important information and are increasingly available in electronic form. However, they are usually available in uncontrolled, free-text narratives. Furthermore, reports from ancillary departments such as radiology and pathology are commonly available in electronic narrative form. If the clinical information contained in these narrative documents can be turned into a standardized format, then automated systems will have a much greater chance of identifying adverse events and even classifying them by cause.

A study by Kossovsky et al.²² found that distinguishing planned from unplanned readmissions required narrative data from discharge summaries and concluded that natural language processing would be necessary to separate such cases automatically. Roos et al.²³ used claims data from Manitoba to identify complications leading to readmission and found reasonable predictive value, but similar attempts to identify whether or not a diagnosis represented an in-hospital complication of care based on claims data met with difficulties resolved only through narrative data (discharge abstracts).

A range of approaches is available to unlock coded clinical information from narrative reports. The simplest is to use lexical techniques to match queries to words or phrases in the document. A simple keyword search, similar to what is available on Web search engines and MEDLINE, can be used to find relevant documents.^12,25–27 This approach works especially well when the concepts in question are rare and unlikely to be mentioned unless they are present.²⁶ A range of improvements can be made, including stemming prefixes and suffixes to improve the lexical match, mapping to a thesaurus such as the Unified Medical Language System (UMLS) Metathesaurus to associate synonyms and concepts, and simple syntactic approaches to handle negation. A simple key-word search was fruitful in one study of adverse drug events based on text from outpatient encounters.¹⁷ The technique uncovered a large number of adverse drug events, but its positive predictive value was low (0.072). Negative and ambiguous terms had the most detrimental effect on performance, even after the authors employed simple techniques to avoid the problem (for example, avoid sentences with any mention of negation).

Natural language processing^28.29 promises improved performance by better characterizing the information in clinical reports. Two independent groups have demonstrated that natural language processing can be as accurate as expert human coders for coding radiographic reports as well as more accurate than simple keyword methods.^30–32 A number of natural language processing systems are based on symbolic methods such as pattern matching or rule-based techniques and have been applied to health care.^30–45 These systems have varied in approach: pure pattern matching, syntactic grammar, semantic grammar, or probabilistic methods, with different tradeoffs in accuracy, robustness, scalability, and maintainability. These systems have done well in domains, such as radiology, in which the narrative text is focused, and the results for more complex narrative such as discharge summaries are promising.^{36,41,46–50}

With the availability of narrative reports in real time, automated systems can intervene in the care of the patient in complex ways. In one study, a natural language processor was used to detect patients at high risk for active tuberculosis infection based on chest radiographic reports.⁴⁵ If such patients were in shared rooms, respiratory isolation was recommended. This system cut the missed respiratory isolation rate approximately in half.

Given clinical data sources, which may include medication, laboratory, and microbiology information as well as narrative data, the computer must be programmed to select cases in which an adverse event may have occurred. In most patient safety studies, someone with knowledge of patient safety and database structure writes queries or rules to address a particular clinical area. For example, a series of rules to address adverse drug events can be written.¹⁷ One can broaden the approach by searching for general terms relevant to patient safety or look for an explicit mention of an adverse drug event or reaction in the record. Automated methods to produce algorithms may also be possible. For example, one can create a training set of cases in which some proportion is known to have suffered an adverse event. A machine learning algorithm, such as a decision tree generator, a neural network, or a nearest neighbor algorithm, can be used to categorize new cases based on what is learned from the training set.

Studies were included in the review if they contained original data about computerized methods to detect nosocomial infections, adverse drug events, adverse drug reactions, adverse events, or falls. We excluded studies that focused on adverse event prevention strategies, such as physician order entry or clinical decision support systems, and did not include detailed information regarding methods for adverse event detection. We also excluded studies of computer programs designed to detect drug-drug interactions.

Included studies evaluated the performance of a diagnostic test (an adverse event monitor). The methodologic quality of each study was determined using previously described criteria for assessing diagnostic tests.⁵² Studies were evaluated for the inclusion of a “gold standard.” For the purpose of this review the gold standard was manual chart review, with the ultimate judgment of an adverse event performed by a clinician trained in adverse event evaluation. Furthermore, the gold standard had to be a blinded comparison applied to charts independently of the application of the study tool. Only studies that evaluated their screening tool against a manual chart review of records without alerts were considered to have properly utilized the gold standard.

Reviewers abstracted information concerning the patients included, the type of event monitor implemented, the outcome assessed, the signals used for detection, the performance of the monitor, and any barriers to implementation described by the authors. The degree of manual review necessary to perform the initial screening for an adverse event was assessed to determine the level of automation associated with each monitor. An event monitor using signals from multiple data sources that generated an alert that was then directly reviewed by the clinician making the final adverse event judgements was considered “high-end” automation. An event monitor that relied on manual entry of specific information into the monitor for an alert to be generated was considered “low-end.” All disagreements were settled by consensus of the two reviewers.