Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 2009 Sep-Oct; 16(5): 607–612.
PMCID: PMC2744710

Computerized Surveillance for Adverse Drug Events in a Pediatric Hospital


There are limited data on adverse drug event rates in pediatrics. The authors describe the implementation and evaluation of an automated surveillance system modified to detect adverse drug events (ADEs) in pediatric patients. The authors constructed an automated surveillance system to screen admissions to a large pediatric hospital. Potential ADEs identified by the system were reviewed by medication safety pharmacists and a physician and scored for causality and severity. Over the 6 month study period, 6,889 study children were admitted to the hospital for a total of 40,250 patient-days. The ADE surveillance system generated 1226 alerts, which yielded 160 true ADEs. This represents a rate of 2.3 ADEs per 100 admissions or 4 per 1,000 patient-days. Medications most frequently implicated were diuretics, antibiotics, immunosuppressants, narcotics, and anticonvulsants. The composite positive predictive value of the ADE surveillance system was 13%. Automated surveillance can be an effective method for detecting ADEs in hospitalized children.

Introduction and Background

Adverse drug events (ADEs) comprise the largest single category of adverse events in large populations studies of patient safety. 1,2 Estimates of the incidence of ADEs vary widely, 3–6 depending upon definitions, measurement methodologies, and populations studied. Methods used for ADE detection have included implicit chart review, 1,7,8 explicit chart review, 9,10 and computerized signal detection with manual validation of alerts. 3,4,11 While chart review has traditionally been considered a gold standard, there is evidence that computerized surveillance detects many events that are not easily detected during chart review. 4 Automated surveillance has advantages over chart review, including the ability to survey a patient population (e.g., all inpatients) comprehensively and continuously, and a significantly lower resource requirement than chart review. 4 It is generally agreed that voluntary reporting of ADEs is low-yield and anecdotal in nature, and not valuable for ADE quantification. 3,4,12,13

Despite the extensive literature on ADEs in adult populations, relatively little is known about the frequency and nature of these events in children. In a study based on concurrent order and chart review and error reporting by providers, Kaushal et al. examined ADEs in pediatric inpatients at two institutions and described an ADE rate of 2.3 per 100 admissions, or 6.6 per 1,000 patient-days. 14 Using a system that the current author (PMK) implemented at another institution, 11 another group has examined ADEs on the pediatric inpatient service of a general hospital; using triggers designed primarily for ADE detection in adults, they described an ADE rate of 1.6 per 100 admissions or 1.8 per 1,000 patient-days. 15

Given the significant differences in the types and frequencies with which various categories of medication are used in pediatrics, we hypothesized that automated ADE detection for pediatrics might be enhanced by consideration of a wider range of rules than those that have been previously employed for computerized surveillance in adults. Our hypothesis was reinforced by preliminary data suggesting greater importance of medications that affect electrolyte balance in pediatric compared with adult populations. 16 This paper describes our implementation of an automated ADE detection system at a large pediatric hospital, and data from the operation of the system over a six month period. We describe our findings regarding the utility of different rule categories for event detection, and our overall findings of ADE rates in our pediatric population.


St Louis Children's Hospital (SLCH) is a 250-bed hospital specializing in the care of acutely ill pediatric patients. The SLCH is a member of BJC HealthCare, a 13 hospital integrated delivery system headquartered in St Louis. The hospital has approximately 14,500 admissions annually, with an average length of stay of 3.4 days. The SLCH is the principal pediatric teaching hospital for the Washington University School of Medicine (WUSM), and is located with Barnes-Jewish Hospital on the WUSM-BJC academic medical center campus in St Louis. Our study population included all patients admitted between Feb 1 and Jul 31, 2008, with the exception of oncology patients, for reasons described below. The study was approved by Washington University School of Medicine's Human Research Protection Office.

Building upon our previous work with expert systems, 17–19 we modified a rules-based computer program to perform real-time surveillance of patient data from SLCH clinical systems, searching for combinations of demographic, encounter, laboratory and pharmacy data that suggest that an ADE may have occurred.

Data from SLCH systems is sent in near-real time by HL7 interfaces to a relational database. Triggers for rule evaluations are identified as data are stored in the database, which prompts our Automated Guideline Monitor (AGM) to evaluate these data against rules. The AGM manages the rule base and database queries in the following manner. 20 An application called event handler queries the database and constructs a Virtual Medical Record (VMR) for any patient on whom one or more rules have been triggered. The VMR is translated into an eXtensible Markup Language (XML) message and sent via HTTP to an open source Active BPEL (Business Process Execution Language) engine that employs Web services Business Process Execution Language (BPEL). The BPEL engine executes the given rule and returns a list of one or more clinical decision support actions (e.g., alert, no action, etc). Rules use XPath expression language, a W3C standard for extracting and evaluating XML data. This architecture is shown in [triangle].

Figure 1
Automated guidelines monitor: architecture.

Alerts generated by AGM are displayed on a Web-based user interface for evaluation by pharmacists. For the purposes of this study, the interface was modified to allow for two independent assessments and a final assessment interface for a third reviewer (PMK) that showed all alert details and the two independent assessments. See [triangle].

Figure 2
Example alert web page.

Our rule set was constructed based on our previous work in adult hospitals, 11 but expanded for the pediatric environment. Additional rules were included in an effort to detect certain ADEs that we suspect to be more common in the pediatric environment than in general hospitals, based on previous experience, 16 event reports, and the frequency and use of different medication classes in our hospital. For example, we hypothesized that a rule for detecting seizures secondary to medications might be useful. Also, we suspected that medication-induced electrolyte abnormalities requiring intervention represent a common and potentially under-appreciated type of pediatric ADE. We altered our previous rule for insulin-induced hypoglycemia, requiring a glucose level of 40 mg/dL or less, in response to the large number of clinically insignificant values between 40 and 50 that we detected in our previous work. 13 We also tested a number of rules targeting medication-induced GI dysfunction. The rule set employed during the study period is shown in [triangle].

Table 1
Table 1 ADE Surveillance Rules

The ADE Surveillance Rules

Using this “broad spectrum” rule set we anticipated a high level of false-positive alerts in our oncology population, due to the high incidence of well-recognized and currently unavoidable adverse events from antineoplastic medications. Therefore, for purposes of this specific investigation, we excluded all oncology patients from our data collection and subtracted their numbers from our admission and hospital-day data.

Each of the two study pharmacists (CS, MN) independently reviewed all the resulting alerts using training and evaluation methodologies described previously. 11,13 To review current alerts, they accessed the system's Web site approximately three times per week. The Web site displays all alerts fired by the system that have not yet been reviewed. Selecting an alert from the list displays the information screen containing information about the alert plus critical patient data, including current medication lists, relevant laboratory values, patient weight, and demographic data. The pharmacists had access to other online systems including the hospital pharmacy system and the enterprise clinical data repository to assist them in their evaluation of alerts. They examined every alert independently, reviewing the patient's record to determine whether an ADE had occurred. Each alert was scored for causality using the Naranjo algorithm for determining probability of causality; 21 events with causality scores 5 or higher (probable or definite ADEs) were then scored for severity using the NCC-MERP scoring system ( They also recorded the responsible medications, and a narrative of the event. All pharmacist findings were then reviewed and adjudicated by a physician expert (PMK), whose evaluation served as the gold standard. Events scoring 5 or higher on the Naranjo scale (probable or definite causation), and with NCC-MERP scores of E or higher (indicating harm to the patient) were considered ADEs in this study.


During the six month study period, 6,889 nononcology patients were admitted to the St Louis Children's Hospital, generating 40,250 patient-days. The automated detection system generated 1226 alerts, and detected 160 true ADEs, representing 4 ADEs per 1,000 patient-days, or 2.3 ADEs per 100 admissions. One hundred thirty-five of the events represented temporary harm to the patient (NCC MERP score E); 20 patients suffered temporary harm that required prolonged hospitalization (F), 4 patients suffered permanent harm (G), and one patient died of multisystem disease complicated by drug-induced nephrotoxicity from gentamicin and vancomycin (I) ([triangle]). The most common true-positive alerts were hypokalemia (66), hypomagnesemia (19), nephrotoxicity (18), and naloxone administration (9). The medications most frequently implicated were diuretics, antibiotics, immunosuppressants, narcotics, and anticonvulsants.

Table 2
Table 2 ADEs and Severity by Rule

The ADEs and Severity by Rule

The average age of patients suffering ADEs was 6.3 years, compared with an average age of 6.8 years for all nononcology patients admitted during this period. The greatest number of ADEs occurred in the hospital's critical care units, with 56 (35%) in cardiac intensive care, 43 (27%) in general pediatric intensive care, and 12 (7.5%) in newborn intensive care.

The composite positive predictive value (PPV) of the rule set (e.g., total # ADEs/total # alerts) was 13%; PPV ranged from 100% to 0 ([triangle]). Only three of the 160 ADEs were reported by clinicians through our hospital's voluntary reporting system.

Table 3
Table 3 ADE Rules: Positive Predictive Value (Rules with No ADEs Not Listed)

The ADE Rules: Positive Predictive Value

The study pharmacists were able to evaluate most (80%) of the alerts using just the information available on the Web page. A minority of alerts required them to refer to other online systems (pharmacy system, clinical data repository); only occasionally was it necessary to examine the patient's paper chart. The pharmacists spent an average of 7 hours per week each evaluating the alerts.


The rate of ADEs detected in our study is comparable to that found in pediatric inpatients by Kaushal et al. 14 using manual chart review. It is roughly half the rate that we detected in adults in a general hospital with similar methods and a more limited rule set; 13 however, it is 50% higher than the rate detected in pediatric patients by others using the limited rule set. 15 Seventy percent of ADEs occurred in critical care units, presumably due to the higher per-patient use of hazardous medications in these settings. The average age of patients affected was similar to that of the overall patient population. The nature of ADEs that we found, however, differs from previous studies in several ways.

The proportion of ADEs due to electrolyte-wasting medications (diuretics, antimicrobials, antirejection drugs) is significant. We believe that this represents an important observation, as drug-induced electrolyte depletion severe enough to result in total body deficits requiring intervention qualify as temporary harm, and if not carefully managed can have serious consequences in pediatric patients.

We found few ADEs due to anticoagulation or insulin. This is not surprising given the relatively infrequent use of these medications in pediatrics compared with adult populations. We also found fewer incidences of C. difficile colitis than in our previous work; 13 this may reflect better infection control practices, or other unknown factors. Some of our new “experimental” rules for detection of drug-induced seizures, pancreatitis, and hyponatremia proved to be of no value; they generated 216 false-positive alerts and no true ADEs.

We detected no instances of true heparin-induced thrombocytopenia (HIT) during the study period, despite generating 82 alerts from this rule. This is consistent with literature suggesting HIT is less common in children than in adults. 22,23 We also found that one group of previously useful “traditional” rules, those for elevated aminoglycoside levels, were less useful in our population, detecting only one ADE during the study period. It may be that in the current era of routine pharmacokinetic monitoring of these agents, as practiced at our hospital, these rules will be of less value. Similarly, as many young children carry C. difficile and have clinically insignificant C. difficile toxin in their stool, a positive C. difficile toxin test does not always denote antibiotic-associated colitis. In the current study, 50% of patients with positive C. difficile toxin tests suffered ADEs.

There are several limitations to this study. The intentional exclusion of oncology patients deprives us of information about ADE rates in this high risk population. Given the findings of the current study, we will examine all ADE detection rules with a PPV of less than 10%, eliminating those of no value and potentially modifying those with low, nonzero PPVs with the goal of striking a better balance between review effort and ADE detection. We will include oncology patients in all future studies; early work suggests that a refined rule set will generate a manageable volume of alerts in these patients. We anticipate their ADE rates to be at least equal to those of the current study population.

An inherent limitation of automated surveillance is that the number and types of ADEs that can be detected is limited by the range of data types available to the rule engine. We are evaluating a natural language processing system with the aim of broadening our data capture to include textual data; we expect that this will significantly expand the range of detectable ADE types.


Automated surveillance for ADEs detects harm from medications in pediatric inpatients, and the nature of ADE types in children may differ significantly from adults. Consistent with previous studies, only a tiny fraction of the ADEs detected by automated surveillance were detected by voluntary reporting. We plan to modify our rule set for future use, eliminating rules that yielded low PPV. We will incorporate a natural language processing component into the detection system, enabling us to search discharge summaries, inpatient consult notes, nursing documentation, and other narrative sources for words and phrases suggestive of ADEs. Previous studies by other groups have suggested that this can significantly increase the yield of automated event detection. 4,6,24,25


Supported by AHRQ Grant 1R18HS017010.


1. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324(6):377-384. [PubMed]
2. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado Med Care 2000;38(3):261-271. [PubMed]
3. Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients J Am Med Assoc 1991;266(20):2847-2851. [PubMed]
4. Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report J Am Med Inform Assoc 1998;5(3):305-314. [PMC free article] [PubMed]
5. Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF. High rates of adverse drug events in a highly computerized hospital Arch Intern Med 2005;165(10):1111-1116. [PubMed]
6. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting J Am Med Assoc 2003;289(9):1107-1116. [PubMed]
7. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard medical practice study I. N Engl J Med 1991;324(6):370-376. [PubMed]
8. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors J Am Med Assoc 1998;280(15):1311-1316. [PubMed]
9. Resar RK, Rozich JD, Simmonds T, Haraden CR. A trigger tool to identify adverse events in the intensive care unit Joint Community J Qual Patient Saf 2006;32(10):585-590. [PubMed]
10. Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools Qual Saf Health Care 2003;12(Suppl 2):ii39-ii45. [PMC free article] [PubMed]
11. Kilbridge P. Implementation of a system for computerized Adverse Drug Event Surveillance and intervention at an Academic Medical Center J Clin Outcomes (JCOM) 2006;13(2):94-100.
12. Cullen DJ, Bates DW, Small SD, et al. The incident reporting system does not detect adverse drug events: a problem for quality improvement Jt Comm J Qual Improv 1995;21(10):541-548. [PubMed]
13. Kilbridge PM, Campbell UC, Cozart HB, Mojarrad MG. Automated surveillance for adverse drug events at a community hospital and an academic medical center J Am Med Inform Assoc 2006;13(4):372-377. [PMC free article] [PubMed]
14. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients J Am Med Assoc 2001;285(16):2114-2120. [PubMed]
15. Ferranti J, Horvath MM, Cozart H, Whitehurst J, Eckstrand J. Reevaluating the safety profile of pediatrics: A comparison of computerized adverse drug event surveillance and voluntary reporting in the pediatric environment Pediatrics 2008;121(5):e1201-e1207. [PubMed]
16. Kilbridge P. Unpublished work.
17. McMullin ST, Reichley RM, Kahn MG, Dunagan WC, Bailey TC. Automated system for identifying potential dosage problems at a large University Hospital Am J Health Syst Pharm 1997;54(5):545-549. [PubMed]
18. McMullin ST, Reichley RM, Watson LA, et al. Impact of a web-based clinical information system on cisapride drug interactions and patient safety Arch Intern Med 1999;159(17):2077-2082. [PubMed]
19. Reichley RM, Seaton TL, Resetar E, et al. Implementing a commercial rule base as a medication order safety net J Am Med Inform Assoc 2005;12(4):383-389. [PMC free article] [PubMed]
20. Huang Y, Noirot LA, Heard KM, et al. Migrating toward a next-generation clinical decision support application: the BJC HealthCare experience AMIA Annu Symp Proc 2007:344-348. [PMC free article] [PubMed]
21. Naranjo CA, Busto U, Sellers EM, et al. A method for estimating the probability of adverse drug reactions Clin Pharmacol Ther 1981;30(2):239-245. [PubMed]
22. Boning A, Morschheuser T, Blase U, et al. Incidence of heparin-induced thrombocytopenia and therapeutic strategies in pediatric cardiac surgery Ann Thorac Surg 2005;79(1):62-65. [PubMed]
23. Frost J, Mureebe L, Russo P, Russo J, Tobias JD. Heparin-induced thrombocytopenia in the pediatric intensive care unit population Pediatr Crit Care Med 2005;6(2):216-219. [PubMed]
24. Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients J Am Med Inform Assoc 2001;8(3):254-266. [PMC free article] [PubMed]
25. Honigman B, Light P, Pulling RM, Bates DW. A computerized method for identifying incidents associated with adverse drug events in outpatients Int J Med Inform 2001;61(1):21-32. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association