|Home | About | Journals | Submit | Contact Us | Français|
Clinical event monitors are a type of active medication monitoring system that can use signals to alert clinicians to possible adverse drug reactions. The primary goal was to evaluate the positive predictive values of select signals used to automate the detection of ADRs in the medical intensive care unit.
This is a prospective, case series of adult patients in the medical intensive care unit during a six-week period who had one of five signals presents: an elevated blood urea nitrogen, vancomycin, or quinidine concentration, or a low sodium or glucose concentration. Alerts were assessed using 3 objective published adverse drug reaction determination instruments. An event was considered an adverse drug reaction when 2 out of 3 instruments had agreement of possible, probable or definite. Positive predictive values were calculated as the proportion of alerts that occurred, divided by the number of times that alerts occurred and adverse drug reactions were confirmed.
145 patients were eligible for evaluation. For the 48 patients (50% male) having an alert, the mean ± SD age was 62 ± 19 years. A total of 253 alerts were generated. Positive predictive values were 1.0, 0.55, 0.38 and 0.33 for vancomycin, glucose, sodium, and blood urea nitrogen, respectively. A quinidine alert was not generated during the evaluation.
Computerized clinical event monitoring systems should be considered when developing methods to detect adverse drug reactions as part of intensive care unit patient safety surveillance systems, since they can automate the detection of these events using signals that have good performance characteristics by processing commonly available laboratory and medication information.
Adverse drug reactions (ADRs) are associated with approximately 1.2 million or 3.1% of all hospitalizations in the U.S.  A meta-analysis of ADRs suggest that these events are between the fourth and sixth leading cause of death in the U.S.  Each ADR is estimated to increase the length of hospital stay by 2.2 days and to increase the cost of hospitalization by $3,244.  A systematic review of 29 articles evaluating ADRs in the ICU indicates that the ADR rate ranges between 5.1 and 87.5 per 1000 patient days.  More specifically, the classic evaluation comparing general care units to ICUs reports ADRs occur in approximately 10 per 1000 patient days in the general medical ward, 14 per 1000 patient days in the surgical intensive care unit, and 25 per 1000 patient days in the medical ICU.  Hence, medical ICUs, have the highest incidence of ADRs among patient care units in the hospital setting.
Most hospitals routinely underreport the incidence of ADRs. The main reason is that hospitals rely primarily on voluntary reporting to detect ADRs . The Institute of Medicine recommends that all healthcare settings assess the safety of medication use through active monitoring to improve ADR detection.  A clinical event monitor, an example of an active monitoring system, analyzes electronic patient data when triggered by the presence of certain information (i.e., signals), leading to notification of healthcare providers by alerts. 
Compared with voluntary reporting for ADR detection, hospital studies indicate that clinical event monitors are faster, less expensive, and can identify ADRs not normally detected by clinicians. [9-14] Clinical event monitors have also been shown in certain studies to prevent the development, progression, or mitigate the seriousness of ADRs by enabling the early detection and appropriate response to events in evolution in the hospital and ambulatory care settings. [15-22] Despite the proven benefits of clinical event monitors, few healthcare organizations have implemented and formally evaluated them. [23-26] When used, these systems have frequently been implemented in non-standardized ways that make it difficult to reproduce findings and compare their effectiveness across patients and healthcare settings. [27,28]
Our research group previously developed and evaluated a clinical event monitor for detecting potential ADRs in the nursing home setting. [29,30] The overall positive predictive value (PPV) for the signals using this system was 81% (54/67). Logically, it would make sense to adapt this system to a clinical environment, such as the MICU, that has a high number of ADRs. Also, it would be expected that the PPV of signals would differ by clinical setting since the medications used, the route of administration and the frequency of laboratory monitoring differ in the MICU compared to the nursing home. While other studies have evaluated the PPVs of laboratory test result signals to detect ADRs, most of these evaluations did not use medications combined with laboratory values as signals. Additionally, these studies were not conducted in ICU populations. Moreover, critically ill patients are at an increased risk for ADRs because of frequently changing organ function altering the pharmacokinetics of drugs, alterations in drug absorption, more frequent administration of medications.  Also, the injury associated with ADRs is more severe in this population. [5,32] Consequently, the development and evaluation of clinical event monitor systems to improve the detection and management of ADRs in this clinical setting is particularly important. The primary goal was to evaluate the positive predictive values of select signals used to automate the detection of ADRs in the medical intensive care unit. A secondary goal was to compare the PPVs of ADR alerts generated by a clinical event monitor in the ICU and nursing home settings. Ultimately, the intention of this project is to increase the detection and management of ADRs in the MICU.
The University of Pittsburgh Medical Center (UPMC) Presbyterian MICU is a 32-bed unit staffed by attending and fellow physicians in the division of pulmonary, allergy, and critical care medicine, and internal medicine housestaff. In 2008, there were 1,611 admissions with an average length of stay of 7.1 days. A clinical pharmacist performs rounds on each weekday as part of the MICU patient care team. Adult patients, with the exception of patients on the transplant service, who were in the MICU during a six-week period (6/15/2008-7/24/2008) were included. Transplant patients were excluded because the frequency of abnormal laboratory values and the complexity of their medication regimens makes a drug causality assessment more complex and could influence the PPVs obtained.
We selected five signals with the highest PPVs (≥ 20%; n= 5) from a comprehensive list of signals evaluated in a previous systematic review: elevated BUN (PPV= 0.22), vancomycin (PPV= 0.26) or quinidine concentrations (PPV= 0.5), or low sodium (PPV= 0.25) or glucose concentrations (PPV= 0.27). Each of these 5 signals were associated with the presence of abnormal laboratory values. There is no evidence about PPVs of signals generated by a clinical event monitor in the MICU, so these signals were selected based on their performance in other settings. The criteria for determining the abnormal laboratory test values based on the references ranges at our Medical Center are described in Table 1. This approach for reference ranges has been used previously.  For 3 of the 5 signals, abnormal laboratory tests values were coupled with medications that may cause the laboratory abnormality (i.e., elevated BUN and on medications that can increase BUN, hypoglycemia and on medications that can cause hypoglycemia, and hyponatremia and on a medications that may cause hyponatremia), prior to the generation of a clinical alert. For the 2 remaining signals, the presence of supratherapeutic serum drug concentrations for vancomycin and quinidine were used to generate alerts. This project was approved by the University of Pittsburgh Medical Center (UPMC) Total Quality Council as a quality improvement evaluation. No additional Institutional Review Board was required prior to publication.
For each of the abnormal laboratory test value and medication combination signals (elevated BUN, hypoglycemia, and hyponatremia), the first author (SKG) consulted standard pharmacy reference textbooks to create an initial list of medications that were reported to be associated with a particular laboratory abnormality. A drug information specialist (AHK) expanded the initial list of medications by using additional online references. The drug information specialist then conducted a comprehensive primary literature search using MEDLINE for articles published in the English language between January 1, 1975, and July 1, 2008, using various MeSH terms and limiting the search to adults.
We used a clinical event monitor that had been developed previously for detecting ADRs in the nursing home setting.  The clinical event monitor is a rule-based expert system that is implemented in the Java programming language and employs the Drools inference engine, which is a forward chaining rules engine. Its overall architecture is shown in Figure 1. The knowledgebase consists of IF-THEN rules that were derived from a set of five signals and was implemented in the Drools rule language. On average, the clinical event monitor took less than a minute to process 24 hours worth of data to generate the alerts. The definition for each signal and associated rules are provided in Table 1. The database consists of patient data that includes laboratory results, medication orders, and demographic information extracted from the Medical Archival System (MARS) data repository at our institution. 
Laboratory results, medication orders, and demographic data for the preceding 24 hours for patients in the MICU were obtained automatically once a day from the MARS data repository. The clinical event monitor applied the previously described five laboratory/medication and drug concentration signals and associated knowledgebase of rules to data in the clinical repository. The clinical event monitor would generate an alert after the criteria for the rules associated with a signal have been met in the form of an email sent to a clinical pharmacist for further clinical evaluation as described in the next section below.
Each alert consisted of an abnormal laboratory test and a potential causal medication for a specific patient. If several potential causal medications were associated with the abnormal laboratory test, a separate alert was generated for each medication. The alert contained the patient's demographic information, attending physician, the signal that was triggered and the suspect medication. Repeated firing of the same alert did not occur in a patient unless the abnormal laboratory test value returned to normal (i.e., reset) and subsequently became abnormal a second time. The logic for generating the alert and the assessment of the alert were independent.
For the six-week period, alerts were evaluated to determine if an ADR was present Monday through Friday with the alerts occurring on Saturday and Sunday being evaluated on the subsequent Monday. The definition by Kramer et al. which defines ADRs as, “an undesirable clinical manifestation that is consequent to and caused by the administration of a particular drug” was used since this is the definition used for causality instruments that we applied in the assessment of ADRs. [35,36] In essence, the alerts generated by the clinical event monitor represent a screening test for further investigation of the presence of potential drug induced abnormal laboratory values (ADRs) using standardized assessment and causality instruments. Alerts were assessed by a clinical pharmacist (SKG) using 3 objective published ADR determination instruments including the modified adverse drug reaction scoring system, the adverse drug reaction probability scale and the Jones algorithm. [36-38] These ADR determination instruments have been widely used in previous pharmacoepidemiology studies to determine the likelihood of whether or not the signs, symptoms, or laboratory abnormalities noted are due to a suspect drug, rather than the result of other patient-specific factors such as underlying comorbid illness. [13,39-42] The three instruments contain assessment themes such as previous reports of this ADR, temporal relationship between drug introduction and occurrence of ADR, if discontinuing the drug results in resolution of the ADR, if repeated exposure to the drug causes a similar reaction, and the presence of objective evidence (i.e., serum drug concentrations). An event was considered an ADR when 2 out of 3 instruments had agreement of possible, probable or definite and this was considered our base case or reference scenario for the sensitivity analysis. An agreement of 2 of 3 instruments was used because the reliability of a single instrument to detect ADRs in the ICU has been questioned.  Patient age, gender and a severity of illness score (the simplified acute physiology score (SAPSII)),  were collected for patients with an ADR.
The unit of analysis was a signal generating an alert which is consistent with previously published studies. [42,45,46] To calculate the PPV for each signal, we divided the number of times that a signal generated an alert and an ADR was confirmed (i.e., the number of true-positives), by the number of times the signal generated an alert with or without an ADR being confirmed (i.e., the sum of true-positives and false-positives). Also, the number needed to alert was calculated by the inverse of the PPV which is the number of alerts clinicians need act on to potentially prevent 1 adverse event.  Lastly, the concept of sensitivity analysis to determine the robustness of our data was applied to the criteria for an ADR.  For our evaluation, the most stringent scenario is the application of 3 out of 3 agreement between instruments, whereas, the least stringent scenario is the application a 1 out of 3 agreement between instruments.
Forty-eight of 145 patients (32.4%) had at least one signal resulting in an email alert generated to the clinical pharmacist for determination of the presence of an ADR. The mean ± SD age for the 48 patients (50% male) was 62 ± 19 years and the mean SAPSII score on admission was 38.4 ± 15.8. The total number of medication orders processed during the evaluation period was 3,199. A total of 253 alerts occurred and were investigated for the presence of an ADR. The rate of ADRs was 122.7 per 1000 patient-days. The overall PPV for all signals combined was 0.44, with a range of 0.33 for the BUN signal to 100 for the vancomycin signal (Table 2). This corresponds to a number needed to alert of 2.3. In other words, an ADR was confirmed for each 2.3 alerts generated. The sensitivity analysis indicated that the overall PPV was 0.36 and 0.60 for 3 out of 3 agreement between instruments (most stringent) and 1 out of 3 agreement between instruments (least stringent), respectively. Also, PPVs for the MICU were compared to the data from the literature for the nursing home and a systematic review as shown in Table 3.
This project demonstrates that a clinical event monitor originally developed and evaluated in the nursing home can be can be adapted for use in a different clinical setting such as the MICU where the number of prescriptions is higher and laboratory data are more frequently obtained. This project also demonstrates that the careful selection and development of signals can lead to improved performance characteristics of clinical event monitors for the detection of ADRs. An overall PPV of 0.44 is higher then what has been previously been reported in the literature and illustrates the effectiveness of this system. [13, 40, 45-47] This evaluation also expands on the MICU literature by including the assessment of four additional signals not previously evaluated and expands on studies that have used manual-based approaches to ADR detection. 
In our evaluation, monitoring elevated vancomycin concentrations predicted an ADR in 100% of the cases. The PPV of 1 is likely a reflection of the ADR definition that does not consider patient injury associated with the event. The concern for vancomycin toxicity has declined through the years because of suggestions that vancomycin alone may not be associated with acute kidney injury as frequently as originally expected; as a result, clinicians have been accepting of higher troughs and random concentrations. [49-51] However, some evidence suggests that vancomycin is associated acute kidney injury at higher trough concentrations and depending on certain underlying risk factors. [52,53] Despite the controversy, it appears monitoring vancomycin concentrations with the use of an electronic clinical event monitor would assist with the detection of a hazardous condition for the patient. [33,54,55]
Hypoglycemia has become more concerning in the MICU because of evidence supporting intensive insulin therapy that targets lower glucose concentrations and results in better patient outcomes.  However, aggressive insulin therapy may result in “overshooting” and undesired hypoglycemic events. Using a manual chart review method investigating all hypoglycemic events in a surgical/trauma ICU and predating the new guidelines, yielded a PPV of 0.09, indicating that many hypoglycemic events are due to disease processes and not medications.  In this evaluation, with the use of a clinical event monitor and an advanced hypoglycemia alert that fires for medication/laboratory combinations, the PPV was 0.55. In addition, to the advanced alert, this variation could be a function of the type of ICU (e.g., medical versus cardiac), suggesting the need to evaluate the performance of signals by each clinical environment. The PPV in our evaluation was also higher than those identified in other in-hospital settings, as well. 
Elevated BUN and hyponatremia predicted an ADR in approximately 30% of cases in this project, which is higher than other inpatient settings but less than the nursing home.  Some of the variability in PPVs can be associated with the detection criteria for an ADR. Investigations have used clinical opinion or a single objective ADR assessment instrument.[13, 37-40, 45, 46] In this evaluation, the assessment of ADRs was more rigorous since we used agreement on 2 out of 3 objective instruments which to our knowledge has not been done before. Also, the difference between the PPV for the nursing home and MICU could simply be a function of the environment. Critically ill patients may have different disease processes contributing to hyponatremic and elevated BUN events. This further emphasizes the importance of evaluating the performance of signals for each clinical environment.
Medication prescribing patterns change as new evidence is introduced. While quinidine concentrations had a relatively high PPV in the systematic review, an alert was not generated in this evaluation since it was not prescribed during the course of the project. This suggests that the clinical event monitor knowledgebase must be updated for new therapies and modified for changing criteria of abnormal laboratory test values.
The underlying prevalence of ADRs can influence the actual PPV as noted by a panel of experts convened by the AHRQ.  To put the PPV of 0.44 obtained in this study in context, the rate of adverse events with the potential to cause injury is between 13.8 and 116.8 per 1,000 patient-days in ICUs. [58,59] The rate of ADRs identified in our study was 122.7 per 1000 patient-days. Our rate is at the higher end of the range. This finding is potentially the result of using a computerized surveillance system, which are known to perform better at identifying events associated with laboratory parameters. 
We believe that validating the performance characteristics of specific signals in particular clinical settings is necessary in order to further refine the signal subset prior to health system-wide clinical event monitor implementation. Validating signals in advance of wide spread implementation will likely make the most of scarce resources and address the sociotechnical issue by reducing the clinician's alert burden. Overall, there are several potential benefits to using this strategy as part of the current medication safety surveillance system such as: 1) using signals with high PPVs ensures the efficient use of health information technology and clinical resources; 2) providing a new mechanism can compliment the voluntary reporting of ADRs; and 3) detecting ADRs allows for analysis and future systematic changes in patient care that will potentially save money and lives.
The electronic data repository we used to obtain patient-specific information including laboratory and medication information was not connected with the electronic medication administration record. So, there was a possibility that an alert was generated despite the patient being ordered, but not receiving the medication. The information in the MARS data repository was from documentation in the pharmacy system and this record may have slight variance from the electronic medication administration record used by the nurses. This discrepancy in data would have reduced the PPVs making our calculated PPVs more conservative, but would have been detected during the ADR determination process. We also recognize that the prospective cohort study design did not allow for the inclusion of a control or usual care arm that would have reduced study bias. Also, the alerts were sent to a pharmacist which is common practice; however, the physician response to these alerts is unknown. [30, 39-42] Since this study was conducted at a single institution, it is also possible that the patient population that was evaluated may not be similar to other MICU patients. We attempted to better characterize these patients and make our findings more generalizable by including acute physiology scores for severity of illness. The clinical event monitor described was developed at our institution and requires significant local clinical and informatics expertise, in addition to certain computer resources that may not be widely available. However, we do believe that clinical event monitors used for the detection of potential ADRs in ICUs are generalizable to other institutions, as there are companies that sell systems with similar functionality, such as TheraDoc® and QS/1®.
We also realize that there are likely unique PPVs for each laboratory/medication combination. One aggregate calculated PPV for a signal as reported in this evaluation may not be sufficient. However, to put this in context, although the overall PPV of 0.44 may seem low, this value is significantly higher than recommended screening tests such as fecal occult blood testing to detect colorectal cancer which have PPVs that range from 0.02 to 0.18 in adults over 50 years old. The acquired knowledge from this clinical event monitor can be used to improve the system and further improve PPVs. We also recognize that a low PPV does not indicate that signal should not be investigated especially if the ADR is serious. The limitation of using a PPV as an evaluation endpoint is that it does not allow for the calculation of sensitivity and specificity, false negative alerts, and is influenced by the underlying event prevalence. We have only just begun to understand the performance of the signals and there is a lot of opportunity for improvement.
The definition of an ADR used in this study is an undesirable clinical manifestation that is consequent to and caused by the administration of a particular drug. Consequently, supratherapeutic medication concentrations, such as vancomycin, represent an ADR. Assessing the patient-specific clinical outcome associated with this abnormal laboratory value, such as drug-induced nephrotoxicity was not assessed. An evaluation of the impact of alerts on patient care was not conducted, as part of the quality improvement project we were determining how best to build the system before full implementation. We intend on conducting studies in the future to determine if alerts associated with potential ADRs can provide the clinician with the opportunity to prevent temporary or permanent end organ damage (i.e., an ameliorable ADR).
Computerized clinical event monitoring systems should be considered when developing methods to detect ADRs as part of ICU patient safety surveillance systems since they can automate the detection of these events using certain signals that have good performance characteristics by processing commonly available laboratory and medication information.
This project was supported in part by a University of Pittsburgh Medical Center (UPMC) eRecord grant, National Institutes of Health (NIH) grant 5KL2RR024151 (NIH Roadmap Multidisciplinary Clinical Research Career Development Award Grant), and an Agency for Healthcare Research Quality (AHRQ) R01HS018721. We would like to acknowledge Dr. Jay Shah for his clinical advice during this process, Elizabeth Forsberg for her assistance with data management and Andrea Ho-Kean the drug information specialist that assisted with the on-line reference review. We would like to acknowledge the Society of Critical Care Medicine for presenting these data in part at the 2009 Annual Congress, Nashville, TN.
Author's Contribution:Sandra L. Kane-Gill, PharmD, MS; 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
Shyam Visweswaran, MD, PhD: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
Melissa I. Saul, MS: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
An-Kwok Ian Wong, MS: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
Louis E. Penrod, MD: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
Steven M. Handler, MD, PhD: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.
Conflict of Interest Statement: No author has a conflict of interest with the contents of this manuscript.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.