This project demonstrates that a clinical event monitor originally developed and evaluated in the nursing home can be can be adapted for use in a different clinical setting such as the MICU where the number of prescriptions is higher and laboratory data are more frequently obtained. This project also demonstrates that the careful selection and development of signals can lead to improved performance characteristics of clinical event monitors for the detection of ADRs. An overall PPV of 0.44 is higher then what has been previously been reported in the literature and illustrates the effectiveness of this system. [13
] This evaluation also expands on the MICU literature by including the assessment of four additional signals not previously evaluated and expands on studies that have used manual-based approaches to ADR detection. [33
In our evaluation, monitoring elevated vancomycin concentrations predicted an ADR in 100% of the cases. The PPV of 1 is likely a reflection of the ADR definition that does not consider patient injury associated with the event. The concern for vancomycin toxicity has declined through the years because of suggestions that vancomycin alone may not be associated with acute kidney injury as frequently as originally expected; as a result, clinicians have been accepting of higher troughs and random concentrations. [49
] However, some evidence suggests that vancomycin is associated acute kidney injury at higher trough concentrations and depending on certain underlying risk factors. [52
] Despite the controversy, it appears monitoring vancomycin concentrations with the use of an electronic clinical event monitor would assist with the detection of a hazardous condition for the patient. [33
Hypoglycemia has become more concerning in the MICU because of evidence supporting intensive insulin therapy that targets lower glucose concentrations and results in better patient outcomes. [56
] However, aggressive insulin therapy may result in “overshooting” and undesired hypoglycemic events. Using a manual chart review method investigating all hypoglycemic events in a surgical/trauma ICU and predating the new guidelines, yielded a PPV of 0.09, indicating that many hypoglycemic events are due to disease processes and not medications. [33
] In this evaluation, with the use of a clinical event monitor and an advanced hypoglycemia alert that fires for medication/laboratory combinations, the PPV was 0.55. In addition, to the advanced alert, this variation could be a function of the type of ICU (e.g., medical versus cardiac), suggesting the need to evaluate the performance of signals by each clinical environment. The PPV in our evaluation was also higher than those identified in other in-hospital settings, as well. [13
Elevated BUN and hyponatremia predicted an ADR in approximately 30% of cases in this project, which is higher than other inpatient settings but less than the nursing home. [13
] Some of the variability in PPVs can be associated with the detection criteria for an ADR. Investigations have used clinical opinion or a single objective ADR assessment instrument.[13
] In this evaluation, the assessment of ADRs was more rigorous since we used agreement on 2 out of 3 objective instruments which to our knowledge has not been done before. Also, the difference between the PPV for the nursing home and MICU could simply be a function of the environment. Critically ill patients may have different disease processes contributing to hyponatremic and elevated BUN events. This further emphasizes the importance of evaluating the performance of signals for each clinical environment.
Medication prescribing patterns change as new evidence is introduced. While quinidine concentrations had a relatively high PPV in the systematic review, an alert was not generated in this evaluation since it was not prescribed during the course of the project. This suggests that the clinical event monitor knowledgebase must be updated for new therapies and modified for changing criteria of abnormal laboratory test values.
The underlying prevalence of ADRs can influence the actual PPV as noted by a panel of experts convened by the AHRQ. [57
] To put the PPV of 0.44 obtained in this study in context, the rate of adverse events with the potential to cause injury is between 13.8 and 116.8 per 1,000 patient-days in ICUs. [58
] The rate of ADRs identified in our study was 122.7 per 1000 patient-days. Our rate is at the higher end of the range. This finding is potentially the result of using a computerized surveillance system, which are known to perform better at identifying events associated with laboratory parameters. [12
We believe that validating the performance characteristics of specific signals in particular clinical settings is necessary in order to further refine the signal subset prior to health system-wide clinical event monitor implementation. Validating signals in advance of wide spread implementation will likely make the most of scarce resources and address the sociotechnical issue by reducing the clinician's alert burden. Overall, there are several potential benefits to using this strategy as part of the current medication safety surveillance system such as: 1) using signals with high PPVs ensures the efficient use of health information technology and clinical resources; 2) providing a new mechanism can compliment the voluntary reporting of ADRs; and 3) detecting ADRs allows for analysis and future systematic changes in patient care that will potentially save money and lives.
Limitations and Opportunities to Improve PPVs
The electronic data repository we used to obtain patient-specific information including laboratory and medication information was not connected with the electronic medication administration record. So, there was a possibility that an alert was generated despite the patient being ordered, but not receiving the medication. The information in the MARS data repository was from documentation in the pharmacy system and this record may have slight variance from the electronic medication administration record used by the nurses. This discrepancy in data would have reduced the PPVs making our calculated PPVs more conservative, but would have been detected during the ADR determination process. We also recognize that the prospective cohort study design did not allow for the inclusion of a control or usual care arm that would have reduced study bias. Also, the alerts were sent to a pharmacist which is common practice; however, the physician response to these alerts is unknown. [30
] Since this study was conducted at a single institution, it is also possible that the patient population that was evaluated may not be similar to other MICU patients. We attempted to better characterize these patients and make our findings more generalizable by including acute physiology scores for severity of illness. The clinical event monitor described was developed at our institution and requires significant local clinical and informatics expertise, in addition to certain computer resources that may not be widely available. However, we do believe that clinical event monitors used for the detection of potential ADRs in ICUs are generalizable to other institutions, as there are companies that sell systems with similar functionality, such as TheraDoc® and QS/1®.
We also realize that there are likely unique PPVs for each laboratory/medication combination. One aggregate calculated PPV for a signal as reported in this evaluation may not be sufficient. However, to put this in context, although the overall PPV of 0.44 may seem low, this value is significantly higher than recommended screening tests such as fecal occult blood testing to detect colorectal cancer which have PPVs that range from 0.02 to 0.18 in adults over 50 years old. The acquired knowledge from this clinical event monitor can be used to improve the system and further improve PPVs. We also recognize that a low PPV does not indicate that signal should not be investigated especially if the ADR is serious. The limitation of using a PPV as an evaluation endpoint is that it does not allow for the calculation of sensitivity and specificity, false negative alerts, and is influenced by the underlying event prevalence. We have only just begun to understand the performance of the signals and there is a lot of opportunity for improvement.
The definition of an ADR used in this study is an undesirable clinical manifestation that is consequent to and caused by the administration of a particular drug. Consequently, supratherapeutic medication concentrations, such as vancomycin, represent an ADR. Assessing the patient-specific clinical outcome associated with this abnormal laboratory value, such as drug-induced nephrotoxicity was not assessed. An evaluation of the impact of alerts on patient care was not conducted, as part of the quality improvement project we were determining how best to build the system before full implementation. We intend on conducting studies in the future to determine if alerts associated with potential ADRs can provide the clinician with the opportunity to prevent temporary or permanent end organ damage (i.e., an ameliorable ADR).