|Home | About | Journals | Submit | Contact Us | Français|
Hospital medication safety event detection predominantly emphasizes the identification of preventable adverse drug events (ADEs) through self-reports. These relatively rare events only provide insight into patient harm and self-reports identify only a small portion of ADEs. A broader system-focused approach to medication safety event detection that uses an array of event detection methods is recommended. This approach illuminates medication system deficits and supports improvement strategies that can prevent future patient risk.
To: (i) describe a system-focused approach to hospital medication safety event detection, and (ii) present a case illustration of approach application.
A three-level medication safety event detection model that ranges from a narrow harm-focused to broader system-focused approach is described. A standardized cross-level methodology to detect medication safety events is presented.
A Level 3 system-focused methodology that incorporated both voluntary and non-voluntary event detection strategies was used in 17 critical care (n = 4), intermediate care (n = 7) and medical-surgical units (n = 6) across two hospitals. A total of 431 events were detected: 78 (18.1%) ADEs and 353 (81.9%) potential ADEs. Of the 353 PADEs, 302 (70.0%) were non-intercepted events. Non-voluntary detection methods yielded the majority of events (367, 85.1%).
The incidence of ADEs was low when compared with non-intercepted PADEs. This was indicative of medication safety system failures that placed patients at risk for potential harm. Non-voluntary detection methods were much more effective at detecting events than traditional self-report methods.
The publication of the Institute of Medicine report entitled To Err is Human: Building a Safer Health Care System raised the nation's consciousness about health-care safety . Despite the increasing research on patient safety, challenges remain with the translation of new approaches into health-care system improvements . This is particularly true for medication safety. As noted by Classen and Metzger , ‘the use of medications remains the most common intervention in health care’ (p. i4l). Adverse drug events (ADEs) occur in 3.7–30% of academic hospital admissions and 28% of these events are preventable [4–6]. These events are predominantly system problems that can, in large part, be addressed with medication process system improvements . However, the complexity of the medication process makes medication process systems prone to latent failures that can lead to unanticipated outcomes [8–10]. The objectives of the paper are to: (i) describe a system-focused approach to hospital medication safety event detection and (ii) present a case illustration of the application of the approach.
The detection of medication safety events is routinely incorporated into hospital risk management and quality improvement processes. Medication safety events are commonly labeled as medication errors that represent variations from established standards of medication safety practice. These errors are typically detected with self-report methods. The challenge with the use of the term medication error is that it presumes an error that has occurred prior to a careful evaluation of the medication safety event. This approach may limit identification of root causes for the event that are required to improve medication safety system deficits. Evidence has indicated that the subjective self-report detection approach under identifies the number of actual medication safety events, which tempers the effectiveness of medication safety quality improvement efforts.
The medication safety event detection model in Fig. 1 depicts an assessment continuum. The model is predicated on identification of medication safety events that are subsequently evaluated and categorized as either error events or non-error events. Medication error events include preventable ADEs, intercepted potential ADEs (PADEs) and non-intercepted PADEs. Non-error events include non-preventable ADEs. Preventable ADEs are defined as harmful medication errors that can be prevented by existing clinical practices. Intercepted PADEs are potentially harmful medication errors that do not reach patients due to adequate system checks and balances. Non-intercepted PADEs are potentially harmful medication errors that reach patients due to medication system failures. Non-preventable ADEs are non-error events that harm patients but are beyond clinician control, e.g. administration of penicillin to a patient with an unknown penicillin allergy.
The primary target of most hospital medication risk management activities is preventable ADEs. These events harm patients, increase litigation potential and grab headlines. The risk management goal is to mitigate the risk of these events. While this emphasis is understandable, it is narrow. It excludes other event types that reflect medication safety system deficits that require quality improvement. The model in Fig. 1 incorporates all types of medication safety events and integrates them into an assessment approach that ranges from a narrow Level 1 preventable ADE or harm-focused approach, to a broader Level 3 or system-focused approach. All levels address patient harm from preventable ADEs. However, each higher level of the continuum becomes more inclusive and provides a broader view of medication system effectiveness. Selection of level focus depends on the overall aims of medication safety event assessment. For example, a Level 1 focus addresses only preventable ADEs in terms of their incidence, characteristics and severity [9–11]. A Level 2 focus expands assessment to include non-intercepted potential ADEs (PADEs) that provide information about medication system failures. A Level 3 focus assesses preventable ADEs, and medication system effectiveness in terms of system failures (non-intercepted PADEs) and adequate system safeguards (intercepted PADEs). Non-preventable ADEs should be included in the Level 3 assessment as they provide insight into current clinical practice evidence and practice standards that require review. Clinical evidence and practice standards change quickly. The inclusion of non-preventable ADEs provides a baseline by which to assess, over time, the impact of new clinical protocols and scientific evidence on medication safety event outcomes. In time, these practice innovations may change non-preventable ADEs to preventable ADEs.
As depicted in Fig. 1, effective medication safety event assessment requires the use of standardized measurement and data collection approaches across all assessment levels. This ensures a consistent, rigorous and comprehensive assessment approach.
As depicted in Fig. 1, a standardized medication safety event measurement approach incorporates: (i) event definitions; (ii) event severity classification; (iii) event preventability criteria; (iv) control event capture; (v) standardized recording tools; and (vi) rater training and ongoing inter-rater reliability (IRR) evaluation.
In 1996, the National Coordinating Council Medication Error Reporting and Prevention (NCC MERP) index was created to standardize medication error definitions and error severity classification . The index was revised to its current form in 2001 and consists of nine error-focused severity categories (A–I) . Additional research found that there was a substantial level of agreement (κ = 0.60) for the overall index [13, 14]. While the index has enhanced standardized definition and classification, its error-driven focus is limited due to the exclusion of non-preventable ADEs. To address this limitation, the index should be adapted as depicted in Table 1 by modifying the word ‘error’ to ‘event’ in category descriptions.
The error emphasis of the NCC MERP index also prevents inclusion of preventability criteria to distinguish preventable from non-preventable ADEs. To address this deficit, it is recommended that Schumock and Thornton's  seven preventability criteria, which range from drug appropriateness to compliance, be used to classify the preventable and non-preventable ADEs (Table 1).
The sensitivity of event detection methods and accuracy of event classification are frequent challenges during the early development of medication safety quality improvement programs. The capture of laboratory abnormality and excluded events can provide insight into the accuracy of event classification, and detection method sensitivity. Laboratory abnormalities are defined as events that are identified by a critical laboratory value in association with a trigger drug for event detection, e.g. blood glucose/dextrose 50%. These events reflect a potentially harmful change in patient condition that, while associated with the use of a medication, is not considered a medication error at the time of event classification. The separate classification of laboratory abnormality events is considered an important measurement control strategy to prevent inflated error rates.
Excluded events are defined as those events that initially appear to be actual or potential ADEs, but, on further investigation, are determined to be false positives. For example, diphenhydramine, a trigger medication indicative of a medication allergic reaction, that is ordered as a stand-by medication prior to a blood transfusion, is an excluded event. Capture of excluded events is considered an important measurement control strategy as they provide valuable information about the sensitivity of event detection methods and classification decision rules.
Observational methods require the use of standardized recording tools, rater training and periodic assessment of tool use across raters, i.e. IRR. These methods ensure that data related to each medication safety event are being consistently captured and recorded on assessment tools. This consistency supports development of high-quality databases by which to conduct medication safety quality improvement analyses. Established recording tools should be used whenever possible to increase data collection rigor and data quality. Two examples of strong recording tools are the Event Identification and Event Classification Forms, developed by Bates et al.  for the ADE Prevention Study. Previous IRR between physician raters using the Event Identification Form was supported with kappas ranging from 0.81 to 0.98, considered excellent, and percentage agreements from 92.5 to 98.5%. Event Classification Form IRR judgments about ADE presence and preventability was supported with kappas ranging from 0.81 to 0.98 and percentage agreements from 92.5 to 98.5%. Event Classification Form IRR judgments about ADE outcome severity kappas ranged from 0.32 to 0.37, considered fair, and percentage agreements from 66 to 85%.
As depicted in Fig. 1, consistent data collection processes must be used for effective medication safety event assessment. They typically include: (i) multi-method event detection; (ii) event classification; and (iii) event detection and classification decision rules.
Medication safety studies have reported event detection methods that range from non-voluntary (e.g. computer triggers) to voluntary methods (e.g. written incident reports) [4, 16, 17]. Self-report, computerized monitoring triggers and chart review have been the most commonly used methods. Reported sensitivity of these methods has varied with detection yields of 2–4% for self-reports; 45–58% for computerized monitoring triggers and 65% for chart reviews [17–19]. The type of information provided has also varied by method. Jha et al.  found that computerized monitoring triggers were more reliable in detecting events associated with changes in laboratory values, whereas chart review more reliably detected events manifested by symptoms, e.g. pain or shortness of breath. Jha et al.  also found that there was limited overlap in the types of events captured by different detection methods. These findings supported the use of a multi-method event detection approach. In a study to evaluate patient medication safety in an academic medical center, Senst et al.  used a multi-method approach comprised of voluntary self-reports, computerized flags (i.e. triggers), a 5% random daily chart review and ICD 9 medical record diagnostic codes. Findings indicated yields of 6.8% from voluntary self-reports; 73% from computer flags; 5.4% from random chart review and 8.1% from medical record codes .
Based on the prior evidence, it is clear that a multi-method event detection approach provides the most comprehensive and effective way by which to identify medication safety events. As reflected in Fig. 2, a multi-method approach can be implemented that incorporates a range of non-voluntary detection methods, i.e. Pyxis®  trigger reports, laboratory value trigger reports, 5% chart review and pharmacist surveillance, to voluntary detection methods, i.e. medication safety hotline reports, verbal reports (i.e. solicited and unsolicited), written incident reports and other methods, e.g. MEDMARX® reports .
Once medication safety events are detected they must be evaluated and classified. Classification is typically done by in-house experts, e.g. a sub-group of the Pharmacy and Therapeutics Committee, using a standardized approach to guide and enhance consistency. For example, based on Fig. 2, each event is initially evaluated as to whether it caused patient harm. If patient harm occurred (i.e. ‘Yes’), then further classification decisions shift to the ADE or right side of the algorithm. The next decision point assesses whether a medication error occurred, i.e. a preventable ADE. The final decision point considers ADE severity based on adapted NCC MERP index severity categories (Table 1). A comparable process is used to traverse the other sections of the algorithm for event classification.
Rater decisions are made as medication safety event detection and classification is done. Each event has unique elements that make consistency of rater decisions a challenge. It is important that rater decisions be captured to determine the need for development of ‘decision rules’ that can support the consistency of subsequent event detection and classification.
A Level 3 system-focused medication safety event assessment was conducted on 17 critical care (n = 4), intermediate care (n = 7) and medical-surgical units (n = 6) in two community hospitals in a Southern California health-care system. Data were collected on Monday through Friday for a total of 20 days. Weekends were excluded from data collection due to unit variations, such as weekend staffing level, that could affect medication error rates. Consistent with Fig. 1, Level 3 data were collected to determine: (i) the degree of patient harm from preventable ADEs; (ii) medication system effectiveness as indicated by intercepted and non-intercepted PADEs; and (iii) baseline assessment of current medication safety practices as indicated by non-preventable ADEs.
The standardized methodology illustrated in Fig. 1 and described above was used to conduct the assessment. Event Identification and Event Classification Forms, developed by Bates et al. , were adapted for use as recording tools. Tool modifications included the addition of: (i) the adapted NCC MERP index criteria and (ii) Schumock and Thornton's  preventability criteria.
Four registered nurse case investigators were trained to use the multi-method detection approaches depicted in Fig. 2 to identify medication safety events and record information on Event Identification Forms. Event classification was done by a clinical pharmacist and three physician case reviewers who used the Fig. 2 algorithm for event classification, and recorded their decisions on Event Classification Forms. Case scenarios were used to train raters in the use of recording tools and Fig. 2 algorithm event classification process prior to beginning the assessment. Periodic IRR evaluation was done during the assessment to determine the levels of agreement for ADE and PADE severity classification for discrete and combined index categories . Findings indicated that the level of rater agreement for judgment-based classification was substantial for discrete severity categories (κ = 0.67), and almost perfect for combined categories (κ = 0.84) [14, 23].
During the assessment, 96 decision rules were proposed by registered nurse case investigators and approved by case reviewers to facilitate event detection and classification consistency. For example, Imodium was used as a trigger medication to detect events secondary to diarrhea from a medication reaction. If Imodium was administered to treat diarrhea secondary to a gastro-intestinal tract surgical procedure, then the detected event was excluded.
During the 20-day event assessment period, a total of 1052 medication safety events were detected among 8245 patient days and 1499 patient admissions across the 17 clinical units. Of the 1052 detected events, 96 (9.1%) laboratory abnormality events and 525 (49.9%) excluded events were not included in final analyses. Findings for the remaining 431 events are discussed in terms of: (i) detection method yields by event type; (ii) detection trigger yields; and (iii) severity by event type.
As indicted in Table 2, the multi-method event detection approach included non-voluntary methods, voluntary methods and other methods. Findings for each method will be summarized in the following sections. Detection method yields for excluded events, i.e. ‘false positives,’ are also described.
Non-voluntary methods yielded a total of 367 (85.1%) events from 5% retrospective chart review, pharmacist surveillance, Pyxis®  trigger reports and laboratory value trigger reports. Chart reviews detected the highest number of events (n = 145, 33.6%), with laboratory value trigger reports detecting the lowest number of events (n = 31, 7.1%) (Table 2). However, some detection methods were more predictive of specific event types than other methods. For example, Pyxis®  triggers detected more preventable (n = 16, 3.7%) and non-preventable (n = 41, 9.5%) ADEs than other methods, whereas chart review detected the most non-intercepted PADEs (n = 119, 27.6%). Pharmacist surveillance also detected a substantial number of non-intercepted PADEs (n = 91, 21.1%) as well as the most intercepted PADEs (n = 24, 5.5%).
The positive predictive validity for non-voluntary methods were: (i) Pyxis®  trigger report (26.4%); (ii) laboratory value trigger report (15.6%); (iii) 5% retrospective chart review (70.0%); and (iv) pharmacist surveillance (66.7%). These findings indicated a lack of sensitivity of single Pyxis®  and laboratory value triggers, and supported a need for composite triggers that can potentially enhance trigger sensitivity. A total of four events were identified with two combined detection methods, which included glucose tabs/gel + serum glucose level and dextrose 50% + serum glucose. This finding provided preliminary evidence of an area for composite trigger development. Of the 10 Pyxis®  trigger medications, glucose tabs/gel ranked first with a yield of 19 (25.7%) events. Of the 7 laboratory value triggers, potassium ranked first with a yield of 8 (25.0%) events. Of the 10 pharmacist surveillance activities, renal drug dosing ranked first with a yield of 72 (62.1%) events.
Voluntary methods yielded a total of 53 (12.2%) events from medication safety hotline reports, solicited verbal reports, unsolicited verbal reports and written incident reports. Written incident reports detected the highest number of events (n = 48, 11.1%). These reports were most effective in detecting non-intercepted PADEs (n = 42, 9.7%). The positive predictive validity for voluntary methods was unsolicited verbal report (100.0%), medication safety hotline (80.0%), written incident report (77.4%), and solicited verbal report (0.0%).
Other methods yielded a total of 15 (3.4%) events from complimentary event detection methods such as case investigation, case review and MEDMARX®  reports. These methods were most effective at detecting PADEs (n = 10, 2.3%).
Excluded events (n = 525, 49.9%) were detected primarily with non-voluntary Pyxis®  trigger reports (n = 202, 38.4%), and laboratory value trigger reports (n = 169, 32.1%). Of the 10 Pyxis®  trigger medications for excluded events, diphenhydramine ranked first with a yield of 60 (29.7%) events. Of the eight laboratory value triggers, potassium ranked first with a yield of 103 (60.9%) events. These findings further supported a need for improved sensitivity and specificity of these non-voluntary detection methods.
Table 3 reflects event severity by NCC MERP index grouped severity categories, i.e. no harm, harm and death; and discrete severity categories, i.e. B–I. Grouped category descriptors reflect adapted index category descriptions that did not use the word ‘error.’ ADEs are rated with severity categories E–I. Of the 431 events, 23 (5.3%) were classified as preventable ADEs and 55 (12.7%) as non-preventable ADEs. The majority of preventable ADEs (n = 17, 3.9%) and non-preventable ADEs (n = 50, 11.6%) were classified at a Category E severity level, i.e. events that contributed to or resulted in temporary patient harm and required intervention. Of the 431 events, 51 (11.8%) were classified as intercepted PADEs and 302 (70.0%) as non-intercepted PADEs. All of the intercepted PADEs were classified at a Category B severity level, i.e. events that did not reach the patient, whereas the majority of the non-intercepted PADEs (n = 246, 57.1%) were classified at a Category C severity level, i.e. events that reached the patient but caused no harm. Overall, findings indicated that the majority of harmful preventable and non-preventable ADEs had a low level of severity. Of the potentially harmful PADEs, a high number of events reached patients but had limited potential to harm them.
Findings indicated a low incidence of harmful ADEs, relative to previous findings, and the majority of these events were of low severity. A lower proportion of ADEs were classified as preventable when compared with previous findings. The majority of PADEs were non-intercepted events that reached the patient. While they were generally of low severity, there were also more severe non-intercepted PADEs. This finding underscored the need for medication system redesign and improvement. If a Level 3 system-focused medication safety event assessment approach had not been used, these events would not have been detected and system improvement needs not discovered. Non-intercepted PADEs represented a warning about medication system failures that provide an opportunity for system improvement that can prevent future events that may cause patient harm.
The standardized methodology enhanced event assessment rigor and process consistency. The use of a multi-method detection approach was particularly helpful as findings indicated that the majority of events were detected with non-voluntary methods. This reinforced the limitations of strict reliance on a self-report process for detecting medication safety events. Detection methods displayed some uniqueness in terms of the type of events they identified. However, a significant limitation was the high yield of false positives, i.e. excluded events and low positive predictive validity of Pyxis®  medication triggers and laboratory value triggers. Limited preliminary evidence emerged to support development of composite triggers that would combine selected Pyxis®  and laboratory value triggers to enhance their detection sensitivity. While yields were low for voluntary detection methods, they were much more specific for true events that non-voluntary methods. The low yields were consistent with previous medication safety studies [17–19], and provided further support for the use of a multi-method detection approach that includes non-voluntary methods.
Decision rules have been used in numerous medication safety studies to ensure recurrent decision consistency in event detection and classification [9, 17, 24, 25]. These rules are typically developed from setting-specific clinical protocols and practice standards. This approach is essential to a rigorous medication safety event assessment. However, significant rule set variation may exist across health-care settings that limit benchmarking potential.
The Level 3 system-focused medication safety event assessment approach proved to be an effective way to gain insight into harmful ADEs, as well as potential ADEs that were indicative of medication system deficits. While the event assessment process was effective, a few caveats exist. First, although a multi-method detection approach was used to identify events, some events were undoubtedly missed. Second, although IRR was generally good, the reliance on multiple reviewers may have introduced bias during the event classification process. Finally, the observational approach and need for trained raters makes the assessment process labor-intensive and, correspondingly, expensive. These caveats become important when considering adoption of the Level 3 system-focused medication safety event assessment approach. They must be carefully weighed to determine what will best serve the organization's medication safety quality improvement goals within existing resources.
This work was supported by Agency for Healthcare Research and Quality [grant R01 HS013131].
The authors gratefully acknowledge the expertise and valuable contributions of research team members who supported the study that made this manuscript possible.