Description of the development and validation of the Canadian Paediatric Trigger Tool

doi:10.1136/bmjqs.2010.041152

BMJ Qual Saf. 2011 May; 20(5): 416–423.

Published online 2011 January 17. doi: 10.1136/bmjqs.2010.041152

PMCID: PMC3088437

Description of the development and validation of the Canadian Paediatric Trigger Tool

Anne G Matlow,^1,^2,³ Catherine M G Cronin,^4,⁵ Virginia Flintoft,⁶ Cheri Nijssen-Jordan,⁷ Mark Fleming,⁸ Barbara Brady-Fryer,⁹ Mary-Ann Hiltz,¹⁰ Elaine Orrbine,¹¹ and G Ross Baker^3,⁶

¹Department of Paediatrics, Hospital for Sick Children, Toronto, Canada

²Department of Paediatrics, University of Toronto, Toronto, Canada

³University of Toronto Centre for Patient Safety, Toronto, Canada

⁴Department of Paediatrics and Child Health, University of Manitoba, Winnipeg, Canada

⁵Child Health Program, Winnipeg Regional Health Authority, Winnipeg, Canada

⁶Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada

⁷Department of Paediatrics, Alberta Children's Hospital, University of Calgary, Calgary, Canada

⁸Department of Psychology, Saint Mary's University, Halifax, Canada

⁹Grant MacEwan University, Edmonton, Canada

¹⁰IWK Health Centre, Halifax, Canada

¹¹Canadian Association of Paediatric Health Centres, Ottawa, Canada

Corresponding author.

Correspondence to Dr Anne Matlow, Infection Prevention & Control Programme, The Hospital for Sick Children, 555 University Avenue, Toronto, ON, Canada; Email: anne.matlow/at/sickkids.ca

Accepted December 6, 2010.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

This article has been cited by other articles in PMC.

Abstract

Objective

To describe the process of developing and validating the Canadian Association of Paediatric Health Centres Trigger Tool (CPTT).

Methods

Five existing trigger tools were consolidated with duplicate triggers eliminated. After a risk analysis and modified Delphi process, the tool was reduced from 94 to 47 triggers. Feasibility of use was tested, reviewing 40 charts in three hospitals. For validation, charts were randomly selected across four age groups, half medical/half surgical diagnoses, from six paediatric academic health sciences centres. 591 charts were reviewed by six nurses (for triggers and adverse events (AEs)) and three physicians (for AEs only). The incidence of trigger- and AE-positive charts was documented, and the sensitivity and specificity of the tool to identify charts with AEs were determined. Identification of AEs by nurses and physicians was compared. The positive predictive value (PPV) of each trigger was calculated and the ratio of false- to true-positive AE predictors analysed for each trigger.

Results

Nurses rated the CPTT easy to use and identified triggers in 61.1% (361/591; 95% CI 57.2 to 65.0) of patient charts; physicians identified AEs in 15.1% (89/ 591, 95% CI 0.23 to 0.43). Over a third of patients with AEs were neonates. The sensitivity and specificity were 0.88 and 0.44, respectively. Nurse and physician AE assessments correlated poorly. The PPV for each trigger ranged from 0 to 88.3%. Triggers with a false/true-positive ratio of >0.7 were eliminated, resulting in the final 35-trigger CPTT.

Conclusions

The CPTT is the first validated, comprehensive trigger tool available to detect AEs in children hospitalised in acute care facilities.

Keywords: Trigger, adverse event, harm, patient safety, research

Introduction

International data demonstrate that the incidence of adverse events (AEs) in hospitalised adults ranges from 2.9% to 16.6%.1–12 Physiological and developmental differences in children could make them more vulnerable to harm, yet no data from methodologically similar studies in full paediatric populations exist. Published AE rates in children range from 0.2 to 154 per 10 000 discharge records,13 and 1 to 2.96 per 100 discharges.14 15 Chart reviews targeting specific paediatric populations and specific types of AEs reveal much higher rates.16–18 Identifying a composite paediatric AE rate has been hampered by the absence of a single tool suitable for examining AEs across paediatric populations and events.

No single method can identify all harm associated with patient care. Concurrent use of complementary methods in real time and retrospectively for AE detection has been recommended.19–24 Trigger tool (TT) methodology, however, is considered the best single tool, given its tiered approach and sensitivity to AE detection. It can be customised to diverse settings and, used consistently, can accurately measure harm over time.25 The use of screening criteria to identify charts with possible harm followed by an in-depth review for actual harm has been used for decades, including the landmark Harvard Medical Practice Study.1–11 More recently, the Institute for Healthcare Improvement (IHI) has recommended use of their Global Trigger Tool (IHI-GTT) for local quality-improvement activities.26 Although focused paediatric TTs have been described, there remains no TT validated to detect AEs across all age groups of hospitalised children and youth.

There are growing efforts to improve patient safety in paediatrics, and quantification of the burden of iatrogenic harm in children could catalyse awareness and stimulate changes in paediatric practice and healthcare policy, as occurred following publication of adult AE rates. The Patient Safety Collaborative of the Canadian Association of Paediatric Health Centres established a working group to develop a paediatric TT (CPTT) to detect AEs in children hospitalised in Canada.27 The goal was to develop a reliable and robust tool that could be used for both local quality-improvement activities and research into the rate, incidence, and factors contributing to AEs across a large population. The rigorous processes used to develop, test the feasibility of use of and validate the CPTT in detecting AEs are outlined in this paper.

Patients and methods

The early development stages of the CPTT have been previously reported.27 Briefly, five TTs identified through a detailed literature review and personal communications were adapted to the modular format of the IHI-GTT. The Care, Medication, Surgical and Intensive care modules were retained, and two modules added: Laboratory and Other. Screening criteria from all other TTs were mapped against this list, duplicate triggers were removed, and additional triggers recommended and approved by consensus were added.

Each trigger in the resulting 94-trigger tool was evaluated for risk priority, frequency and detectability. Triggers above a pre-established cut-off or mean risk score were retained. Using a modified Delphi process, the remaining triggers were evaluated by a team of clinical paediatric patient safety experts who eliminated half, resulting in the preliminary 47-trigger CPTT.

Pilot testing of the CPTT

The CPTT was piloted in three acute care paediatric settings in Alberta: a stand-alone paediatric unit in a community hospital, a paediatric hospital-within-a-hospital and a regional paediatric centre. At each site, five medical and five surgical charts were selected for audit from fiscal year 2004 from each of three age groups: 29–365 days, 366 days to 5 years and over 5 years; 10 charts were randomly selected from the fourth age group of 0–28 days.

Two nurses visited the hospitals between June and August 2005, and independently reviewed all 40 charts at each hospital using the draft trigger tool. Each nurse had undergone detailed chart abstraction training as carried out for the Canadian Adverse Event Study,1 and reliability was tested using a set of standard charts. Reviewers were asked to record any difficulties with interpretation or application of trigger definitions with particular emphasis on potential duplication or impracticality. Nurses were also asked to identify any triggers they thought should be considered for exclusion by the Research Team. Their experience was that the tool could be used to assess the charts, and no triggers were identified that could be eliminated. The κ statistic for the measurement of inter-rater reliability (IRR) for triggers in 120 charts reviewed was 0.81, and the validation study was commenced.

Definitions

Based on the definition used in previous national studies of AEs, an AE was defined as an unintended injury or complication that results in disability at the time of discharge, death, prolonged hospital stay or subsequent hospitalisation, and is caused by healthcare management.1 4 6 7 10 11 The AE had to occur within a 3-month period before or during the index admission and be detected during or within the 3-month period before or after the admission. ‘Healthcare management’ included the actions of individual hospital staff, as well as the broader systems and care processes, and included acts of omission and commission. To qualify as an AE, healthcare management must have received a causation rating of at least 4 on a six-point scale, that is, greater than 50% likelihood of being caused by healthcare management (box 1).

Box 1. Six-point scale for physician determination of causation2

Virtually no evidence for management causation.
Slight to modest evidence for management causation.
Management causation not likely; less than 50/50 but close call.
Management causation more likely; more than 50/50 but close call.
Moderate to strong evidence for management causation.
Virtually certain evidence for management causation.

The tool was validated by separate nurse and physician reviews of the same charts of children hospitalised in six academic paediatric health sciences centres (IWK Hospital Halifax; Children's Hospital of Winnipeg; Alberta Children's Hospital Calgary; Stollery Children's Hospital Edmonton; Children's Hospital of Eastern Ontario Ottawa; SickKids Hospital Toronto) during the fiscal year 2005/2006. Validation included the use of the TT on a sample of paediatric patient charts and the assessment of those charts to determine if AEs were present.

Training for nurses and physicians

Two nurses from each of the six participating institutions, and three physicians from the validation study team, with at least 10 years' academic paediatric centre experience, attended a 2-day training session at the University of Toronto. A customised study training manual and standard set of blinded paediatric hospital charts were used. Following training, nurses and physicians each reviewed 20 standardised charts, and the IRR was calculated.

Data collection and IRR calculation

At each hospital, two nurses reviewed a total of 100 selected charts for evidence of triggers. Nurses determined the presence of an AE based on the presence of an injury and harm associated with healthcare management and then assigned a National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) classification of harm for each event (table 1). Blinded to the nurses' results, one of the three physicians reviewed the same 100 charts at two sites and classified all injuries as AEs or not, using the standard definition of AE used in other studies (table 1). Ten per cent of charts were reabstracted, and the IRR for the six nurse teams calculated.

Table 1

Distribution of nurse and physician assessment of adverse events (AEs) as compared with the nurses' classification of harm using the National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) scale

Validation study

Biostatistical consultation determined that 600 charts from six paediatric healthcare centres would enable the reliability and validity of the tool to be established with a high degree of confidence (α 0.05; β 0.10). A standardised chart selection algorithm was applied to the Discharge Abstract Database for all paediatric separations in participating hospitals for fiscal 2006. One hundred charts plus a 20% oversampling (n=120) to accommodate for missing charts, stratified by age group as described above, were selected for review from those identified by the algorithm at each site.

Charts of children receiving all levels of inpatient care were eligible for inclusion. However, patients were excluded if they were 18 years of age and older or their most responsible diagnosis at discharge was obstetrical or psychiatric. For patients with more than one hospitalisation during fiscal year 2006, one stay was randomly selected to be reviewed for the study. No more than 100 eligible charts were reviewed at any single centre.

Analysis

Descriptive statistics were used for the trigger frequency and the number of AEs identified. The validity of the CPTT was judged by its sensitivity, that is, the percentage of trigger-positive charts that physicians judged to have an AE; and specificity, that is, the percentage trigger-negative charts that had no AE according to physician review. The positive predictive value (PPV) of each trigger, and the sensitivity and specificity of the overall tool were also determined.

Ethics

Research Ethics Board approval was obtained at each participating centre as well as the University of Toronto, and provincial legislation regarding privacy and protection of evidence was respected.

Results of validation testing

After training for the validation study, the κ statistic for IRR for nurse-identified triggers and physician-identified AEs was 0.62 and 0.57, respectively, and 0.67 for the IRR for nurses at each participating site. Five hundred and ninety-one of the 600 charts reviewed by nurses were available for physician review; the others were misfiled or on the clinical wards where the patient was currently hospitalised. Of the 591 charts reviewed by both nurses and physicians, 361 (61.1%; 95% CI 57.2 to 65.0) had at least one trigger: 148 (41%) had one, 145 (40.2%) had two, three or four, and 59 (16.3%) had five or more triggers.

Physicians identified 340 injuries in 180 of the 591 charts. In 89 charts, the injuries met the criteria for AEs, for an AE rate of 15.1% (95% CI 12.2% to 17.1%). The distribution of AEs by age and gender is displayed in table 2. Of the 89 patients with an AE, 37 (41.6%) were designated as resulting from acts of commission, 34 (38.2%) from acts of omission and six events (6.7%) from both; and 12 (13.5%) were not categorised.

Table 2

Characteristics and distribution of patients with adverse events (AEs)

Table 3

Sensitivity and specificity of the 47-Trigger Canadian Paediatric Trigger Tool

Nurses' assessment of harm and AEs varied by the classification method used, and also differed from physicians' assessment. MDs found AEs in 34 charts that nurses assigned ‘No Harm’; otherwise, nurses were more likely to identify AEs in charts at all levels of harm severity (table 1). Nurses and physicians agreed on the presence of an AE in 489 (82.7%) of the charts for a κ of 0.34 (95% CI 0.23 to 0.43).

The PPV (table 4) and ratio of false-positive to true positive predictors of an AE for each trigger were calculated. The PPV ranged from 0 to 83.3. Based on these results, and in order to improve the efficiency of the CPTT, 11 triggers were removed: eight with a low predictive value and ratio of >0.7, and three that were never selected. A fourth unselected trigger, ‘wrong site surgery,’ was retained, and the two readmission triggers merged (C03 and C16). These revisions resulted in the final 35-trigger CPTT (table 5).

Table 4

Forty-seven screening criteria of the preliminary Canadian Association of Paediatric Health Centres Trigger Tool (CPTT)^* and positive predictive values for each trigger

Table 5

35-Trigger Canadian Paediatric Trigger Tool

Discussion

Several agencies, including the Agency for Healthcare Research and Quality, have argued that indicators needed to be developed for specific populations, including paediatrics.28 We concluded that a robust TT, validated for paediatric use, was necessary for evaluating AEs and conducting a national paediatric study in Canada. Unlike the IHI-GTT where individual modules were validated separately, the CPTT has been validated for use as a single tool for all paediatric populations.

Similar to the other national studies of AEs, use of our tool calls for a two-stage review process: first, nurses use the trigger tool to identify charts likely to have an AE, followed by physicians who review the triggered charts for AEs. However, in order to establish the sensitivity of the nurse review, all charts in this study were reviewed independently for triggers (by nurses) and for AEs (by physicians and nurses). This process is the first of its kind to establish a false-negative rate for triggered charts, by identifying patients with AEs whose charts were not triggered. The number of false-negative charts may be reduced through education highlighting use of the ‘other’ trigger by the nurses. Nurse and physician review of all charts for AEs also demonstrated differences between these groups in designating AEs. Nurses and physicians assessed the charts using somewhat different methods. Nurses reviewed charts using the CPTT and assessed all charts in terms of the NCC-MERP Harm Scale and the presence of AEs,29 whereas physicians reviewed all charts for AEs only. These different methods mean their results are not strictly comparable. However, the physicians read each case from admission to discharge, and based on these findings, it appears that physicians may be better qualified to assess the presence of AEs. This difference between nurse and physician assessment of AEs is consistent with previous findings,30 31 but these findings should be explored further.

There are inherent limitations in TT methodology. The utility of any chart review is subject to the quality of the documentation.32 Like other screening tests, TTs have a high sensitivity and relatively low specificity. However, their purpose is to cull charts unlikely to have an AE, leaving behind for physician review only those charts with a high probability of having an AE. The IRR of identifying AEs through chart review is variable; previous studies reported κ scores ranging from −0.077 to 0.66.1–9 16 19 32–35 Combining ratings from more than one physician19 36 and monitoring reviewer performance during the study with personal feedback may improve IRR.30 However, a recent report suggests that a review process involving two physicians per chart is no more reliable than only one in detecting AEs.36 Additional training that includes a consensus approach on the number of AEs in the training charts has been used successfully.35 Finally, previous national studies of AEs have focused on those characterised by significant harm, that is NCC-MERP categories F–I, underestimating the total burden of harm by disregarding events with temporary harm requiring intervention only (category E events). Although achieving high IRR for category E can be challenging,35 capturing these events in future studies will result in a more comprehensive view of the nature of iatrogenic injury, and permit comparisons with recent studies of AEs in children and youth that have used the expanded definition.17 18

Despite these limitations, the use of a broad paediatric population sample to validate the CPTT permits its use in other acute care paediatric institutions. With its final refinements, the validated CPTT offers the first comprehensive evidence-based tool for assessing harm in hospitalised children and youth.

Conclusion

The CPTT is the first valid and reliable TT for detecting harm in children and youth of all ages hospitalised in acute care. This 35-trigger tool is reliable and robust, and can be used in quality-improvement initiatives and for more rigorous research agendas.

Future research should focus on improving the efficiency of the CPTT and investigating the differences between nurse and physician assessments of AEs. The CPTT is likely to be of interest to researchers wishing to comprehensively study the epidemiology and burden of iatrogenic harm in hospitalised children and youth, using a methodology similar to the previous national studies of AEs in hospitalised adults. Such studies will enhance current efforts to raise the profile of paediatric patient safety issues.

Acknowledgments

We would like to thank the Alberta Children's Hospital, Calgary Alberta; Canadian Association of Paediatric Health Centres (CAPHC); Children's Hospital of Eastern Ontario, Ottawa, Ontario; Hospital for Sick Children Toronto, Ontario, IWK Health Centre, Halifax, Nova Scotia; Stollery Children's Hospital, Edmonton, Alberta; and the Child Health Quality Team, Winnipeg Regional Health Authority, Winnipeg, Manitoba for their support.

Footnotes

Funding: This study was supported by the Canadian Medical Protection Association; Canada's Research Based Pharmaceutical Companies (Rx&D); Department of Paediatrics University of Calgary; Health Canada; Manitoba Institute of Patient Safety (MIPS); Winnipeg Regional Health Authority (WRHA); and Spelman Cronin Consulting (SCC).

Competing interests: None.

Ethics approval: Ethics approval was provided by the Hospital for Sick Children, Stollery Children's Hospital, Calgary Children's Hospital, Children's Hospital of Eastern Ontario, Winnipeg Children's Hospital, and IWK Hospital.

Contributors: All authors contributed substantially to the study and write-up of the manuscript.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678–86. [PMC free article] [PubMed]

2. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–6. [PubMed]

3. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–84. [PubMed]

4. Wilson RM, Runciman WB, Gibberd RW, et al. The Quality in Australian Health Care Study. Med J Aust 1995;163:458–71. [PubMed]

5. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261–71. [PubMed]

6. Davis P, Lay-Yee R, Briant R, et al. Adverse Events in New Zealand Public Hospitals: Principal Findings from a National Survey. Wellington, NZ: Ministry of Health, 2001, Occasional Paper No. 3.

7. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001;322:517–19. [PMC free article] [PubMed]

8. Davis P, Lay-Yee R, Briant R, et al. Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J 2002;115:U271. [PubMed]

9. Davis P, Lay-Yee R, Briant R, et al. Adverse events in New Zealand public hospitals II: preventability and clinical context. N Z Med J 2003;116:U624. [PubMed]

10. Zegers M, de Bruijne MC, Wagner C, et al. Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Qual Saf Health Care 2009;18:297–302. [PubMed]

11. Soop M, Fryksmark U, Köster M, et al. The incidence of adverse events in Swedish hospitals: a retrospective medical record review study. Int J Qual Health Care 2009;21:285–91. [PMC free article] [PubMed]

12. Kobayashi M, Ikeda S, Kitazawa N, et al. Validity of retrospective review of medical records as a means of identifying adverse events: comparison between medical records and accident reports. J Eval Clin Pract 2008;14:126–30. [PubMed]

13. Miller MR, Elixhauser A, Zhan C. Patient safety events during paediatric hospitalizations. Paediatrics 2003;111:1358–66. [PubMed]

14. Slonim AD, LaFleur BJ, Ahmed W, et al. Hospital-reported medical errors in children. Paediatrics 2003;111:617–21. [PubMed]

15. Woods D, Thomas E, Holl J, et al. Adverse events and preventable adverse events in children. Paediatrics 2005;115:155–60. [PubMed]

16. Larsen GY, Donaldson AE, Parker HB, et al. Preventable harm occurring to critically ill children. Pediatr Crit Care Med 2007;8:331–6. [PubMed]

17. Sharek PJ, Horbar JD, Mason W, et al. Adverse events in the neonatal intensive care unit: development, testing, and findings of an NICU-focused trigger tool to identify harm in North American NICUs. Paediatrics 2006;118:1332–40. [PubMed]

18. Takata GS, Mason W, Taketomo C, et al. Development, testing, and findings of a paediatric-focused trigger tool to identify medication-related harm in US children's hospitals. Paediatrics 2008;121:e927–35. [PubMed]

19. Classen DC, Pestotnik SL, Evans RS, et al. Computerized surveillance of adverse drug events in hospital patients. JAMA 1991;266:2847–51. [PubMed]

20. Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events; development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5:305–14. [PMC free article] [PubMed]

21. Olsen S, Neale G, Schwab K, et al. Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care 2007;16:40–4. [PMC free article] [PubMed]

22. Wetzels R, Wolters R, van Weel C, et al. Mix of methods is needed to identify adverse events in general practice: a prospective observational study. BMC Fam Pract 2008;9:35. [PMC free article] [PubMed]

23. Naessens JM, Campbell CR, Huddleston JM, et al. A comparison of hospital adverse events identified by three widely used detection methods. Int J Qual Health Care 2009;21:301–7. [PubMed]

24. Ferranti J, Horvath MM, Cozart H, et al. Reevaluating the safety profile of paediatrics: a comparison of computerized adverse drug event surveillance and voluntary reporting in the paediatric environment. Paediatrics 2008;121:e1201–7. [PubMed]

25. Resar RK, Rozich RD, Classen DC. Methodology and rationale for measurement of harm with trigger tools. Qual Safe Health Care 2003;12(Suppl II):ii39–45. [PMC free article] [PubMed]

26. Griffin FA, Resar RK. IHI Global Trigger Tool for Measuring Adverse Events. IHI Innovation Series White Paper. Cambridge, MA: Institute for Healthcare Improvement, 2009. http://www.IHI.org (accessed 31 Jan 2010).

27. Matlow A, Flintoft V, Orrbine E, et al. The development of the Canadian paediatric trigger tool for identifying potential adverse events. Healthc Q 2005;(8 Spec No):90–3. [PubMed]

28. Pediatric Quality Indicators Overview. http://www.qualityindicators.ahrq.gov/pdi_overview.htm (accessed 31 Jan 2010).

29. NCC MERP Taxonomy of medication errors. http://www.nccmerp.org/pdf/taxo2001-07-31.pdf (accessed 31 Jan 2010).

30. Rowin EJ, Lucier D, Pauker SG, et al. Does error and adverse event reporting by physicians and nurses differ? Jt Comm J Qual Patient Saf 2008;34:537–45. [PubMed]

31. Zegers M, de Bruijne MC, Wagner W, et al. Design of a retrospective patient record study on the occurrence of adverse events among patients in Dutch hospitals. BMC Health Services Res 2007;7:27. [PMC free article] [PubMed]

32. Thomas EJ, Studdert DM, Brennan TA. The reliability of medical record review for estimating adverse event rates. Ann Int Med 2002;136:812–16. [PubMed]

33. Forster AJ, O'Rourke K, Shojania KG, et al. Combining ratings from multiple physician reviewers helped to overcome the uncertainty associated with adverse event classification. J Clin Epidemiology 2007;60:892–901. [PubMed]

34. Kunac DL, Reith DM, Kennedy J, et al. Inter- and intra-rater reliability for classification of medication related events in paediatric inpatients. Qual Safe Health Care 2006;15:196–201. [PMC free article] [PubMed]

35. Classen DC, Lloyd RC, Provost L, et al. Development and evaluation of the Institute for Healthcare Improvement global trigger tool. J Patient Saf 2008;4:169–77.

36. Zegers M, de Bruijne MC, Wagner C, et al. The inter-rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record. J Clin Epidemiology 2010;63:94–102. [PubMed]

Articles from BMJ Open Access are provided here courtesy of
BMJ Group