PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Intensive Care Med. Author manuscript; available in PMC 2010 June 1.
Published in final edited form as:
PMCID: PMC2730460
NIHMSID: NIHMS106930

Validation of an electronic surveillance system for acute lung injury

Vitaly Herasevich, M.D., Ph.D.,1 Murat Yilmaz, M.D.,1,3 Hasrat Khan, M.D.,1,2 Rolf D. Hubmayr, M.D.,1 and Ognjen Gajic, M.D.1

Abstract

Objective

Early detection of acute lung injury (ALI) is essential for timely implementation of evidence-based therapies and enrollment into clinical trials. We aimed to determine the accuracy of computerized syndrome surveillance for detection of ALI in hospitalized patients and compare it with routine clinical assessment.

Design

Using a near-real time copy of the electronic medical records we developed and validated a custom ALI electronic alert (ALI “sniffer”) based on the European-American Consensus Conference Definition and compared its performance against provider derived documentation.

Patients and setting

Consecutive 3795 critically ill patients admitted to 9 multidisciplinary intensive care units (ICUs) of a tertiary care teaching institution.

Measurements and main results

ALI developed in 325 patients and was recognized by bedside clinicians in only 86 (26.5%). Under-recognition of ALI was associated with not implementing protective mechanical ventilation (median tidal volumes of 9.2 vs 8.0 mL/kg predicted body weight, p<0.001). ALI “sniffer” demonstrated excellent sensitivity, 96% (95% CI 94 to 98) and moderate specificity, 89% (95% CI 88 to 90) with a positive predictive value ranging from 24% (95% CI 13 to 40) in the heart-lung transplant ICU to 64% (95% CI 55 to 71) in the medical ICU.

Conclusions

Computerized surveillance system accurately identifies critically ill patients who develop ALI syndrome. Since the lack of ALI recognition is a barrier to the timely implementation of best practices and enrollment into research studies, computerized syndrome surveillance could be a useful tool to enhance patient safety and clinical research.

Keywords: Respiratory Distress Syndrome, Adult, Syndrome surveillance, Database, Datamart, Diagnosis

Introduction

Acute lung injury (ALI) and its more severe form acute respiratory distress syndrome (ARDS) represent a major health problem with an estimated prevalence of 7% of intensive care unit (ICU) admissions [1, 2] for which the appropriate treatment is commonly instituted too late or not at all [3,4]. Timely recognition of ALI syndrome is an important determinant of outcome of critical illness and is essential for appropriate enrollment into research studies. However, recognizing specific patterns of ICU syndromes is difficult [3]. This is particularly true in elderly patients with multiple comorbidities in whom the incidence of ALI is particularly high [2].

Under-recognition of ALI is an important barrier to a timely institution of specific treatment (i.e. low tidal volume mechanical ventilation) [4, 5]. We have recently reported that in our medical ICU 70% of patients with ALI in whom appropriate treatment was not instituted were in fact not recognized or documented to have ALI by bedside providers [6]. Standard ICU monitoring and alarm systems are grossly inadequate, have a poor specificity and lack the ability to recognize complex physiologic syndromes [7, 8], because this would require integration of monitored variables such as vital signs with specific laboratory and radiological findings. Computerized syndrome surveillance has been used in public health practice and emergency departments to detect adverse events for the last decade, but have only recently been applied in the ICU setting. A previous study suggested that electronic screening may be a useful mechanism for enrolling patients with ALI into a prospective research study [9]. However, the tool, which relied entirely on laboratory criteria (not taking into consideration radiology reports) had a very poor specificity [9]. Electronic review of narrative radiology reports has been comparable to a physician reviews in a prior study [10] and may improve the performance of the electronic surveillance tool. During a pilot study in the medical ICU, we have recently demonstrated the feasibility of electronic ALI screening using a tool which combines a near real time query of the chest x-ray reading with arterial blood gas values (“ALI sniffer”) [11]. The objective of the present work was to determine the accuracy of the ALI sniffer in detecting ALI throughout nine ICUs in two hospitals of a tertiary care institution.

Methods

The study was conducted at the Mayo Clinic in Rochester, Minnesota, a tertiary care teaching institution with 2 hospitals comprising of 1900 inpatient beds and 201 ICU beds. We evaluated a ALI sniffer (see below) during 4 months (July – November 2006) in a prospective cohort of consecutive patients admitted to 9 ICUs including Cardiac Surgery, Medical Cardiac ICU (CCU), Medical, Mixed, Neurology, Pediatric and Neonatal, Thoracic and Vascular, Heart-Lung Transplant and Trauma/General Surgery. The characteristics of ICUs have been previously described [12]. Readmissions, patients younger than 6 months of age and those who denied research authorization were excluded. The Institutional Review Board approved the study protocol and waived the need for informed consent.

Description of the ALI Sniffer

The Mayo Epidemiology and Translational Research in Intensive Care (METRIC) database (METRIC Datamart) was used in this project. This SQL-based integrative database accumulated data within one hour from its entry into the electronic medical records and served as the main data source for the rules of development providing the linked demographic, monitoring, laboratory, intervention, and outcome data required for this study [13]. Data access was accomplished through open database connectivity (ODBC) connections to a relational database using Microsoft SQL Server. ALI sniffer was triggered when both criteria listed below were met within a single 24-hour period:

  1. Qualifying arterial blood gas analysis: the ratio of partial pressure of oxygen over inspired oxygen concentration (PaO2/FIO2) <300 and
  2. Qualifying chest radiograph report: free text Boolean query containing trigger words: (“bilateral” AND “infiltrate”) OR “edema”.

In our institution, the radiologist is available 24 hours a day and the reports of portable chest radiographs of ICU patients are available in the electronic form average two hours after the chest x-ray is obtained.

Patients were monitored by the system during the entire ICU stay. The log file with the output of ALI sniffer was fed back to the SQL server. Daily e-mail messages alerted study investigators about potential ALI cases. Two intensivist researchers (MY and HK) blinded to clinicians' diagnoses reviewed the electronic medical records within 48 hours of the alert) and assigned the diagnosis of ALI according to the American-European Consensus Conference criteria (AECC) [14]. Patients who were not triggered by ALI sniffer were reviewed after the ICU discharge. Patients who did not have a chest x-ray or arterial blood gas analysis, or in whom ratio of the partial pressure of arterial oxygen to the fraction of inspired oxygen (PaO2/FiO2) was >300 during the entire ICU stay were determined not to be ALI without further review.

MY and HK reviewed a structured ALI tutorial [15] prior to study onset and independently reviewed digital portable chest x-ray images. The presence or absence of left atrial hypertension as the principal explanation for pulmonary edema was determined by integrated clinical evaluation based on the combination of echocardiographic findings, brain natriuretic peptide levels, venous filling pressures and the response to therapy (response to diuretics and positive pressure ventilation favors hydrostatic edema). This process yielded moderate inter-observer reliability (kappa of 0.6-0.7) in previous studies [17-19].

Clinical recognition of ALI by bedside providers was determined by the use of the terms “Acute Lung Injury”, “Acute Respiratory Distress Syndrome”, “ALI” or “ARDS” in the clinical notes at any time during ICU stay as well as discharge summary.

To determine if the recognition of ALI by the bedside providers influences the adherence to protective mechanical ventilation we compared the average tidal volume and the percentage of patients ventilated with potentially injurious settings between patients whose providers did or did not document the presence of ALI syndrome. Serial tidal volume readings were averaged for each patient from the time of ALI onset to the beginning of weaning (spontaneous breathing trial or pressure support ventilation mode) and summarized as median. Tidal volume of >8 mL/kg predicted body weight (PBW) was considered potentially injurious [16].

Sensitivity and specificity of the ALI sniffer and routine clinical assessment were determined against the gold standard consensus conference definition applied by the MY and HK (see above). Continuous variables were summarized as median and interquartile ranges (IQR). Exact binomial confidence intervals were calculated for predictive values and sensitivity and specificity. Mc'Nemar's test was used to compare sensitivity between clinical assessment of ALI and ALI sniffer results. JMP statistical software (version 6.0, SAS, Cary, NC) was used for all data analyses.

Results

Among 3795 patients who met inclusion criteria (Figure 1), expert reviewers identified 325 cases of ALI (incidence 8.6%). During the same period the ALI sniffer sent 680 alerts. It demonstrated sensitivity, 96% (95% CI 94 to 98) and specificity, 89% (95% CI 88 to 90).

Fig. 1
Outline of study. Flow diagram of patients, ALI sniffer, and Routine Clinical Assessment. * - AECC - American-European Consensus Conference criteria

The most important reasons of false positive alerts were cardiogenic pulmonary edema (n=152, 41%), atelectases (n=90, 25%), soft tissue shadows (n=15, 4%) and reduced lung volumes (n=15, 4%).

ALI sniffer missed 12 patients with ALI: In the six patients FIO2 was missing on the arterial blood gas (inspired oxygen concentration was presented as O2 flow, liters per minute instead of percentages) since the patients did not receive mechanical ventilation. In four cases the radiologist used the term “consolidation” or “opacities” instead of “infiltrate” or “edema” and “both lungs” instead “bilateral”. Another two cases occurred during the brief period of ALI sniffer outage.

Table 2 provides the performance characteristics of ALI sniffer and a routine clinical assessment. Among 3795 patients who met inclusion criteria, clinicians identified 86 cases of ALI, missed 239 and over diagnosed 17. The sensitivity of ALI detection by routine clinical assessment was 26.5 % (95% CI 21.8 - 31.6) and specificity was 99.5 % (95% CI 99.2 - 99.7). The majority of ALI cases were not recognized or documented as such by bedside clinicians.

Table 2
Performance Characteristic of ALI sniffer and the routine clinical assessment

Under-recognition of ALI was associated with not implementing protective mechanical ventilation. The majority (70%) of patients who were not recognized as ALI by the providers were exposed to potentially injurious mechanical ventilation (Vt >8 mL/kg PBW) compared to 48% of those who were recognized (p=0.001). Tidal volumes over the course of mechanical ventilation were higher in patients in whom ALI was not recognized (median 9.2 vs 8.0 mL/kg PBW, p<0.001).

Table 3 outlines the performance of ALI sniffer in different ICUs. Positive predictive value (PPV) of the ALI sniffer ranged from 24% (95% CI 13% - 40%) in the heart-lung transplant ICU to 64% (95% CI 55% - 71%) in the medical ICU. Negative predictive value (NPV) was ranging from 98% (95% CI 97% - 99%) in Mixed medical-surgical ICU to 100% in Cardiac Surgery, Neuro, Pediatrics, Thoracic/Vascular surgery and Heart/Lung Transplant ICUs

Table 3
The performance of ALI sniffer and routine clinical assessment in different ICUs

Discussion

The main findings of our study are the following: 1) Electronic ALI sniffer based on arterial blood gas values and free text radiologist interpretation of portable chest x-rays was feasible and had high sensitivity and moderate specificity in detecting ALI. 2) Under-recognition is associated with inappropriate therapy: higher tidal volume ventilation. 3) The presence of ALI syndrome is commonly unrecognized by the clinicians.

Our findings support previous reports that ARDS appears under-recognized by clinicians even in patients who die with this syndrome [3]. ICU physicians are exposed to information overload from different sources including clinical notes and reports, flowcharts, bedside monitors, and laboratory results. Over 50 years ago, Miller had shown that even experienced clinicians could not consistently integrate more then seven variables for information processing [17]. More recent observation even consider smaller capacity limit from three to five information chunks [18].

Previous reports suggest that under-recognition of ALI is an important barrier to a timely institution of specific treatment (i.e. low tidal volume mechanical ventilation) [4]. We have previously reported that in our medical ICU, 70% of patients with ALI in whom appropriate treatment was not instituted were in fact not recognized to have ALI [19].

Widespread introduction of electronic medical records may allow for the development of electronic surveillance tools (sniffers) that would enhance physicians' and researchers' ability to recognize important critical care syndromes in a timely manner. In this study, we have demonstrated feasibility of a near real time ALI sniffer based on arterial blood gas values and free text radiologist interpretation of portable chest x-rays. Automated surveillance for adverse events in hospitalized patients was first demonstrated by Evans et al. [20] and proved to be superior in detection of adverse drug events (at a rate 3.6 times that of voluntary reporting at the university hospital and 12.3 times that at the community hospital [21]). Bedside monitoring is a significant part of critical care workflow, but most current monitoring approaches are of dubious clinical value. Schoenberg et al. found that most of the alarms generated by bedside monitors are clinically insignificant and only detract the bedside providers [22]. The study by Tsien and Fackler has shown that 86% of alarms triggered in an ICU are false, another 6% true but clinically irrelevant and only 8% are of true clinically importance [23]. Compared to routine ICU alarms, limited to abnormal values of single physiologic parameters, “smart alarms” that combine multiple data sources such as ALI sniffer should have a much higher specificity in detecting clinically relevant conditions.

Our findings suggest that the ALI sniffer can become a useful tool for enrollment of patients in clinical studies, as well as decision support at the bedside. A study by Finlay et al. [9], using a laboratory based computer screening system for detection of transfusion-related acute lung injury (TRALI), provides additional support for the value of computerized ALI detection systems.

The specificity and positive predictive value of ALI sniffer were mainly limited by radiologic mimickers of ALI (cardiogenic pulmonary edema and atelectasis) and varied according to different ICUs, being the lowest in cardiac transplant, cardiac surgery and coronary care unit (Table 3). Natural language processing that can detect more complex word structures, synonyms and semantic relatedness, and can make context sensitive spelling corrections, could improve the performance characteristics of electronic alerts for ALI. Also one of the possible ways to improve detection of ALI is to have structured radiology report with clear consideration about ALI. In addition, computerized screening systems that incorporate echocardiography reports and data from hemodynamic monitoring can further enhance the performance of electronic alerts for ALI.

Our study was performed in a single tertiary center potentially limiting the generalizability of results. We assumed that portable chest x-ray is done quickly in all patients with acute respiratory failure secondary to ALI. An important limitation of automatic screening tools that rely on radiologist interpretation are the timing of x-ray reading and inherent subjectivity in the interpretation. Radiologists in our institution interpreted portable chest x-rays in a timely manner with remarkable consistency in the terminology to describe pulmonary edema (almost uniformly using the words “edema” or “bilateral” + “infiltrates”).

Expert assessment of ALI is additional limitation of our approach. Although inter-rater reliability was not measured in this study (cases were not reviewed in duplicate), we have successfully used this approach in several previous studies on ALI epidemiology with good interobserver variability between the experts.

Finally, while this study suggests that an electronic alert may satisfactorily detect patients with ALI, there is no evidence that alerting clinicians that their patient may have ALI means the clinicians will treat them as such or change their behavior.

In conclusion, we have shown that the automatic ALI sniffer accurately identifies patients who develop ALI in critically ill patients. From a research perspective, such a tool can be used for timely identification of patients with ALI eligible for enrollment into clinical trials. It remains to be seen if the implementation of this tool at the bedside will improve the adherence to evidence-based treatments and overall outcome of patients with ALI. Success of such a tool would prompt the development of sophisticated electronic screening tools to allow for timely identification of other major critical care syndromes such as shock, severe sepsis or acute renal failure.

Table 1
Baseline characteristic of the study patients (n=3795). For continuous variables, mean ± standard deviations are reported, while for categorical variables counts with percentages in parenthesis are reported.

Acknowledgments

Funding: This publication was made possible by Grant Number 1 KL2 RR024151 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), the NIH Roadmap for Medical Research and the Mayo Foundation. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. Information on NCRR is available at http://www.ncrr.nih.gov/. Information on Reengineering the Clinical Research Enterprise can be obtained from http://nihroadmap.nih.gov/clinicalresearch/overviewtranslational.asp. This study was supported in part by NHLBI K23 HL78743-01A1 and NIH KL2 RR024151

References

1. Brun-Buisson C, Minelli C, Bertolini G, Brazzi L, Pimentel J, Lewandowski K, Bion J, Romand JA, Villar J, Thorsteinsson A, Damas P, Armaganidis A, Lemaire F. Epidemiology and outcome of acute lung injury in European intensive care units. Results from the ALIVE study. Intensive Care Med. 2004;30:51–61. [PubMed]
2. Rubenfeld GD, Caldwell E, Peabody E, Weaver J, Martin DP, Neff M, Stern EJ, Hudson LD. Incidence and Outcomes of Acute Lung Injury. N Engl J Med. 2005;353:1685–1693. [PubMed]
3. Ferguson ND, Frutos-Vivar F, Esteban A, Fernandez-Segoviano P, Aramburu JA, Najera L, Stewart TE. Acute respiratory distress syndrome: underrecognition by clinicians and diagnostic accuracy of three clinical definitions. Crit Care Med. 2005;33:2228–2234. [PubMed]
4. Rubenfeld GD, Cooper C, Carter G, Thompson BT, Hudson LD. Barriers to providing lung-protective ventilation to patients with acute lung injury. Crit Care Med. 2004;32:1289–1293. [PubMed]
5. Umoh NJ, Fan E, Mendez-Tellez PA, Sevransky JE, Dennison CR, Shanholtz C, Pronovost PJ, Needham DM. Patient and intensive care unit organizational factors associated with low tidal volume ventilation in acute lung injury. Crit Care Med. 2008;36:1463–1468. [PubMed]
6. Mullon J, Gajic O, Afessa B. Compliance with Evidence-Proven Mechanical Ventilation Strategy in Acute Lung Injury. Proc Am Thorac Soc. 2006:A570.
7. Chambrin MC. Alarms in the intensive care unit: how can the number of false alarms be reduced? Crit Care. 2001;5:184–188. [PMC free article] [PubMed]
8. Chambrin MC, Ravaux P, Calvelo-Aros D, Jaborska A, Chopin C, Boniface B. Multicentric study of monitoring alarms in the adult intensive care unit (ICU): a descriptive analysis. Intensive Care Med. 1999;25:1360–1366. [PubMed]
9. Finlay HE, Cassorla L, Feiner J, Toy P. Designing and testing a computer-based screening system for transfusion-related acute lung injury. Am J Clin Pathol. 2005;124:1–9. [PubMed]
10. Hripcsak G, Friedman C, Alderson PO, DuMouchel W, Johnson SB, Clayton PD. Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med. 1995;122:681–688. [PubMed]
11. Herasevich V, Yilmaz M, Khan H, Chute CG, Gajic O. Rule base system for identification of patients with specific critical care syndromes: The “sniffer” for acute lung injury. Proc AMIA Symp. 2007:972. [PubMed]
12. Afessa B, Keegan MT, Hubmayr RD, Naessens JM, Gajic O, Long KH, Peters SG. Evaluating the performance of an institution using an intensive care unit benchmark. Mayo Clin Proc. 2005;80:174–180. [PubMed]
13. Herasevich V, Keegan MT, Tines D, Malinchoc MM, Hanson AC, Katyal P, Hubmayr RD, Afessa B, Gajic O. “ICU of the future”: informatics infrastructure for syndrome surveillance, decision support, data mining, and modeling of critical illness. Journal of Critical Care. 2007;22:340.
14. Bernard GR, Artigas A, Brigham KL, Carlet J, Falke K, Hudson L, Lamy M, Legall JR, Morris A, Spragg R. The American-European Consensus Conference on ARDS. Definitions, mechanisms, relevant outcomes, and clinical trial coordination. Am J Respir Crit Care Med. 1994;149:818–824. [PubMed]
15. Improving the Radiographic Diagnosis of Acute Lung Injury. Tutorial. (n.d.) [November 11, 2008]. from http://depts.washington.edu/kclip/about.shtml.
16. Moran JL, Bersten AD, Solomon PJ. Meta-analysis of controlled trials of ventilator therapy in acute lung injury and acute respiratory distress syndrome: an alternative perspective. Intensive Care Med. 2005;31:227–235. [PubMed]
17. Miller GA. The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol Rev. 1956;63:81–97. [PubMed]
18. Cowan N. The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci. 2001;24:87–114. discussion 114-185. [PubMed]
19. Wolthuis EK, Korevaar JC, Spronk P, Kuiper MA, Dzoljic M, Vroom MB, Schultz MJ. Feedback and education improve physician compliance in use of lung-protective mechanical ventilation. Intensive Care Med. 2005;31:540–546. [PubMed]
20. Evans RS, Pestotnik SL, Classen DC, Bass SB, Menlove RL, Gardner RM, Burke JP. Development of a computerized adverse drug event monitor. Proc Annu Symp Comput Appl Med Care. 1991:23–27. [PMC free article] [PubMed]
21. Kilbridge PM, Campbell UC, Cozart HB, Mojarrad MG. Automated surveillance for adverse drug events at a community hospital and an academic medical center. J Am Med Inform Assoc. 2006;13:372–377. [PMC free article] [PubMed]
22. Schoenberg R, Sands DZ, Safran C. Making ICU alarms meaningful: a comparison of traditional vs. trend-based algorithms. Proc AMIA Symp. 1999:379–383. [PMC free article] [PubMed]
23. Tsien CL, Fackler JC. Poor prognosis for existing monitors in the intensive care unit. Crit Care Med. 1997;25:614–619. [PubMed]