|Home | About | Journals | Submit | Contact Us | Français|
Validate the American Thoracic Society-Infectious Disease Society of America 2007 (IDSA/ATS 2007) criteria for predicting severe community-acquired pneumonia (SCAP) and evaluate a health-services definition for SCAP.
Retrospective cohort study.
LDS Hospital, an academic tertiary care facility in the Western United States
Consecutive patients with ICD-9 codes and chest radiographs consistent with community-acquired pneumonia from 1996-2006 seen at LDS Hospital
We utilized the electronic medical record to examine intensive care unit admission, intensive therapies received, and predictors of severity, as well as 30-day mortality. We also developed logistic regression models of mortality and disease severity. We calculated the IDSA/ATS 2007 criteria as well as three other pneumonia severity scores. We defined SCAP as receipt of intensive therapy in the intensive care unit. In 2,413 episodes of pneumonia, 1,540 were admitted to the hospital, while 379 were admitted to the intensive care unit. Overall 30-day mortality was 3.7% but was 16% among ICU patients. The IDSA/ATS 2007 minor criteria predicted SCAP with a c-statistic of 0.88 (0.85-0.90), which improved to 0.90 (0.88 - 0.92) with weighting. Competing models had c-statistics of 0.76-0.83. Using four rather than three minor criteria improved the positive predictive value from 54% to 81%, with a stable negative predictive value of 94-92%.
The IDSA/ATS 2007 criteria predicted pneumonia severity better than other models. Using four rather than three minor criteria may be a superior cutoff, though this will depend on institutional characteristics.
Community-acquired pneumonia (CAP) is an important public health problem. When combined with influenza, it is the sixth-leading cause of death in the United States.(1, 2) Approximately 500,000 adults are admitted to the hospital in the US annually for CAP.(3) Since site of care is the major determinant of cost and appropriate site of care presumably improves outcome, triage of patients with CAP is important.(4, 5) One persistent problem in studies of CAP is the difficulty in defining and predicting pneumonia severity, although by any measure severe CAP (SCAP) is a significant clinical and public health problem.(6) ICU admission, the most frequent definition(7, 8), varies considerably based on local practice patterns.(9) SCAP has a higher mortality rate than non-severe CAP(10) and has a distinct microbial etiologic predominance(4, 11). However, defining SCAP merely as pneumonia that leads to death is unsatisfactory. Such a definition fails to account for comorbidities, does not reflect the medical interventions required in treatment, and is applied in retrospect. Angus and coauthors evaluated hospital costs, late convalescence, hospital and ICU length of stay as alternative outcomes of SCAP. They compared these outcomes based on four different definitions of severity—ICU admission, receipt of mechanical ventilation, development of medical complications, and mortality.(9) Leroy et al evaluated mechanical ventilation, shock, or medical complications to define SCAP,(12) while Buising et al proposed mortality, ICU admission, mechanical ventilation or inotrope/vasopressor therapy.(13) No study has assessed the full range of intensive therapies provided to patients with SCAP. Given constraints on healthcare resources and the need to define appropriate care, understanding and predicting the therapies required for patients with CAP is important. A health-services definition of SCAP has a logical basis, particularly with regard to triage and therapy.
In addition to problems with determining the best definition, SCAP has proved difficult to predict. Several groups have proposed models and prediction rules to improve triage of patients with CAP generally. These models, primarily the Pneumonia Severity Index (PSI)(14) and the British Thoracic Society simplified prediction model (CURB or CURB-65)(15, 16), have demonstrated utility in recommending outpatient therapy for low-risk patients.(7, 8, 17-19) These models do not perform well at predicting which patients will require ICU admission.(8, 17, 19) A high percentage of ICU admissions are in patients with low-risk scores.(9, 20)
Models specific to SCAP have been developed, including a recent Australian model called SMART-COP(21), and a Spanish model called CURXO-80.(22) The American Thoracic Society (ATS) has also proposed several models, beginning in 1993. Ewig et al evaluated the 1993 ATS predictors(23) and found a low positive predictive value for ICU admission, resulting in revised predictors in the next guidelines iteration.(7) Three studies assessed the 2001 ATS predictors(24) of SCAP,(8, 9, 20) although the positive predictive value continued to be limited and was artificially inflated by use of major criteria—preadmission mechanical ventilation or vasopressor therapy—as predictors of ICU admission.(25) The current guidelines, issued in collaboration with the Infectious Disease Society of America in 2007 (hereafter IDSA/ATS 2007), include new predictors that have not yet been validated.(4) The accuracy of IDSA/ATS 2007 is uncertain, as well as its performance relative to other models.
This paper seeks to address two central questions related to SCAP. First, do certain reference definitions of SCAP better predict thirty-day mortality and hospital length-of-stay than simply ICU admission? Second, do the IDSA/ATS 2007 criteria predict the need for intensive therapies and ICU admission? Preliminary findings from this study have been reported in abstract form.(26)
The study was performed at LDS Hospital, a tertiary care, university-affiliated teaching hospital, with 520 total and 68 ICU beds in Salt Lake City, UT, USA. There are approximately 26,500 Emergency Department (ED) visits per year, resulting in approximately 7,000 hospital admissions. Beginning in 1995, the LDS Hospital ED initiated a standardized pneumonia therapy protocol and deployed an electronic medical record (EMR), which in conjunction with the hospital’s primary EMR prospectively recorded a wide array of clinical, therapeutic, and biometric data.(27, 28) During the study period, hospital policy required ICU admission for any patient requiring ≥60% inspired oxygen.
In a retrospective analysis of prospectively acquired electronic data using a previously validated algorithm(29), we identified all patients seen in the LDS Hospital ED or directly admitted from a clinic, outside ED, or outside hospital within 72-hours of initial admission with International Classification of Diseases (ICD-9) codes compatible with a primary diagnosis of pneumonia (480-487.x) or respiratory failure or organism-specific sepsis (518.x, 038.x) with a secondary diagnosis of pneumonia (480-487.x). We did not include the primary diagnosis of aspiration pneumonia (ICD-9 507.0), as this generally represents healthcare-associated pneumonia (HCAP) in patients with severe comorbidities and overlaps with aspiration pneumonitis. An admission chest radiograph compatible with pneumonia was required, as extracted manually by all three physician authors from radiologist-dictated reports. (A 3-way kappa-value was determined for a random sample of 100 reports and was 0.95.) We excluded patients residing in a skilled nursing facility, those discharged from a hospital within 90 days, and those receiving chronic hemodialysis as representing HCAP.(4) We also excluded patients with significant immune suppression on the basis of receipt of anti-retrovirals or ICD-9 codes indicating Acquired Immune Deficiency Syndrome (042), solid organ transplantation (V420-7), or present or past hematologic malignancies (V106x; 200.x-208.x). In addition we excluded patients with do-not-resuscitate/do-not-intubate (DNR/DNI) orders at admission(30), as well as patients who did not survive their ED stay. In order to avoid patients with recurrent pneumonia, we included only the first episode of pneumonia in a given 12-month period.
Using the electronic clinical database, we extracted all data elements in IDSA/ATS 2007 consensus guidelines severity criteria (Table 1). These include clinical evidence of respiratory failure, clinical or laboratory evidence of organ dysfunction, and radiographic evidence of multilobar pneumonia. Confusion was routinely recorded by clinical ED nurses in the EMR and reflected disorientation to person, place, or time. We also extracted the available secondary predictors suggested in IDSA/ATS 2007 and all data relevant to calculation of SMART-COP (age, shock, multilobar infiltrates, tachypnea, hypalbuminemia, tachycardia, hypoxia and acidemia) and CURXO-80 (shock, age, acidemia, tachycardia, hypoxia, multilobar infiltrates, confusion, uremia) scores. We also calculated SAPS2(31) scores and predicted mortality based on data available in the emergency department, the point of ICU triage for most patients. For purposes of calculating PaO2/FiO2 (P/F) ratios in patients without arterial blood gas results, we used the SaO2/FiO2 (S/F) conversion equation proposed by Rice, et al.(32) In patients not on a measured FiO2, we estimated FiO2 by the equation liter flow/min multiplied by 0.03 plus 0.21. Because our hospital is located at 1400m altitude, we corrected PaO2, P/F, and S/F cutoffs for usual atmospheric pressure at the hospital by dividing by the ratio (0.85) of local (645 mmHg) to sea-level barometric pressure.
ICU admission and length of stay (LOS) were determined from the EMR. Thirty-day all cause mortality, the primary mortality endpoint based on consensus that it best reflects CAP-associated mortality(14, 33, 34), was determined from the merger of the EMR with vital status information from the Utah Population Database.(35) To determine receipt of intensive therapies, we used clinical respiratory therapy, pharmacy, and nursing documentation, laboratory results, and hospital charge codes, all of which were electronic for the duration of the study. We defined as critical therapies receipt of mechanical ventilation, non-invasive ventilation (except as maintenance therapy for obstructive sleep apnea), FiO2≥ 0.6, central venous or arterial catheterization, continuous or acute renal replacement therapy in the ICU, receipt of ≥4L of crystalloid within a 2 hour period, and receipt of vasopressor or inotropic agents. Our a priori proposed definition for SCAP was receipt of intensive therapy in the ICU.
Microbiologic etiology was defined by results of cultures of blood, sputum, bronchoalveolar lavage (BAL), tracheal aspirate, and urinary or respiratory antigen screens obtained within 24 hours of admission. Antibiotic administration was determined by review of electronic pharmacy records. Receipt of initial appropriate antibiotic therapy was determined on the basis of the IDSA/ATS 2007 guidelines.(4)
This study was approved by the Intermountain Healthcare Institutional Review Board and authorized by the Utah Population Database; individual patient consent was not required.
We compared mortality and hospital length-of-stay for three possible definitions of SCAP—receipt of any intensive therapy regardless of site of care, receipt of any intensive therapy in the ICU, or ICU admission regardless of therapy received. Our a priori reference definition was receipt of intensive therapy in the ICU. Mortality was modeled with logistic regression, while LOS was modeled with linear regression after log transformation to achieve normality. For logistic models, test characteristics were evaluated with area under the receiver operating characteristic curve (AUC) with goodness of fit evaluated by the Hosmer-Lemeshow technique (c-statistic); comparisons of AUC among definitions were made using the technique of DeLong.(36)
To evaluate prediction of SCAP, logistic regression models were developed with our reference definition of SCAP (receipt of intensive therapy in the ICU) as the dependent variable, and CURB-65, SMART-COP, CURXO-80, and IDSA/ATS 2007 scores as the independent variables. We used the minor criteria from the IDSA/ATS 2007 guidelines, ignoring the presence of major criteria (mechanical ventilation or vasopressors prior to admission) to avoid falsely inflating the strength of prediction. In a sensitivity analysis, we excluded patients who met major criteria at the time of admission. These SCAP-prediction models were restricted to the subset of patients seen in the ED. In this same subset we developed a backward stepwise selection multiple regression model for the prediction of SCAP based on the IDSA/ATS 2007 predictors, including clinically reasonable interaction terms, eliminating predictors if p>0.1 and interaction terms if the percent change in the effect odds-ratio was less than 10% for the interacting predictors. We evaluated a factor for year of admission in this prediction model to confirm the consistency of data over time. In an exploratory analysis, we fitted cubic splines to model predictors likely to have polytonic effects.
Missing predictors were assumed to be normal when calculating the SAPS2, SMART-COP, CURB-65, IDSA/ATS 2007, and CURXO-80 scores and set to the mean of normal values for other patients for analyses requiring continuous data. Two sensitivity analyses were performed to evaluate the effect of missingness: exclusion of patients with missing data and multiple imputation by chained equations using the technique of van Buuren, et al (Stata command ice).(37) Statistical significance was defined as two-tailed p<0.05. Analyses were performed with Stata 10 (Stata Corporation, College Park, TX).
A total of 4,970 encounters in 4,375 patients had ICD-9 codes compatible with pneumonia. Of those, 3,287 had admission chest radiographs compatible with pneumonia. Of that group a total of 873 patients were excluded for chronic dialysis (58), prior pneumonia within a year (96), prior hospitalization within 90 days (435), or admission DNR/DNI status (333). This yielded a cohort of 2,413 episodes of radiographically confirmed CAP in 2,354 patients (see Figure 1). Of this cohort 1,540 (64%) were admitted to the hospital; 1,935 (80%) were evaluated in the ED, with 873 discharged to home from the ED. Of the hospitalized patients, 378 (25%) were admitted to the ICU during the hospital stay; 94 (25%) of these were initially admitted to the hospital ward with later transfer for worsening clinical status. The characteristics of the patient population are displayed in Table 2, including mean values for prediction scores, which differed significantly between ward and ICU patients.
Overall 30-day mortality was 3.7% (N=89). Thirty-day mortality was 16.1% (N=61) for ICU patients, 5.6% (N=86) for admitted patients, and 0.3% (N=3) for patients discharged from the ED. The predicted SAPS-2 mortality for the entire population was 12.9%. Of patients admitted to the ward, 855 (74%) received guidelines-compliant antibiotic therapy. Of patients admitted to the ICU, 264 (70%) received guidelines-compliant antibiotic therapy. The most common types of non-compliant therapy for both groups were fluoroquinolone monotherapy (in the ICU) or failure to include coverage for atypical organisms and Legionella in addition to a beta-lactam. The rate of microbiological diagnosis was 17% for ICU patients, 9% for ward patients, and 6% for all-comers. The predominant organisms were S. pneumoniae and S. aureus.
Most pre-admission predictors had few missing values. For hemodynamic predictors, <0.1% were missing. For BUN, 13.8% were missing (4.3% of admitted patients). Less common biochemical results were often missing, including albumin (49% missing among admitted patients) and arterial pH (74% missing among admitted patients). Formal P/F ratios before admission were only available for 13% of patients (37% of those admitted to the ICU). Admission S/F ratios were available for 87% of patients; the remainder lacked simultaneous recorded FiO2 and SpO2.
Of the 378 patients admitted to the ICU, 298 (79%) received an intensive therapy. Of those receiving any intensive therapy regardless of location of care (N=454), 171 (38%) received mechanical ventilation, 46 (10%) non-invasive positive pressure ventilation, 363 (80%) received FiO2≥0.6, 115 (25%) vasopressor therapy, 28 (6%) inotropic therapy, 17 (4%) emergent renal replacement, and 84 (19%) high-volume fluid resuscitation. The 30-day mortality rate for patients admitted to the ICU was 16.1%, while mortality was 13.9% for patients receiving any critical therapy, 27.5% for those receiving mechanical ventilation or vasopressors, and 37% for patients receiving both mechanical ventilation and vasopressors. Rates of receipt of intensive therapy and associated mortality are displayed in Table 3.
The three reference definitions of SCAP—ICU admission, receipt of intensive therapy, or receipt of intensive therapy with ICU admission—had similar performance in predicting thirty-day mortality. The AUC for the various definitions of SCAP were 0.77-0.78 (95% CI: 0.72-0.83), without significant difference among definitions. Incorporating SAPS-2 predicted mortality into the mortality prediction improved the AUC of the three definitions to 0.89-0.90 (95% CI: 0.85-0.93), again without significant difference among SCAP definitions.
For ICU-admitted patients, log-normalized mean ICU LOS was substantially longer (81 vs. 20 hours) among those who received a critical therapy. Hospital LOS was significantly longer for ICU-admitted patients than for patients admitted only to the hospital ward (6.9 vs. 2.9 days, p<0.01). The log-normalized mean hospital LOS for ICU admitted patients who received intensive therapy was 8.5 days, which was significantly longer (p<0.01) than the 3.9 days for ICU admitted patients who did not receive intensive therapy (excluding patients who died in the ICU). Patients who received an intensive therapy regardless of ICU admission had a hospital LOS of 6.5 days. Linear regression of the log-transformed hospital and ICU LOS for admitted patients had poor fit regardless of SCAP definition employed, including when SAPS-2 predicted mortality was incorporated into the regression model (R2 ~ 0.2 for all models).
Using the reference SCAP definition (receipt of intensive therapy in the ICU) in the subset of patients evaluated in the ED, the IDSA/ATS 2007 criteria performed better (p<0.01) than CURB-65, SMART-COP, and CURXO-80, with an AUC of 0.88 (95% CI 0.85-0.90). Significance remained even after a Bonferroni correction for multiple comparisons. Table 4 displays the performance of the various predictive models. Neither incorporation of SAPS-2 predicted mortality nor restricting the analysis to admitted patients affected the relative performance of the IDSA/ATS 2007 predictors to a significant degree. Sensitivity analysis using the other two possible definitions of SCAP (ICU admission or receipt of intensive therapy) demonstrated slightly lower AUC, but IDSA/ATS 2007 continued to perform better than the other models. Exclusion of patients who met major criteria at admission did not significantly change results. Consistent with its small AUC, specific CURB-65 cutoffs also performed poorly. Of patients with CURB-65≥ 2, 26% (N=207) were admitted to the ICU and received an intensive therapy, while 54% of patients with CURB-65≥ 4 (N=28) were admitted to the ICU and received an intensive therapy.
The probability of SCAP increased in a generally linear fashion with higher numbers of IDSA/ATS 2007 minor criteria, though with a clear stepup at 4 minor criteria, as displayed in Figure 2. The IDSA/ATS 2007 proposed cutoff of 3 minor criteria yielded a PPV for SCAP of 54% and a NPV of 94%. Using 4 minor criteria as a diagnostic cutoff yielded a PPV of 81% and a NPV of 92%. This 27 percentage point increase in PPV is larger than for any other threshold value of minor predictors. The positive likelihood ratio, a Bayesian measure independent of disease prevalence, reached an impressive 34.9 with four criteria. Positive and negative predictive values, sensitivity and specificity, and positive and negative likelihood ratios are displayed in Table 5. Exclusion of patients meeting major criteria before admission had only a minor effect on these estimates. Patients (N=41) meeting major criteria (pre-admission mechanical ventilation and/or vasopressor therapy) met on average 3.9 minor criteria, though with a range from 0 to 7 (median 4, inter-quartile range 3-5). Fifteen percent of patients meeting major criteria met two or fewer minor criteria.
The unweighted dichotomized minor criteria (AUC 0.88, 95% CI 0.85-0.90) were inferior by a small but statistically significant margin to the weighted dichotomized minor criteria for IDSA/ATS 2007 (AUC 0.90, 95% CI 0.88 - 0.92), p<0.01 for the comparison. When weighted continuous minor criteria were used, including cubic splines for heart rate, blood pressure (mean) and respiratory rate, the AUC was 0.92 (95% CI 0.90 - 0.94). The results of logistic regression are displayed in Table 6. Confusion was the most predictive of SCAP, though tachycardia, hypotension, low P/F ratio, and hypothermia were also predictive. Advanced age was negatively associated with SCAP, likely reflecting lower ICU referral rates for elderly patients independent of admission DNR/DNI documentation. A sensitivity analysis using S/F ratios only (excluding P/F ratios), did not significantly affect the validity of the IDSA/ATS 2007 minor criteria (data not shown).
Our study validates in a single US hospital the IDSA/ATS 2007 model for predicting SCAP in the ED. In our patient population, IDSA/ATS 2007 outperformed competing models and did so with a low data collection burden. We found that a cutoff of four minor criteria was more accurate than three minor criteria, though this will depend to some extent on local administrative and clinical characteristics in individual healthcare settings. Our results show improved prediction of SCAP over prior models. In our study IDSA/ATS 2007 clearly outperformed CURB-65, as expected on the basis of other studies.(17) Though we did not collect the data on comorbid conditions required to calculate a PSI, another study found PSI to be inferior to CURB-65 in predicting pneumonia severity.(8) On the basis of our data, we recommend that major criteria be automatic grounds for ICU admission and the minor criteria be used to triage patients not currently requiring mechanical ventilation or vasopressor therapy.
Our reference definition, when compared with other proposed definitions, was associated with similar 30-day mortality but clearly indicated a group of patients with longer hospital LOS. This definition has the benefit of describing types of therapies required and actual need for receipt of intensive therapy in addition to predicting longer hospital LOS, all important factors in healthcare triage. We also describe for the first time the range of intensive therapies received, the majority of which are high-dose oxygen therapy, vasopressors, invasive monitoring or access catheters, and mechanical ventilation. Our results are comparable to other studies for the most commonly described intensive therapies: 53% of ICU admits were mechanically ventilated in one study,(8) while in our ICU, 45% of admitted patients were mechanically ventilated.
In important respects our cohort is similar to other published cohorts. ICU admission rates for hospitalized pneumonia patients ranged from 9-27% in the four PORT hospitals.(9) Other studies suggest rates of 8-15%.(12, 14, 38, 39) In our cohort, 25% of hospitalized patients were admitted to the ICU. The higher rate of ICU admission could limit generalizability of our findings, though our reference definition for SCAP includes actual receipt of intensive therapies. The lower-than-predicted mortality in our cohort (versus SAPS-2 predicted mortality) may reflect our exclusion of patients with significant immune suppression, HCAP, and admission DNR/DNI orders. It may also reflect prevalent comorbidities or the beneficial effects of the locally implemented pneumonia guidelines.(28, 40)
Our study does have some limitations. It was retrospective and relied on ICD-9 coding rather than prospective screening. In the Netherlands and Sweden, this methodology missed some pneumonias, particularly in patients with prolonged hospital admissions.(41, 42) In the United States, and particularly in our hospital, however, ICD- 9 codes have been demonstrated to have reasonable sensitivity for diagnosis of CAP.(29, 43) The specificity of our technique was augmented by manual review of chest radiology reports. In addition, our use of real-time electronic charting for the extraction of critical therapies and outcomes significantly reduces the loss of accuracy generally associated with retrospective analyses. The electronic database contains a wealth of clinical detail for defining SCAP absent from older prospective cohorts. Our use of the prospectively gathered Utah Population Database for mortality outcomes further strengthens our analysis, although some out-of-state deaths may have escaped detection. Our identification of additional deaths within 30 days not reflected in hospital mortality underscores the importance of gathering actual 30-day mortality data rather than merely death during hospitalization. Because we did not capture patients who were neither admitted to the hospital nor seen in the hospital emergency department, we do not provide a broad portrait of non-severe CAP. Our results are comparable to other ED-based CAP cohorts, as few non-ED outpatient cohorts have been published.
Areas for future research include incorporation of the IDSA/ATS 2007 SCAP predictors into an electronic clinical care environment and evaluation of the effect of such methods on patient outcome and triage. Further work is also needed on characterizing the significance of appropriate initial triage. The IDSA/ATS 2007 guidelines appear to be a reasonable collection of predictors for development of SCAP and should be preferred to other models for these purposes.
We are grateful to Dominik Aronsky for his critical review of the manuscript, to Greg Stoddard for statistical consultation, to Susan Crapo and Joe Dalto for data extraction, and to Marc-Aurel Martial for data quality assurance.
This study was supported by a grant from the Deseret Foundation, with additional support from Public Health Services research grant UL1-RR025764 from the National Center for Research Resources.
The authors have not disclosed any potential conflicts of interest.