|Home | About | Journals | Submit | Contact Us | Français|
Transcripts from admission chest radiographs could aid in identification of pneumonia cases for public health surveillance. We assessed the reliability of radiographic data abstraction and performance of radiographic key terms to identify pneumonia in patients hospitalized with laboratory-confirmed influenza virus infection.
We used data on patients hospitalized with laboratory-confirmed influenza virus infection from October 2008 through December 2009 from 10 geographically diverse U.S. study sites participating in the Influenza Hospitalization Surveillance Network (FluSurv-NET). Radiographic key terms (i.e., bronchopneumonia, consolidation, infiltrate, airspace density, and pleural effusion) were abstracted from final impressions of chest radiograph reports. We assessed the reliability of radiographic data abstraction by examining the percent agreement and Cohen's k statistic between clinicians and surveillance staff members. Using a composite reference standard for presence or absence of pneumonia based on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes and discharge summary data, we calculated sensitivity, specificity, positive predictive value (PPV), and percent agreement for individual and combined radiographic key terms.
For each radiographic key term, the percent agreement between clinicians and surveillance staff members ranged from 89.4% to 98.6% and Cohen's k ranged from 0.46 (moderate) to 0.84 (almost perfect). The combination of bronchopneumonia or consolidation or infiltrate or airspace density terms had sensitivity of 66.5%, specificity of 89.2%, PPV of 80.4%, and percent agreement of 80.1%. Adding pleural effusion did not result in significant changes in sensitivity, specificity, PPV, or percent agreement.
Radiographic key terms abstracted by surveillance staff members from final impressions of chest radiograph reports had moderate to almost perfect reliability and could be used to identify pneumonia among patients hospitalized with laboratory-confirmed influenza virus infection. This method can inform pneumonia surveillance and aid in public health response.
Pneumonia is a common complication in patients hospitalized with influenza virus infection1–3 and an important clinical outcome for monitoring the severity of influenza or the impact of a particular influenza virus strain circulating in a given season. Although pneumonia can be identified by International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes,4 the utility of using ICD-9-CM codes for pneumonia surveillance in a public health response is limited because discharge ICD-9-CM codes are usually not available until weeks after hospital discharge, and the accuracy of ICD-9-CM codes can vary according to local coding practices.5,6 Chest radiographs are often obtained on patients with suspected pneumonia to confirm the diagnosis, rule out other potential causes of illness, and monitor pneumonia-associated complications.7,8 Using chest radiographic data that can become available for abstraction within 24 hours of hospital admission could enhance pneumonia surveillance among patients with influenza and provide a more timely tool for estimating the severity of the influenza season. Using data from the Centers for Disease Control and Prevention's (CDC's) Influenza Hospitalization Surveillance Network (FluSurv-NET), we assessed the reliability of radiographic data abstraction and performance of radiographic key terms to identify pneumonia in patients hospitalized with laboratory-confirmed influenza virus infection.
FluSurv-NET conducts population-based surveillance of laboratory-confirmed influenza-associated hospitalizations among children (<18 years of age) and adults.9 FluSurv-NET covers >80 counties and represents approximately 9% of the U.S. population. The network consists of 13 geographically diverse study sites, including 10 sites participating in the Emerging Infections Program (EIP). EIP sites, which include selected counties in California, Colorado, Connecticut, Georgia, Maryland, Minnesota, New Mexico, New York, Oregon, and Tennessee, were the only participating sites prior to the 2009–2010 influenza season. Clinical and demographic data are abstracted into a standardized data collection form from medical charts at participating sites. During each influenza season (October through April of the following year), these data are used to estimate weekly age-specific hospitalization rates and to assess the severity of the influenza season. Data elements (age, sex, and evidence of a positive influenza test) that are necessary for calculating age-specific hospitalization rates are abstracted as soon as laboratory-confirmed influenza-associated hospitalization is identified. Other data elements are abstracted after hospital discharge.
In FluSurv-NET, information on pneumonia diagnosis can be obtained based on abstraction of the first nine ICD-9-CM discharge codes and whether or not pneumonia is noted in the clinical discharge summary. During the 2008–2009 influenza season, detailed chest radiographic information was added to the surveillance data collection form in an attempt to capture data on radiographically confirmed pneumonia at hospital admission among patients hospitalized with laboratory-confirmed influenza virus infection. Surveillance staff members reviewed chest radiograph reports on chest radiographs performed within 24 hours of hospital admission; from the final chest radiograph impression, they abstracted the text and the radiographic key terms. The key terms were indicated by checked boxes and included bronchopneumonia (bronchopneumonia or pneumonia), consolidation, infiltrate (single lobar, multiple lobar, or interstitial), airspace density (airspace density/opacity), and pleural effusion. Because radiographic key terms were abstracted directly from the final chest radiograph impression, the abstraction of radiographic key terms should not have been influenced by other clinical data. A single impression could include one or multiple radiographic key terms. We assumed that impressions in which no radiographic key term was checked had no radiographic findings that could be associated with pneumonia.
To assess the reliability of the abstraction of radiographic key terms, two CDC physicians (reviewers A and B) independently conducted blinded reviews of a randomly selected sample (1,616 of 5,192 [31.1%]) of chest radiograph impression transcripts. For each radiographic key term abstracted from these impression transcripts, we assessed inter-observer agreement between each reviewer and surveillance staff members by calculating the percent agreement and Cohen's κ statistic. We calculated the percent agreement for the presence or absence of each radiographic key term as the proportion of transcripts read by both surveillance staff members and reviewer A or B as having or not having the radiographic key term. We classified inter-observer agreement as moderate (κ=0.41–0.60), substantial (κ=0.61–0.80), or almost perfect (κ=0.81–1.00).10
To determine which radiographic key terms could be used to identify patients with pneumonia through public health surveillance without the need to obtain full radiograph transcripts, we analyzed data only for patients whose surveillance data collection form had an ICD-9-CM code, a discharge summary, and radiographic data. We calculated sensitivity, specificity, -positive predictive value (PPV), and percent agreement for each radiographic key term and combinations of key terms. For these calculations, a composite reference standard for the presence or absence of pneumonia was determined by evaluating abstracted ICD-9-CM data and discharge summary data. We classified patients who had an ICD-9-CM code of 480–487.0 and a mention of pneumonia in the discharge summary as having pneumonia. We classified patients who did not have an ICD-9-CM code of 480–487.0 and who did not have a mention of pneumonia in the discharge summary as not having pneumonia. To minimize misclassification of presence or absence of pneumonia, we excluded patients with pneumonia identified by ICD-9-CM codes only or by discharge data only. We also excluded patients whose records did not indicate any radiographic key terms but did indicate that pneumonia could not be ruled out.
We calculated sensitivity, specificity, and PPV for each radiographic key term and for different combinations of radiographic terms classified into five groups based on specificity and PPV. We defined group 1 as a single radiographic key term with the highest specificity and PPV, group 2 as a radiographic key term included in group 1 or one additional radiographic term with the next highest specificity and PPV, and group 3 as radiographic key terms included in group 2 or one additional radiographic term with the next highest specificity and PPV. We followed the same algorithm for groups 4 and 5. We also calculated percent agreement for each group of terms to further assess diagnostic accuracy. Although all analyses were conducted on children and adults separately, we only report the adult data here. (Pediatric results are available on request.) We used Pearson's c2 or Fisher's exact tests to compare sensitivity, specificity, PPV, and percent agreement between the five groups of radiographic terms. The comparisons were as follows: group 1 vs. group 2, group 2 vs. group 3, group 3 vs. group 4, and group 4 vs. group 5. We performed all analyses using SAS® version 9.3.11
From October 2008 through December 2009, 5,192 (87.5%) of 5,935 adults hospitalized with laboratory-confirmed influenza virus infection had an available transcript of the final impression for a chest radiograph collected within 24 hours of admission. In a randomly selected sample of 1,616 (31.1%) transcripts, the percent agreement between surveillance staff members and reviewer A or B ranged from 89.4% for the key term infiltrate to 98.6% for the key term consolidation (Table 1). We found almost perfect agreement between surveillance staff members and reviewer (A or B) for the key term consolidation (κ=0.82 or 0.84) and substantial agreement for the terms pleural effusion (κ=0.78 or 0.77), airspace density (κ=0.75 or 0.78), and bronchopneumonia (κ=0.61 or 0.66). For the term infiltrate, we found substantial agreement (κ=0.70) between surveillance staff members and reviewer B but only moderate agreement (κ=0.46) between surveillance staff members and reviewer A.
Of 5,192 adults hospitalized with laboratory-confirmed influenza virus infection who had available radiographic data, 4,806 (92.6%) also had both -ICD-9-CM code and discharge summary data. We excluded 701 of 4,806 (14.6%) adults with pneumonia identified by ICD-9-CM only or by discharge summary data only. After excluding an additional 58 (1.2%) adults who had no radiographic key terms checked but who had a report indicating that pneumonia could not be ruled out, records for 4,047 of 4,806 (84.2%) adults were eligible for application of the composite reference standard definition for the presence or absence of pneumonia. Of these 4,047 adults, 1,618 (40.0%) were identified as having pneumonia based on concordance of ICD-9-CM codes with discharge summary data (Figure). The most common key radiographic terms predicting pneumonia diagnosis were infiltrate (15.3%, sensitivity = 32.1%), airspace density (13.6%, sensitivity = 24.9%), and bronchopneumonia (9.4%, sensitivity = 22.1%). Specificity for each radiographic key term ranged from 94.0% for airspace density to 99.0% for bronchopneumonia and 99.1% for consolidation. The key term bronchopneumonia also had the highest PPV (93.7%) (Table 2).
When we considered the five groups of radiographic key terms, starting with group 1 (bronchopneumonia only), sensitivity increased with each added radiographic key term (Table 3). Sensitivity increased from 22.1% to 28.7% (p<0.001) when the term consolidation was added to bronchopneumonia and to 51.6% (p<0.001) when the term infiltrate was added; percent agreement increased to 77.3% (p<0.001) when the term infiltrate was added. Specificity and PPV decreased and, respectively, reached 94.5% (p<0.001) and 86.2% (p=0.004), when the term infiltrate was added to the terms bronchopneumonia and consolidation. With the addition of airspace density, although sensitivity reached 66.5% (p<0.001) and percent agreement reached 80.1% (p=0.002), specificity decreased to 89.2% (p<0.001) and PPV to 80.4% (p<0.001). We found no significant changes in sensitivity, specificity, PPV, or percent agreement when the term pleural effusion was added.
Using chest radiograph data abstraction, we identified patients with pneumonia as part of public health surveillance for influenza-associated hospitalizations. Surveillance staff members can reliably abstract key terms from radiograph impressions, and combined terms such as bronchopneumonia, consolidation, infiltrate, and airspace density can, with reasonable accuracy, identify pneumonia based on the composite reference standard of ICD-9-CM codes and discharge summary data.
In our study, FluSurv-NET surveillance staff members performed similarly to physicians in abstracting radiographic key terms from chest radiograph impressions, demonstrating the reliability of trained public health surveillance staff members in identifying pneumonia in patients through medical chart abstraction for surveillance purposes. Although the percent agreement was high for all radiographic key terms and Cohen's κ indicated substantial to almost perfect agreement for most terms, we found only moderate agreement for the term infiltrate for one reviewer. Based on a survey of 151 physicians assessing the usefulness of including the term infiltrate on a radiographic report, Patterson and Sponaugle12 concluded that the term infiltrate is nonspecific and imprecise. According to the Fleishner Society glossary of terms for thoracic imaging by Hansell et al., the term opacity with relevant qualifiers is preferred over the use of infiltrate.13 Use of other terms in place of infiltrate could explain lower percent agreement for the term infiltrate in our study.
We observed low sensitivity and high specificity for each single radiographic key term considered in this analysis. Therefore, each radiographic key term may only be able to identify a proportion of all pneumonias. PPVs for single radiographic key terms ranged from 67.1% for pleural effusion to 93.7% for bronchopneumonia, indicating that certain key terms may be better predictors of pneumonia than others. Multiple key terms need to be considered when attempting to estimate a burden of pneumonia.
Although different combinations of radiographic key terms resulted in variable sensitivity, specificity, PPV, and percent agreement, the combination of bronchopneumonia, consolidation, infiltrate, and airspace density resulted in reasonable sensitivity, percent agreement, specificity, and PPV. Although ICD-9-CM codes are often used in epidemiologic studies to identify pneumonia, studies to validate the use of ICD-9-CM codes to identify pneumonia also show varying results based on the combination of ICD-9-CM codes used, the reference standard used, and the population studied.5,14–18 In studies conducted among adults, the sensitivity of ICD-9-CM codes to identify pneumonia ranged from 48% to 98% and PPV ranged from 57% to 96%.5,14–17 In a pediatric study that examined 12 combinations of ICD-9-CM codes representing pneumonia and pneumonia complications, the sensitivity for identification of pneumonia ranged from 61% to 99% and the PPV ranged from 54% to 90%.18 Given the variability in how pneumonia is identified by different combinations of radiographic key terms or different combinations of ICD-9-CM codes, the application of each combination depends on the population and outcome being studied.
Although using ICD-9-CM codes from administrative data is less labor intensive than radiographic data abstraction, radiographic key terms offer added specificity. First, radiographic key terms can be used to help identify pneumonia that is radiographically confirmed. Second, radiographic key term abstractions can be limited to radiographs performed within a certain time period after admission and can, therefore, distinguish between community-acquired pneumonia (CAP), ventilator-acquired pneumonia (VAP), or hospital-acquired pneumonia (HAP). When using ICD-9-CM codes, these designations are not always clear, and information on the timing of the evolution of pneumonia is not available. In addition, because radiographic data may be available within 24 hours of hospital admission, abstraction or electronic extraction of radiographic key terms could help identify pneumonia more quickly than the use of ICD-9-CM codes, which may not become available until two to three weeks after the initial medical encounter.19
Our data were subject to several limitations. First, only transcripts of radiographic impressions, not films, were available for review, and radiographic key terms were abstracted from the radiographic impressions. However, obtaining radiographic films is not practical for public health surveillance because of resource limitations. Second, we assumed that patients without any radiographic term checked had no radiographically confirmed pneumonia. If radiographic key terms were missed because of abstraction error, some pneumonia cases may have been misclassified. Third, we used a composite reference standard for presence or absence of pneumonia that was based on both ICD-9-CM codes and discharge summary diagnoses of pneumonia abstracted from the hospital discharge summary. Neither data source was perfect for identifying pneumonia; therefore, using our composite reference standard may also have led to misclassification of some pneumonia cases. However, because our composite reference standard was based on concordance between two data sources—ICD-9-CM and hospital discharge summary—the misclassification in our composite reference standard should be less than if we had used ICD-9-CM codes or discharge diagnoses alone. Nevertheless, we recognize that the data elements we used for our composite reference standard may not have differentiated between CAP, HAP, or VAP and could have led to an underestimation of statistical accuracy. Fourth, although we conducted this analysis using laboratory-confirmed influenza surveillance data, pneumonia identified using radiographic key terms could also have resulted from bacterial or other viral etiologies, because influenza alone or along with other bacterial and/or viral pathogens can lead to pneumonia.20 Finally, our study was based on data collected in an inpatient setting; therefore, our results may not be generalizable to other clinical settings.
Our study demonstrates that radiographic data can be reliably abstracted by trained public health surveillance staff members and that radiographic key terms can be used with reasonable accuracy to identify pneumonia in patients hospitalized with influenza. Timely abstraction of radiographic key terms could help identify pneumonia cases before ICD-9-CM discharge codes or discharge summaries become available. Our findings about combinations of radiographic terms could inform future efforts to identify cases of radiographically confirmed pneumonia in patients hospitalized with influenza virus infection and could be used as a marker for seasonal or pandemic influenza severity.
The authors thank the following people for their help with surveillance efforts: Susan Brooks and Katie Wymore, California Emerging Infections Program, Oakland, California; Deborah Aragon and Steve Burnite, Colorado Department of Public Health and Environment, Denver, Colorado; Darcy Fazio and James Meek, Yale University, Connecticut Emerging Infections Program, New Haven, Connecticut; Olivia Almendares, Wendy Baughman, Danielle Smith Finch, Norisse Tellman Misdary, and Kyle Openo, Georgia Emerging Infections Program—a collaboration among Georgia Division of Public Health, Emory University, and the Atlanta Veterans Administration Medical Center, Atlanta, Georgia; Maya Monroe, Maryland Department of Health and Mental Hygiene, Baltimore, Maryland; Dave Boxrud, Richard Danila, Craig Morin, Sara Vetter, and Team Influenza, Minnesota Department of Health, St. Paul, Minnesota; Kathy Angeles, Lisa Butler, Sarah Khanlian, and Robert Mansmann, New Mexico Emerging Infections Program, a collaboration between the New Mexico Department of Health, Santa Fe, New Mexico, and the University of New Mexico, Albuquerque, New Mexico; Nancy Spina, New York State Health Department, Albany, New York; Ruth Belflower, University of Rochester School of Medicine and Dentistry, Center for Community Health and Department of Medicine, Rochester, New York; Meredith Vandermeer, Oregon Public Health Division, Portland, Oregon; and Katie Dyer and Karen A. Leib, Vanderbilt University School of Medicine, Nashville, Tennessee.
The Influenza Hospitalization Surveillance Network (FluSurv-NET) is a collaboration of state health departments, academic institutions, and local partners, and is funded by the Centers for Disease Control and Prevention (CDC). This publication was supported in part by cooperative agreement #CDC-RFA-CK12-1202. The findings and conclusions in this article are those of the authors and do not necessarily represent the views of CDC.
Data collection for adult patients during the 2008–2009 influenza season was approved by the CDC Institutional Review Board (IRB) and was either approved or received exempt status by all surveillance site IRBs. Data collection for children was determined by CDC to be for routine public health purposes and, thus, did not require IRB approval. Participating sites submitted the pediatric protocol to their local IRBs for review as needed.