|Home | About | Journals | Submit | Contact Us | Français|
To assess the accuracy of physician billing claims for identifying acute respiratory infections in primary care.
Nine primary care physician practices in Montreal, Canada (2002–2005).
A validation study was carried out to compare diagnoses in 3,526 physician billing claims with diagnoses documented in the corresponding patient medical records.
In-office medical record abstraction.
Claims had a high positive predictive value (PPV), negative predictive value, and specificity for identifying respiratory infections; however, their sensitivity was below 50 percent. Large variation in sensitivity and PPV was observed among physicians.
Because claims data are now routinely used to monitor antibiotic prescribing in primary care, future research should determine if acute respiratory infection diagnoses are missing from claims at random, or if bias is present.
Several randomized, placebo-controlled trials of antibiotic use have shown that antibiotics do not provide clinical benefit to children or adults with upper respiratory tract infections (Hoaglund et al. 1950; Cronk et al. 1954; Howie and Clark 1970; Stott and West 1976; Verheij, Hermans, and Mulder 1994; Kaiser et al. 1996; Arroll 2005) and fail to prevent complicated bacterial infections (Gadomski 1993; Heikkinen et al. 1995). Yet, 75 percent of oral antibiotics prescribed to ambulatory patients are for pharyngitis, otitis media, sinusitis, bronchitis, common cold, and unspecified upper respiratory tract infection of likely viral etiology (McCaig and Hughes 1995), and 22–49 percent are estimated to be unnecessary (Kozyrskyj et al. 2004; Cadieux et al. 2007). Inappropriate use of antibiotics for respiratory infections promotes antibiotic resistance (Seppala et al. 1997; Austin, Kristinsson, and Anderson 1999; Pihlajamaki et al. 2001), increases health services utilization and costs (Little et al. 1997), and increases the risk of preventable drug-related adverse events (Classen et al. 1991). To enable the development of effective interventions to reduce inappropriate antibiotic use in primary care, determinants of inappropriate antibiotic prescribing and accurate methods for monitoring antibiotic use need to be identified.
Monitoring antibiotic prescribing in primary care is challenging because well-developed measures of antibiotic prescribing are scarce, often inaccurate, and may not reflect real prescribing practices. Studies of antibiotic prescribing in primary care have relied on physician self-reported prescribing (Mangione-Smith et al. 1999; Nash et al. 2002; Steinman, Landefeld, and Gonzales 2003), chart review or audit (Hueston, Jenkins, and Mainous 2000; Hutchinson et al. 2001; Mangione-Smith et al. 2002), or prescription claims (Mainous, Hueston, and Clark 1996; Majeed and Moser 1999; Wang et al. 1999; Steinke et al. 2000; Gill and Roalfe 2001; Kozyrskyj et al. 2004; Cadieux et al. 2007). Self-reported antibiotic prescribing was shown to underestimate actual antibiotic prescribing by about 30 percent (Mangione-Smith et al. 2002), and the cost of chart review is too high for wide-scale use. Prescription claims data avoid self-report bias, do not require additional data collection, and because they involve financial transactions, they are carefully audited by payers and have been found to be highly accurate (Tamblyn et al. 1995). Owing to these advantages, prescription claims are now used routinely to monitor antibiotic prescribing for respiratory infections in primary care (Mainous, Hueston, and Clark 1996; Majeed and Moser 1999; Wang et al. 1999; Steinke et al. 2000; Gill and Roalfe 2001; McCaig, Besser, and Hughes 2002; Kozyrskyj et al. 2004; Cadieux et al. 2007).
However, an important limitation of using prescription claims to monitor antibiotic prescribing is that treatment indication is not recorded on prescription claims. Treatment indication is required to determine the appropriateness of antibiotic prescribing; therefore, it must be inferred from other sources of information, such as physician billing claims for patient visits. If prescription claims are to be used to monitor antibiotic prescribing, then the accuracy of using diagnostic information in physician billing claims to infer the indication for antibiotic treatment needs to be assessed.
Two previous studies have assessed the accuracy of physician billing claims for identifying respiratory infection diagnoses, and both have shown promising results. The first was a study of administrative claims data from seven health insurance providers in Colorado, and it found that 79 percent of bronchitis diagnoses and 83 percent of pharyngitis diagnoses in administrative claims had a corresponding diagnosis in the written medical record (PPV; Maselli and Gonzales 2001). However, this study did not investigate what proportion of bronchitis and pharyngitis diagnoses documented in patient medical records was accurately documented in physician billing claims (sensitivity and specificity). The second study assessed the accuracy of Research Patient Data Repository (RPDR) claims from nine primary care clinics in the Brigham and Women's Primary Care Practice-Based Research Network in Boston and reported that 86 percent of respiratory infection diagnoses in RPDR claims had a corresponding diagnosis in the electronic health record (Linder et al. 2006). However, sensitivity and specificity estimates were not corrected for the verification bias introduced by over-sampling claims with a diagnosis of respiratory infection relative to claims without such a diagnosis (i.e., the study design inflated the prevalence of respiratory infection in the sample, relative to the true population prevalence; Begg and Greenes 1983; Irwig et al. 1994).
The objective of this study was to assess the accuracy of physician billing claims for identifying episodes of acute respiratory infection in primary care. In particular, we sought to estimate the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of physician billing claims.
A validation study was carried out to assess the sensitivity, specificity, PPV, and NPV of physician billing claims for identifying episodes of respiratory infection, as compared with the patient medical record. The study population comprised 34 Montreal region family physicians and 17,002 of their patients who were participating in the MOXXI electronic medication management trial (Tamblyn et al. 2006) in 2002–2005. All patients participating in the MOXXI trial had previously consented to share their medical records and provincial health insurance (RAMQ) data with researchers. These data were available for a period starting 1 year before patient enrollment date (2001 or later) until 2005, when the present study was conducted. From the available physician billing claims, we identified those with a diagnostic code (International Classification of Diseases, 9th Revision, or ICD-9) for laryngitis/tracheitis (464), common cold (460), influenza (480, 487), acute unspecified upper respiratory infection (465), pharyngitis/tonsillitis (462, 463, 034), otitis media (381, 382), sinusitis (461), acute bronchitis (466), or bacterial pneumonia (481–486); all decimal place suffixes of these ICD-9 codes were included. We purposefully selected 10 physicians who had been enrolled in the MOXXI trial for at least 2 years and had the most MOXXI-consenting patients (and therefore also had the most physician billing claims available for research purposes), and requested their consent.
Among the 10 physicians selected, we identified all MOXXI-consenting patients who had at least one physician billing claim with a diagnosis of acute respiratory infection during the study period, and randomly sampled 635 of those patients. We also identified all patients without any physician billing claim with a diagnosis of acute respiratory infection during the study period, and randomly sampled 94 of those patients. To improve the efficiency of data collection, we over-sampled patients with at least one diagnosis of acute respiratory infection, relative to those with no diagnosis of acute respiratory infection (Begg and Greenes 1983; Irwig et al. 1994). For each of the 729 patients sampled, we identified all physician billing claims generated during the study period (i.e., from 2001 or later, depending on the enrollment date, until 2005) and validated each one against the paper-based patient medical record. Information available in the physician billing claims included the patient's lifelong RAMQ personal identifier, physician license number, visit date, and ICD-9 diagnostic code.
Once the physician billing claims had been sampled, a list of sampled patients' names and RAMQ personal identifiers was generated and sent to each consenting physician's office. The selected patients' paper-based medical records were retrieved by the office staff and reviewed by one of the authors (G. C.). For each sampled physician billing claim, the corresponding visit was identified in the medical record, the date of the visit was recorded, and the presence or absence of all acute respiratory infections under study was ascertained from the clinical notes. At the time of the medical record abstraction, the reviewer was blinded to the ICD-9 code in the corresponding physician billing claim. Information abstracted from patient medical records was entered directly in an electronic, structured chart abstraction form and stored in an MS Access database, which had been prepopulated with participating physicians' license numbers, sampled patients' RAMQ identifiers, and visit dates according to physician billing claims. Intra-rater reliability was measured on 25 randomly selected visits that were reviewed a second time, and the percent agreement between the first and second review was 100 percent.
Data retrieved from patient medical records were linked directly to physician billing claims using the patient lifelong RAMQ personal identifier, physician license number, and visit date ±1 day.
For each type of acute respiratory infection under study, a 2 × 2 table of diagnoses abstracted from patient medical records versus diagnoses obtained from the corresponding physician billing claims was generated using SAS statistical software (Version 9.3, SAS Institute Inc., Cary, NC). Information retrieved from the patient medical record was treated as a gold standard. The prevalence, PPV, and NPV of physician billing claims for identifying acute respiratory infections were estimated. Sensitivity and specificity estimates were corrected for the over-sampling of claims with a diagnosis of acute respiratory infection relative to claims without such a diagnosis (Begg and Greenes 1983) using MS Excel 2003 (Version 5.1). To investigate between-physician variation in physician billing claim diagnosis accuracy, these analyses were repeated for each physician individually, combining all nine types of acute respiratory infection under study (because each physician contributed too few of each type of acute respiratory infection to analyze each type individually).
Because we sampled several claims (and medical record visits) per patient, we conducted a sensitivity analysis to assess the effect of clustering of claims within patients on our estimates of sensitivity, specificity, PPV, and NPV. We did this by generating 100 random samples of one claim per patient (n=729 claims) from our total sample of 3,526 claims, and averaging the estimates of sensitivity, specificity, PPV, and NPV over all 100 random samples, which is similar to bootstrapping methodology (Efron and Tibshirani 1994).
Ten physicians participating in the MOXXI trial were purposefully selected for this study, and nine agreed to participate. Among these nine physicians' patients enrolled in the MOXXI trial, we randomly selected 635 patients who had at least one claim with a diagnosis of acute respiratory infection, and 94 patients without any claims with a diagnosis of acute respiratory infection. These 729 patients made 3,526 visits to their respective MOXXI physicians during the study period (duration of 1–5 years, depending on the date of enrollment), for an average of 4.8 visits per patient. The medical records of all 729 sampled patients were abstracted, and written documentation for each of the 3,526 visits identified from physician billing claims was found in the corresponding patient's medical record. In all, 1,173 (33.3 percent) of sampled claims were positive for respiratory infection (Figure 1). Sixty-six percent of sampled patients were women, and the mean age of sampled patients was 47.6 (SD 21.0, range <1–90 years). The characteristics of patients enrolled in the MOXXI trial, as compared with those of the general population, have been discussed previously (Bartlett et al. 2005).
The agreement between the diagnosis in the medical record and the ICD-9 code in the physician billing claim is shown in Table 1, where shaded areas indicate concordant diagnoses. For example, there were 63 physician billing claims with a diagnosis of laryngitis, and for all 63 claims, a diagnosis of laryngitis was also documented in the medical record at the corresponding date; however, an additional 16 diagnoses of laryngitis were documented in medical records that were not documented in physician billing claims. The overall percent agreement for the presence of any acute respiratory infection was 72.5 percent, which is the sum of all diagnoses of respiratory infection present in the physician billing claim and the corresponding medical record (969) divided by the sum of all diagnoses of respiratory infection documented in medical records (1,337).
The proportion of physician billing claims with a diagnosis of acute respiratory infection confirmed in the patient medical record (PPV) was 0.93, 95 percent CI (0.91, 0.94), for all acute respiratory infection combined, and 0.84, 95 percent CI (0.81, 0.88), for respiratory infections of likely viral etiology (Table 2). The PPV for acute respiratory infections of potentially bacterial etiology was 0.89, 95 percent CI (0.87, 0.92), and ranged from 0.72, 95 percent CI (0.67, 0.78), for acute bronchitis to 0.91, 95 percent CI (0.85, 0.97), for bacterial pneumonia. Sensitivity of physician billing claims for all acute respiratory infections combined was 0.49, 95 percent CI (0.45, 0.53). With the exception of influenza, sensitivity was markedly lower for viral respiratory infections than for bacterial ones. Specificity was 0.99 or higher for all types of acute respiratory infection studied.
The prevalence of acute respiratory infection diagnoses in physician billing claims varied between physicians from 19.5 to 111.4 per 1,000 claims (Table 3). Sensitivity and PPV varied between physicians from 1.00, 95 percent CI (1.00, 1.00), to 0.19, 95 percent CI (0.06, 0.47), and from 0.98, 95 percent CI (0.96, 1.00), to 0.70, 95 percent CI (0.53, 0.87), respectively. The accuracy of physician billing claims for identifying acute respiratory infections did not appear to be higher among physicians who diagnosed more acute respiratory infections.
Our sensitivity analysis using only one claim per patient yielded estimates for sensitivity (0.55, 95 percent CI 0.45, 0.64), specificity (0.99, 95 percent CI 0.99, 1.00), PPV (0.93, 95 percent CI 0.90, 0.96), and NPV (0.94, 95 percent CI 0.91, 0.97) that were similar to the estimates obtained when all visits were used. The confidence intervals from the sensitivity analysis are wider because the sample size is smaller: 729 claims (one per patient) were used instead of all 3,526. This shows that the effect of within-patient clustering of claims on sensitivity, specificity, PPV, and NPV estimates is small.
The PPV of physician billing claims was high for all types of acute respiratory infection studied. Our study was the first to estimate the prevalence, sensitivity, and specificity of physician billing claims for identifying chart-documented acute respiratory infections in primary care. For all but one type of acute respiratory infection investigated, our sensitivity estimates were below 0.50. Our study was also the first to look at between-physician variation in physician billing claim diagnosis accuracy. We found that prevalence of respiratory infections in physician billing claims varied widely between primary care physicians. We also observed large unexplained between-physician variation in sensitivity and PPV of physician billing claims for identifying acute respiratory infections.
If physician billing claims had many false-positive diagnoses of respiratory infection, they would not be a useful data source for monitoring antibiotic prescribing. Therefore, a high PPV, or a high likelihood that diagnoses of respiratory infection in physician billing claims are also present in the corresponding patient medical record, provides support for using health administrative data for monitoring antibiotic prescribing. For most of the acute respiratory infection diagnoses investigated, our estimates of PPV were similar to those previously reported in the literature (Maselli and Gonzales 2001; Linder et al. 2006). However, our PPV estimate for influenza (0.66, 95 percent CI 0.58, 0.74) was much higher than the 0.20 reported by Linder et al. (2006), but the latter was aberrantly low as compared with other PPV estimates in the same study.
Previous studies have emphasized the high PPV of health administrative data for identifying episodes of respiratory infection, but have overlooked the importance of sensitivity (Maselli and Gonzales 2001; Linder et al. 2006). A high sensitivity is desirable because it suggests that the data capture a majority of visits for respiratory infections. A low sensitivity is problematic because it suggests that several visits for respiratory infections are not documented in health administrative data. Nondocumentation of visits for respiratory infections may or may not be associated with antibiotic prescribing, which may result in bias when using health administrative data to monitor antibiotic prescribing.
Our study estimated the sensitivity of physician billing claims for identifying acute respiratory infections. Our sensitivity estimates were below 0.50 for all types of acute respiratory infection studied except acute bronchitis, which raises concerns about the potential for bias. Whereas one previous study estimated the sensitivity of claims for identifying respiratory infections (Linder et al. 2006), the authors did not correct their sensitivity estimate for the verification bias introduced by over-sampling claims with a diagnosis of acute respiratory infection relative to claims without such a diagnosis (Begg and Greenes 1983; Irwig et al. 1994); consequently, they greatly overestimated sensitivity. For example, if we had not corrected our estimates for verification bias, our estimate of the sensitivity of physician billing claims for identifying laryngitis would have been 0.80, as compared with the corrected sensitivity estimate of 0.20.
We were first to investigate between-physician variation in physician billing claim diagnosis accuracy for acute respiratory infections. We found almost a sixfold variation between physicians in the prevalence of acute respiratory infections. We observed similar between-physician variation in the sensitivity and PPV of physician billing claims for identifying acute respiratory infections. We expected that claims submitted by physicians who diagnosed more acute respiratory infections would be more accurate for identifying acute respiratory infections, but we found that neither frequency nor prevalence of acute respiratory infections seemed to be related to physician billing claim diagnosis accuracy. This finding suggests that other factors are likely responsible for the observed between-physician variation in physician billing claim diagnosis accuracy.
A limitation of our study is that medical records may not represent a true gold standard for identifying acute respiratory infections diagnosed in primary care. The use of a single rater was also a limitation of our study, and systematic misclassification of acute respiratory infection diagnoses may have occurred as a result. Another limitation of our study was its small convenience sample of primary care physicians. Whereas physicians participating in the MOXXI trial are generally similar to other eligible physicians in the Montreal region, they tend to be younger than MOXXI nonparticipants. If physician billing claim diagnosis accuracy is related to physician age or practice experience, then our study results may not be applicable to older or more experienced physicians. Also, the MOXXI trial involves physicians practicing in urban and suburban areas, and our results may not be generalizable to physicians practicing in rural areas. Furthermore, patients enrolled in the MOXXI trial tend to differ from nonparticipating patients in that they are generally older, with more complex health status, and have more visits to the MOXXI physician (Bartlett et al. 2005). Younger, healthier patients may be underrepresented in our study sample. Future research should involve a large random sample of primary care physician from both urban and rural areas, and a stratified random sample of patients from each physician's practice population.
Because physician billing claims and prescription claims are now routinely used to monitor antibiotic prescribing for acute respiratory infections in primary care (Mainous, Hueston, and Clark 1996; Majeed and Moser 1999; Wang et al. 1999; Steinke et al. 2000; Gill and Roalfe 2001; Kozyrskyj et al. 2004; Cadieux et al. 2007), it is important for future research to determine whether half of all acute respiratory infections diagnoses are missing from physician billing claims at random, or whether bias is present. If bias is present, future research should also focus on identifying determinants of physician billing claim diagnosis accuracy, so that appropriate corrections for the resulting bias can be developed and applied when physician billing claims are used to infer treatment indication for antibiotic prescribing. As suggested by the large between-physician variation observed in this study, physician characteristics may be associated with physician billing claim diagnosis accuracy. The effect of physician characteristics, as well as patient, encounter, practice, and billing characteristics, on physician billing claim diagnosis accuracy should be assessed.
Joint Acknowledgement/Disclosure Statement: Operational funding for this study was provided by the Canadian Institutes of Health Research. We would like to thank the MOXXI field coordinator, Rola El Halabieh, for her help in contacting physicians to participate in this study, and the MOXXIIT team (Jimmy Fragos and Marc-André Larochelle) for their help with database management.
The following supplementary material for this article is available online:
Verification Bias Correction.
This material is available as part of the online article from: http://www.blackwell-synergy.com/doi/abs/10.1111/j.1475-6773.2008.00873.x (this link will take you to the article abstract).
Please note: Blackwell Publishing is not responsible for the content or functionality of any supplementary materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.