Administrative data have high variation in validity for recording heart failure

Can J Cardiol. 2010 October; 26(8): e306–e312.

PMCID: PMC2954539

Administrative data have high variation in validity for recording heart failure

Susan Quach, MSc,¹ Claudia Blais, PhD,² and Hude Quan, MD PhD¹

¹Department of Community Health Sciences, University of Calgary, Calgary, Alberta;

²Institut national de santé publique du Québec, Québec, Québec

Correspondence: Dr Hude Quan, Department of Community Health Sciences, University of Calgary, 3330 Hospital Drive Northwest, Calgary, Alberta T2N 4N1. Telepone 403-270-7307, fax 403-944-8950, e-mail hquan/at/ucalgary.ca

Received September 26, 2009; Accepted May 10, 2010.

This article has been cited by other articles in PMC.

Abstract

BACKGROUND:

Many studies have relied on administrative data to identify patients with heart failure (HF).

OBJECTIVE:

To systematically review studies that assessed the validity of administrative data for recording HF.

METHODS:

English peer-reviewed articles (1990 to 2008) validating International Classification of Diseases (ICD)-8, -9 and -10 codes from administrative data were included. An expert panel determined which ICD codes should be included to define HF. Frequencies of ICD codes for HF were calculated using up to the 16 diagnostic coding fields available in the Canadian hospital discharge abstract during fiscal years 2000/2001 and 2005/2006.

RESULTS:

Between 1992 and 2008, more than 70 different ICD codes for defining HF were used in 25 published studies. Twenty-one studies validated hospital discharge abstract data; three studies validated physician claims and two studies validated ambulatory care data. Eighteen studies reported sensitivity (range 29% to 89%). Specificity and negative predictive value were greater than 70% across 17 studies. Nineteen studies reported positive predictive values (range 12% to 100%). Ten studies reported kappa values (range 0.39 to 0.84).

For Canadian hospital discharge data, ICD-9 and -10 codes 428 and I50 identified HF in 5.50% and 4.80% of discharge records, respectively. Additional HF-related ICD-9 and -10 codes did not impact HF prevalence.

CONCLUSION:

The ICD-9 and -10 codes 428 and I50 were the most commonly used to define HF in hospital discharge data. Validity of administrative data in recording HF varied across the studies and data sources that were assessed.

Keywords: Administrative data, Heart failure, Hospital discharge data, Validity

Résumé

HISTORIQUE :

De nombreuses études se fient à des données administratives pour repérer les patients souffrant d’insuffisance cardiaque (IC).

OBJECTIF :

Procéder à une analyse systématique des études qui ont évalué la validité des données administratives pour documenter l’IC.

MÉTHODOLOGIE :

Des articles révisés par des pairs rédigés en anglais (1990 à 2008) validant les codes de données administratives de la Classification internationale des maladies (CIM)-8, 9 et 10 étaient inclus dans l’étude. Un comité d’experts a déterminé les codes du CIM qui devraient être inclus pour définir l’IC. La fréquence des codes du CIM pour l’IC était calculée au moyen de jusqu’à 16 champs de codage diagnostique inscrits dans les données canadiennes sur les congés des patients pendant les exercices financiers 2000–2001 et 2005–2006.

RÉSULTATS :

Entre 1992 et 2008, plus de 70 codes de CIM différents pour définir l’IC ont été utilisés dans 25 études publiées. Vingt et une études ont validé les données canadiennes sur les congés des patients, trois études, les réclamations des médecins et deux études, les données sur les soins ambulatoires. Dix-huit études ont fait état de la sensibilité (plage de 29 % à 89 %). La spécificité et la valeur prédictive négative étaient supérieures à 70 % dans 17 études. Dix-neuf études ont indiqué une valeur prédictive positive (plage de 12 % à 100 %). Dix études ont précisé les valeurs kappa (plage de 0,39 à 0,84).

Pour ce qui est des données canadiennes sur les congés des patients, les codes 428 et I50 du CIM-9 et du CIM-10 ont permis de recenser une IC dans 5,50 % et 4,80 % des dossiers de congé, respectivement. Les autres codes liés à l’IC du CIM-9 et du CIM-10 n’avaient pas de répercussions sur la prévalence d’IC.

CONCLUSION :

Les codes 428 et I50 du CIM-9 et du CIM-10 étaient les plus utilisés pour définir l’IC dans les données de congé hospitalier. La validité des données administratives pour documenter l’IC variait selon les études et les sources de données évaluées.

Heart failure (HF) has been widely studied because of its high mortality and morbidity rates (1–4). Studies on HF have relied on multiple data sources such as surveys, disease registries, hospital charts and administrative data. Of these sources, administrative data have been increasingly used for health services utilization and outcome evaluation.

Administrative data result from implementing health care delivery, enrolling members into health insurance plans and reimbursing health care providers for services (5). The types of administrative data depend on the health care system and the funding structure. For example, Canada has a nationally funded health care system that collects hospital discharge data at a national level, while insurance registry, physician services and emergency room visit information is collected at a provincial level. Although these data are not intended for research, they are widely used in surveillance and outcome studies. Therefore, the validity of the research results rely on the quality of the data. Data quality is critical for interpreting variation across studies from different geographical areas.

The purpose of the present study was to systematically review the literature pertaining to the validity of administrative data for recording HF and apply the International Classification of Diseases (ICD) HF definitions to Canadian hospital discharge abstract data to explore the prevalence of HF across ICD code definitions.

METHODS

Literature review

Search strategy:

Only English language articles that compared the validity of ICD codes in administrative databases (eg, physician claims data, hospital discharge) with a reference standard (eg, chart review, registry, survey, case notes) were included. The literature search focused on articles published from 1990 to 2008. To identify peer-reviewed journal articles, both the Ovid MEDLINE database and EMBASE were searched. Because validation studies were not indexed consistently with standard MeSH terms, the following key search terms were used: ‘admin data’, ‘administrative data’, ‘population surveillance’, ‘heart failure’, ‘congestive heart failure’, ‘hospital data’, ‘physician claims’, ‘claims’, ‘hospital discharge’, ‘validation’ and ‘registries’. Next, Boolean operators were used to join search terms within the database search engines to locate articles. The reference lists from these articles were scanned to identify any additional articles that met the study criteria. Finally, using the articles identified, the authors were contacted by e-mail to identify any additional related articles. The articles selected from the citations and abstracts were reviewed by one investigator (SQ) to ensure they met the study criteria. When a decision could not be made regarding applicability, two other researchers (HQ and CB) were consulted to review the article for inclusion or exclusion. In cases for which it was unclear whether the publication used administrative data for validation, the article was excluded.

Data abstractions:

Studies were abstracted to review measures of validity including sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). When not reported, these measures were calculated using data within the article. For cases in which ICD codes were missing, an attempt was made to contact the corresponding authors for further details. Validity is an expression of the degree to which a parameter reflects what it is intended to measure (6). In validation studies, it is the ability of administrative databases to distinguish between diseased versus non-diseased. Sensitivity measures how well administrative data detect the presence of a disease for individuals who actually have the disease according to a reference standard, while specificity refers to how well the administrative data avoid the problem of falsely classifying individuals as diseased when they do not have the disease. Agreement was assessed by the kappa statistic, which was classified as poor (kappa less than 0.2), fair (kappa 0.2 to 0.39), moderate (kappa 0.4 to 0.59), good (kappa 0.60 to 0.79) and very good agreement (0.8 to 1.00) (7). In addition, the following descriptive data were abstracted from the studies: author, year of publication and data collection, study location, sample size, ICD version and codes, reference/‘gold’ standard and administrative data.

Application of HF ICD codes in Canadian hospital discharge data

The review showed great variation in ICD codes for HF definition. To ensure that all HF-related ICD codes were captured, four physicians independently reviewed a list of ICD-9-Clinical Modification and ICD-10 codes that were extracted from HF validation studies. These physicians have extensive experience with using administrative data to conduct cardiology research. For each diagnostic code, the physicians decided whether the code should be used to identify patients with HF in administrative data, responding with “yes”, “no” or “unsure”. Based on these responses, a list of ICD codes was analyzed in the Canadian national hospital Discharge Abstract Database for fiscal years 2000/2001 and 2005/2006 to determine whether inclusion of these codes would change the frequency of documented HF. The Discharge Abstract Database contains up to 16 diagnosis fields for ICD-9 data in 2000/2001 and 25 diagnosis fields for ICD-10 data in 2005/2006. The province of Quebec was excluded from the 2005/2006 data because the province was using the ICD-9 catalogue at that time. The frequency of discharge records for each diagnosis code was calculated. Descriptive statistical analyses were performed using SAS version 9.1 (SAS Institute Inc, USA).

RESULTS

Literature review

The literature search identified 2596 citations for review (Figure 1), with 47 citations related to the study objectives. Of 25 studies published between 1992 and 2008, nine were completed in Canada, 11 in the United States, and one each in Sweden, Portugal, the United Kingdom, Australia and Denmark. The sample size across the studies ranged from 126 to 58,816 patients (Table 1).

Figure 1)

Flow diagram illustrating the selection of articles that were reviewed

TABLE 1

Summary of heart failure validation studies

More than 70 ICD codes were used to define HF in the studies. Of these studies, 17 validated ICD-9 codes only, three validated ICD-10 codes only, one study examined ICD-8 codes, and four validated both ICD-9 and ICD-10 codes. Two studies (8,9) did not report the ICD codes used. ICD-8 and -9 code 428, and ICD-10 code I50 were commonly used across studies.

From 25 validation studies, 19 used the medical chart review as the reference standard (Table 2). The remaining six studies (10–15) relied on surveys, registries or recoded records. Methods for determining HF presence varied across studies, especially when chart review was used as the reference standard. Seven studies accepted HF diagnosis documented in the chart (12,16–21) and four studies used documented HF diagnosis or a combination of clinical judgement based on information in the chart (eg, test results, diagnostics) (22–25). Eight studies (15,25–31) ascertained HF presence based on diagnostic criteria used by scoring systems from Framingham, Carlson, Boston, the National Health and Nutrition Examination Survey (NHANES), the New York Heart Association (NYHA) or the European Society of Cardiology (ESC). Four studies (25–27,32) further classified HF according to levels of evidence such as definite or possible HF using the ESC or Boston criteria. Two studies compared the PPV against different clinical definitions of HF, and found that the PPV varied between criteria used (Lee et al [30], 94% Framingham versus 89% Carlson; McCullough et al [31], 56% NHANES versus 64% Framingham versus 83% NYHA).

TABLE 2

Results of validation studies according to in-hospital discharge data and other types of administrative data

Twenty-one studies validated hospital discharge data, three studies (9,14,21) examined physician claims data and two studies (8,21) examined ambulatory care data. One study (14) linked multiple administrative data sources (facility, physician claims and pharmacy).

Six studies (10,12,21,25,30,31) defined HF using the principal diagnosis or main diagnosis, or most responsible diagnosis only. Four studies (13,16,27,11) examined the difference between using the principal or major diagnosis versus secondary diagnoses. Information about the number of diagnosis fields or position of diagnosis in the coding field was not reported in the remaining studies.

Eighteen studies reported sensitivity values ranging from 29% to 89%, and PPV ranging from 12% to 100%. Specificity was higher than 70% across 14 studies and NPV was higher than 74%. Kappa was reported in 10 studies, ranging from 0.39 to 0.84, suggesting fair to very good agreement.

Analysis of Canadian hospital discharge databases

The results of the physician’s review on HF code are shown in Table 3. In 2000/2001, 180,055 (5.5%) of all discharges were coded with HF (ICD-9-Clinical Modification code 428) in any one of the 16 diagnosis fields, with 34% (n=60,723/180,055) of discharges in the first diagnosis field. In 2005/2006, there were 118,029 (4.8%) discharges with ICD-10 code I50 in any of the coding fields, with 36% (n=42,490/118,029) of discharges in the primary diagnosis field. Adding other HF-related ICD codes to 428 or I50 did not change the proportion significantly (from 5.50% to 5.53% for ICD-9 data, and from 4.80% to 4.90% for ICD-10 data).

TABLE 3

Frequency of hospital discharge records with International Classification of Diseases (ICD)-9/ICD-9-Clinical Modification (CM) heart failure codes in 2000/2001 and ICD-10 heart failure codes in 2005/2006

DISCUSSION

Our review demonstrated high variation in the validity of administrative data for recording HF. The sensitivity and PPV ranged widely, but specificity and NPV were substantially high across studies. This suggests that the validity of administrative data should be considered before analyzing and interpreting results. Differences in validity can be explained by factors related to ICD coding systems, organization, coder’s experience, reference standards and type of administrative database.

The validity of administrative data is extremely important given the increased use of these data for surveillance and outcome research. For population surveillance, those admitted to the hospital with HF will most likely represent severe cases of the disease in the late stages of the diagnosis; therefore, relying solely on the HF-defining ICD codes (428 or I50) in hospital discharge data will underestimate the prevalence of disease (32,33). For the majority of studies reviewed, hospital discharge data were the primary administrative data source used for validation. More research is needed to examine the validity of outpatient or community data for population surveillance.

More than 70 ICD-8, -9 and -10 codes were used to define HF across the validation studies. The European Cardiovascular Indicators Surveillance Set (EUROCISS) project (34) suggests that validation studies on HF should consider ICD codes for HF, hypertensive heart, other primary cardiomyopathies, alcoholic cardiomyopathy, secondary cardiomyopathy and chronic cor pulmonale. However, ICD-9 code 428 and ICD-10 code I50 accounted for the majority of HF cases in the Canadian hospital discharge data, and other codes had a smaller impact on the prevalence. However, the impact of each code on HF outcomes should be evaluated in the future.

We noticed that the prevalence of HF differed between ICD-9 and ICD-10 data. There are several potential explanations for this difference, possibly related to the learning curve from ICD-9 to the new coding system ICD-10, the declining incidence of HF in that period or improved HF outpatient management. Quan et al (19) evaluated the validity of HF and found little impact of the coding system change from ICD-9 to ICD-10 on HF validity in Canadian data (sensitivity 71.6% in ICD-9 and 68.6% in ICD-10; PPV 90.5% in ICD-9 and 90.2% in ICD-10).

We found that of nine Canadian studies, six demonstrated high PPV estimates (greater than 85%) and two studies reported PPV values of 51% and 65%. The variation can be partially explained by the variation in reference standards for validating HF across studies. Some studies used HF recorded in the patient’s chart as a reference standard, while other studies defined HF based on clinical or diagnostic evidence in the chart. Studies that relied on chart documentation of HF are likely to produce higher sensitivity values but lower PPVs than those that defined HF using a consistent clinical definition (Framingham, NYHA, Boston and ESC criteria). However, previous studies (35–37) have shown that clinical information is often missing in the patient chart, which can leave the diagnosis questionable. Furthermore, many countries have shown that diagnostic tests are not consistently conducted in all patients with suspected HF (38). Diagnosis of HF is challenging, and cases can be misclassified under other diseases because symptoms can be nonspecific to HF, particularly in the elderly (39).

Various administrative data definitions were used to define HF across studies, and this can explain much of the variability across the results. For example, many studies used a variety of ICD codes to define HF with ICD-9 and -10 codes 428 and I50 as the only codes being consistently applied across all studies. Some studies (10–12,16,25,29,30) defined HF using the principal/most responsible/major diagnosis alone. Using this method meant including patients with HF who were severe enough to be hospitalized or whose HF contributed to the length of stay most likely to generate high PPV estimates. Other studies (16–20,23,28) defined HF using all available diagnosis coding fields, resulting in inclusion of patients with HF as a comorbidity or complication. HF is likely to be under coded when HF is present as a comorbidity (ie, it does not contribute to length of stay significantly), which likely generates low PPV estimates.

CONCLUSION

The validity of administrative data in recording HF varies across studies; therefore, researchers or users of the data should consider data validity before addressing research questions.

Acknowledgments

The authors thank Dr Andy Wielgosz, Dr Jack Tu, Dr Douglas Lee and Dr Finlay McAlister for their expertise in reviewing the ICD codes. Special thanks to Dr Jennifer Pereira, Amanda Shane, Dr Sulan Dai, Dr Christina Bancej and Peter Walsh for their assistance in reviewing this manuscript. Dr Quan is supported by Alberta Innovates – Health Solutions.

REFERENCES

1. Annema JH, Jaarsma T, van Veldhuisen DJ. Reasons for hospital readmission for heart failure from different perspectives. Eur J Cardiovasc Nurs. 2008;7:S44–S45.

2. Jong P, Vowinckel E, Liu P, Gong Y, Tu J. Prognosis and determinants of survival in patients newly hospitalized for heart failure: A population-based study. Arch Intern Med. 2002;162:1689–94. [PubMed]

3. Ko D, Tu J, Masoudi F, Wang Y, Harranek E. Quality of care and outcomes of older patients with heart failure hospitalized in the United States and Canada. Arch Intern Med. 2005;165:2486–92. [PubMed]

4. Lee D, Johansen H, Gong Y, et al. Regional outcomes of heart failure in Canada. Can J Cardiol. 2004;20:599–607. [PubMed]

5. Iezzoni L. Risk Adjustment for Measuring Health Outcomes. 4th edn. Chicago: Health Administration Press; 2003.

6. Last J. A Dictionary of Epidemiology. 4th edn. New York City: Oxford; 2001.

7. Altman D. Practical Statistics for Medical Research. London: Chapman & Hall; 1991.

8. Fowles J, Lawthers A, Weiner J, Garnick D, Petrie D, Palmer R. Agreement between physicians’ office records and Medicare Part B claims data. Health Care Financ Rev. 1995;16:189–99. [PubMed]

9. Fowles JB, Fowles EJ, Craft C. Validation of claims diagnosis and self reported conditions compared with medical records for selected chronic diseases. J Ambul Care Manage. 1998;21:24–34. [PubMed]

10. Austin P, Daly PA, Tu JV. A multicenter study of the coding accuracy of hospital discharge administrative data for patients admitted to cardiac care units in Ontario. Am Heart J. 2002;144:290–6. [PubMed]

11. Fisher ES, Whaley FS, Krushat WM, et al. The accuracy of Medicare’s hospital claims data: Progress has been made, but problems remain. Am J Public Health. 1992;82:243–8. [PubMed]

12. Juurlink D, Preyra C, Croxford R, et al. Canadian Institute for Health Information Discharge Abstract Database: A Validation Study. Toronto: Institute for Clinical Evaluative Sciences; 2006.

13. Kashner T. Agreement between administrative files and written medical records: A case of the Department of Veterans Affairs. Med Care. 1998;36:1324–36. [PubMed]

14. Rector T, Wickstorm M, Greenlee T, et al. Specificity and sensitivity of claims-based algorithms for identifying members of Medicare+Choice health plans that have chronic medical conditions. Health Serv Res. 2004;39:1839–58. [PMC free article] [PubMed]

15. Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems: Implications for outcomes research. Ann Intern Med. 1993;119:844–50. [PubMed]

16. Birman-Deych E, Waterman A, Yan Y, Nilasena D, Radford M, Gage B. Accuracy of ICD-9-CM codes for identifying cardiovascular and stroke risk factors. Med Care. 2005;43:480–5. [PubMed]

17. Levy A, Tamblyn R, Fitchett D, McLeod P, Hanley J. Coding accuracy of hospital discharge data for elderly survivors of myocardial infarction. Can J Cardiol. 1999;15:1277–82. [PubMed]

18. Quan H, Parsons G, Ghali W. Validity of information on comorbidity derived from ICD-9-CCM administrative data. Med Care. 2002;40:675–85. [PubMed]

19. Quan H, Li B, Saunders L, et al. Assessing validity of ICD-9-CM and ICD-10 administrative data in recording clinical conditions in a unique dually coded database. Health Serv Res. 2007;43:1424–41. [PMC free article] [PubMed]

20. So L, Quan H, Evans D. ICD-10 coding algorithms for defining comorbidities of acute myocardial infarction. BMC Health Serv Res. 2006;6:161. [PMC free article] [PubMed]

21. Wilchesky M, Tamblyn RM, Huang A. Validation of diagnostic codes within medical services claims. J Clin Epidemiol. 2004;57:131–41. [PubMed]

22. Goff D, Dilip K, Pandey M, Chan F, Ortiz C, Nichaman M. Congestive heart failure in the United States: Is there more than meets the I(CD code)? The Corpus Christi Heart Project. Arch Intern Med. 2000;160:197–202. [PubMed]

23. Humphries KH, Rankin JM, Carere RG, Buller CE, Kiely FM, Spinelli JJ. Co-morbidity data in outcomes research: Are clinical data derived from administrative databases a reliable alternative to chart review? J Clin Epidemiol. 2000;53:343–9. [PubMed]

24. Schellenbaum GD, Heckbert SR, Smith NL, et al. Congestive heart failure incidence and prognosis: Case identification using central adjudication versus hospital discharge diagnoses. Ann Epidemiol. 2006;16:115–22. [PubMed]

25. Teng K, Finn J, Hung J, Geelhoed E, Hobbs M. A validation study: How effective is the Hospital Morbidity Data as a surveillance tool for heart failure in Western Australia? Aust N Z J Public Health. 2008;32:405–7. [PubMed]

26. Fonseca C, Sarmonto PM, Marques F, Ceia F. Validity of a discharge diagnosis of heart failure: Implications of misdiagnosing. Congest Heart Fail. 2008;14:187–91. [PubMed]

27. Inglesson E, Arnlov J, Sundstrom J, Lind L. The validity of a diagnosis of heart failure in a hospital discharge register. Eur J Heart Fail. 2005;7:787–91. [PubMed]

28. Khand AU, Shaw M, Gemmel I, Cleland JGF. Do discharge codes underestimate hospitalisation due to heart failure? Validation study of hospital discharge coding for heart failure. Eur J Heart Fail. 2005;7:792–7. [PubMed]

29. Kumler T, Gislason GH, Kirk V, et al. Accuracy of a heart failure diagnosis in administrative registers. Eur J Heart Fail. 2008;10:658–60. [PubMed]

30. Lee DS, Donovon L, Austin PC, et al. Comparison of coding of heart failure and comorbidities in administrative and clinical data for use in outcomes research. Med Care. 2005;43:182–8. [PubMed]

31. McCullough PA, Philbin EF, Spertus JA, Kaatz S, Sandberg KR, Weaver WD. Confirmation of a heart failure epidemic: Findings from the Resource Utilization Among Congestive Heart Failure (REACH) study. J Am Coll Cardiol. 2002;39:60–9. [PubMed]

32. Akosah K, Moncher K, Schaper A, Havlik P, Devine S. Chronic heart failure in the community: Missed diagnosis and missed opportunities. J Card Fail. 2001;7:232–8. [PubMed]

33. Di Bari M, Pozzi C, Cavallini M. The diagnosis of heart failure in the community. Comparative validation of four sets of criteria in unselected older adults: The ICARe Dicomano Study. J Am Coll Cardiol. 2004;44:1601–8. [PubMed]

34. Eurociss Working Group EUROCISS: Cardiovascular Indicators Surveillance Set. 1–225. 2008.

35. Norman G, Neufeld V, Walsh A, Woodward C, McConvey G. Measuring physicians’ performances by using simulated patients. J Med Educ. 1985;60:925–34. [PubMed]

36. Rethans J, Martin E, Metsemakers J. To what extent do clinical notes by general practioners reflect acutal medical performance? A study using simulated patients. Br J Gen Pract. 1994;44:153–6. [PMC free article] [PubMed]

37. Walraven C, Talijaard M, Bell C, et al. Information exchange among physicians caring for the same patient in the community. CMAJ. 2008;179:1013–8. [PMC free article] [PubMed]

38. The Study Group of Diagnosis of the Working Group on Heart Failure of the European Society of Cardiology. Komajda M, Follath F, et al. The EuroHeart Failure Survey programme – a survey on the quality of care among patients with heart failure in Europe: Part 2: Treatment. Eur Heart J. 2003;24:464–74. [PubMed]

39. Rutten FH, Cramer MJ, Grobbee DE, et al. Unrecognized heart failure in elderly patients with stable chronic obstructive pulmonary disease. Eur Heart J. 2005;26:1887–94. [PubMed]

40. Heckbert S, Kooperberg C, Safford MM, et al. Comparison of self report, hospital discharge codes, and adjudication of cardiovascular events in the women’s health initiative. Am J Epidemiol. 2004;160:1151–8. [PubMed]

Articles from The Canadian Journal of Cardiology are provided here courtesy of
Pulsus Group