|Home | About | Journals | Submit | Contact Us | Français|
Asthma imposes a heavy and expensive burden on individuals and populations. A population-based surveillance and research program based on health administrative data could measure and study the burden of asthma; however, the validity of a health administrative data diagnosis of asthma must first be confirmed.
To evaluate the accuracy of population-based provincial health administrative data in identifying adult patients with asthma for ongoing surveillance and research.
Patients from randomly selected primary care practices were assigned to four categories according to their previous diagnoses: asthma, chronic obstructive pulmonary disease, related respiratory conditions and nonasthma conditions. In each practice, 10 charts from each category were randomly selected, abstracted, then reviewed by a blinded expert panel who identified them as asthma or nonasthma. These reference standard diagnoses were then linked to the patients’ provincial records and compared with health administrative algorithms designed to identify asthma. Analyses were performed using the concepts of diagnostic test evaluation.
A total of 518 charts, including 160 from individuals with asthma, were reviewed. The algorithm of two or more ambulatory care visits and/or one or more hospitalization(s) for asthma in two years had a sensitivity of 83.8% (95% CI 77.1% to 89.1%) and a specificity of 76.5% (95% CI 71.8% to 80.8%).
Definitions of adult asthma using health administrative data are sensitive and specific for identifying adults with asthma. Using these definitions, cohorts of adults with asthma for ongoing population-based surveillance and research can be developed.
L’asthme impose un lourd et coûteux fardeau aux individus et à la société. Un programme de surveillance et de recherche basé dans la population à partir des données administratives sanitaires permettrait de mesurer et d’étudier le fardeau de l’asthme. Or, il faut d’abord confirmer la validité des diagnostics d’asthme dans les bases de données administratives sanitaires.
Évaluer l’exactitude des données administratives sanitaires provinciales basées dans la population pour recenser les patients adultes asthmatiques dans le cadre d’un jarogramme continu de surveillance et de recherche.
Des patients suivis en médecine générale sélectionnés au hasard ont été catégorisés en quatre classes, selon leurs diagnostics antérieurs : asthme, maladie pulmonaire obstructive chronique, maladie respiratoire connexe et maladie non asthmatique. Dans chaque province, dix dossiers de chaque catégorie ont été sélectionnés au hasard, résumés et passés en revue par un comité d’experts à l’aveugle qui a confirmé s’il s’agissait d’asthme ou non. Ces diagnostics de référence standard ont ensuite été reliés aux dossiers provinciaux des patients et comparés aux algorithmes administratifs sanitaires conçus pour reconnaître l’asthme. Les analyses ont été conduites à l’aide de méthodes d’évaluation des tests diagnostiques.
En tout, 518 dossiers, dont 160 cas d’asthme, ont été passés en revue. L’algorithme de deux consultations ou plus en soins ambulatoires et/ou d’une hospitalisation ou plus pour asthme sur une période de deux ans offrait une sensibilité de 83,8 % (IC à 95 %, 77,1 % à 89,1 %) et une spécificité de 76,5 % (IC à 95 %, 71,8 % à 80,8 %).
Les définitions de l’asthme adulte reposant sur les données administratives sanitaires sont sensibles et spécifiques pour le recensement des cas d’asthme adulte. À l’aide de ces définitions, il est possible de créer des cohortes d’adultes asthmatiques pour un programme continu de surveillance et de recherche basé dans la population.
Asthma is a common chronic condition that imposes a heavy and expensive burden on individuals, populations and health care systems. Previous population-based studies evaluating the burden of asthma have depended on cross-sectional surveys and cohort studies in specific populations (1). Although these sources of data are useful, they are only able to provide an estimate at one point in time, and are often unable to provide comparison estimates for smaller regional areas or patient groups. A prospective clinical disease registry of all individuals with asthma would be ideal; however, the resources and costs required to operate such a registry would be prohibitive. Health administrative (HA) databases provide a significantly less expensive, practical, and unbiased method of measuring and surveying asthma in a population. In regions with a universal, single-payer health care system such as the province of Ontario, where health care workers essentially submit standardized data to HA databases, they can be particularly useful.
The Ontario Asthma Surveillance Information System was created to initiate and maintain a population-based longitudinal surveillance system for asthma based on HA data. Such a program, however, cannot succeed without first confirming that the HA identification of asthma is valid. In Ontario, physicians provide diagnostic data for which there are no incentives associated with accuracy (2). HA definitions of asthma in children have been previously validated; however, due to differences in presentation, diagnosis and treatment of asthma, and the presence of more and different comorbidity in adults, it cannot be assumed that these definitions also apply to adults (3,4). Therefore, to find a valid HA definition to identify physician-diagnosed asthma in adults, a case verification study was conducted (5).
Patients with and without asthma were randomly selected from primary care physician (PCP) practices that were randomly selected from all PCP practices across Ontario – a Canadian province with a population of approximately 12 million. The sampling frame included all PCPs identified in the Canadian Medical Directory Database who had treated 30 or more asthma and 30 or more chronic obstructive pulmonary disease (COPD) patients in 2003. A letter was mailed and a follow-up telephone call made to the selected PCPs, inviting them to participate in the study. PCPs were excluded if they did not have an office-based practice, did not have the time or space to accommodate a chart abstractor working in their office or did not use an electronic billing system. If a PCP did not respond, refused or was excluded from participation, a replacement PCP was randomly selected and invited.
In each recruited PCP practice, the electronic billing system was used to create a sampling frame that included all the charts of patients 19 to 80 years of age who were currently residing in Ontario and who were seen by the PCP in the previous five years. Population-based samples of individuals are often used to validate diseases in the population; however, to evaluate the ability of HA data to distinguish between asthma and COPD, and other related respiratory conditions, a more conservative cohort that included higher proportions of these patients than was present in the general population was used (3,6). Therefore, the study population consisted of equal numbers of patients who had asthma, COPD, asthma-related conditions and other conditions according to the their PCP’s electronic billing system. Patients with asthma were included to ensure that the HA definition could truly identify individuals with asthma. Patients with asthma-related conditions and COPD were included to ensure that the algorithm could distinguish between asthma and clinically similar conditions, and because patients with occult clinically significant asthma in the population were likely to be found in these groups. Patients not specifically identified as having ‘other’ conditions not included in any of the other categories were included as respiratory healthy controls. To simplify subject selection and ensure that these patients did not have respiratory disease, this final category was limited to patients with hypertension or musculoskeletal problems (Table 1). Of note, it was possible for an individual to be eligible for inclusion in an asthma and a nonasthma category. These patients were included in the most recent category that they were identified to be in by their PCP.
To obtain a study population with equal numbers of asthma patients, COPD, asthma-related and ‘other’ diagnoses, all of the patients in each PCP practice were subdivided into these four categories. Ten charts were then randomly selected from each of the four categories and abstracted by a trained abstractor. The patient’s entire chart was reviewed and a standardized abstraction form completed. Also abstracted was the unique health insurance number given to all individuals in Ontario. On completion of the abstractions, the abstracted data were reviewed for completeness.
Based on earlier studies validating asthma in HA databases and expected sensitivities and specificities (3,7), an approximate sample size of 150 individuals with asthma was determined to be required to obtain 95% CIs with a precision of plus or minus 12%.
Two respirologists, who were blinded to the diagnostic category but not to the objectives of the study, independently reviewed all of the abstracted patient charts from all four categories to identify the cases as asthma or nonasthma. The identification of an asthma diagnosis was determined according to predefined criteria based on the Canadian Asthma Consensus Guidelines (8). Cases in which there was disagreement were discussed and an attempt to reach consensus was made. If consensus could not be reached, a third respirologist was consulted to be the ‘tie-breaker’. Without pulmonary function tests being performed on every patient, this reference standard may not be considered the most rigorous available; however, it surpasses the standard by which the majority of patients with asthma are diagnosed (ie, by a lone PCP) and, therefore, was considered acceptable (9). It was believed to be particularly suitable because it was being used to validate an HA definition of physician-diagnosed asthma for use in the general population where most asthma is diagnosed by PCPs (10). Of note, to facilitate analysis, reviewers were asked to ‘force’ patients into asthma or nonasthma groups although, in some situations, it may have been appropriate for them to have been in both.
Reference standard diagnoses of physician-diagnosed asthma were linked to patients’ HA records via their unique health card number. Using fields from two HA databases, predefined HA definitions (eg, two ambulatory care visits and/or one hospitalization for asthma) that were hypothesized to accurately identify physician-diagnosed asthma were constructed and tested against this reference standard.
The Ontario Health Insurance Plan (OHIP) physician services claims database, which contains information on out-patient claims for all Ontario residents (including claims for physician visits, laboratory tests and diagnostic imaging), was used. Physicians are reimbursed through OHIP by submitting claims for medical services provided. A disease code is provided as part of a claim. The inpatient database used was the Canadian Institute for Health Information discharge abstract database, which contains clinical administrative data relating to the hospital services provided to patients by all hospital facilities in Ontario. This includes diagnostic codes for each hospital visit from the International Classification of Diseases – 9th or 10th Revision, depending on the year. Before 2002, it was possible to have up to 16 diagnostic codes; as of 2002, it has been possible to enter 25 diagnostic codes. If any of these diagnostic codes was for asthma, regardless of what the other codes were for, the hospitalization was used toward the HA definition. Collection of hospital discharge abstract data for submission to Canadian Institute for Health Information is mandatory for all hospitals in Canada.
Demographic characteristics and clinical profiles were compared between the asthma and nonasthma groups using the Student’s two-tailed t test for continuous variables and the χ2 test for categorical variables. Agreement between panel reviewers regarding asthma diagnosis was analyzed using the kappa statistic. The accuracy of the predefined HA definitions compared with the reference standard diagnoses were determined using the concepts of diagnostic test evaluation. Ninety-five per cent CIs for the test characteristics, where appropriate, were also calculated using the binomial distribution. Recognizing that PCPs could have had pertinent patient information that was not documented in the patient chart, and that would not have been available for the expert panel to base their reference standard diagnosis on, the same analysis was repeated using the PCP’s chart abstract as the reference standard. In this case, if any mention of asthma was made in the chart or if a patient received treatment that was consistent with asthma treatment (but did not necessarily have a formal diagnosis of asthma), they were considered to have asthma as per this reference standard. All statistical analyses were performed using SAS version 9.1 (SAS Institute Inc, USA).
Ethics committee approval was obtained through The Hospital for Sick Children (Toronto, Ontario).
A total of 5821 PCPs saw 30 or more asthma and 30 or more COPD patients in 2003. Of these, 81 PCPs were contacted and invited to participate in the study. Forty responded – a crude response rate of 49.3%. Of those who responded, 22 (55%) chose not to participate and five were deemed to be ineligible because they did not have an office-based practice or did not have space for a chart abstractor to work. This left 13 PCPs who agreed and were eligible to participate in the study. From the 13 participating clinics, data from the charts of 518 patients were abstracted.
The abstracted charts were reviewed by the expert panel to determine whether the patients had asthma. The overall agreement between reviewers expressed as a kappa and controlling for site was 0.79 (95% CI 0.73 to 0.85) (Table 2). Of the 518 charts reviewed, there were 56 with disagreements that were resolved by a tie-breaker.
According to the reference standard panel diagnosis, 160 patients (31%) in the study population had physician-diagnosed asthma. Of note, more than 20% of these patients were not originally diagnosed with asthma by their PCP. The patients for whom the PCP and expert panel disagreed did not significantly differ from patients for whom the PCP and expert panel agreed in terms of age, sex, site of PCP practice or previous history of smoking. Characteristics of the asthma and nonasthma patients are summarized in Table 3.
Tables 4 and and55 summarize the diagnostic test characteristics of the predefined HA definitions compared with the reference standard panel diagnosis and PCP chart diagnosis, respectively. According to the PCP diagnosis, 196 (37.8%) of the 518 patients had physician-diagnosed asthma.
For surveillance purposes, if one must trade off sensitivity for specificity or vice versa, a more sensitive definition of disease is more desirable because it ensures that fewer affected individuals are missed (11). In the present case-verification study of HA definitions to identify physician-diagnosed asthma for surveillance purposes, two algorithms were shown to have both higher sensitivity and specificity when either an expert panel diagnosis or PCP designation of asthma was used as the reference standard. A second consideration when choosing an HA definition is the time period needed to satisfy it, because one that requires less time to identify cases will be more responsive. Given our results and these considerations, we chose the algorithm of two or more ambulatory claims, and/or one or more hospital claim(s) for asthma in two years for the Ontario Asthma Surveillance Information System surveillance protocol.
Once an HA definition that meets one’s needs has been chosen, it can then be used to construct a cohort of all individuals with physician-diagnosed asthma. This can, in turn, be used for many purposes (eg, to determine the prevalence of physician-diagnosed asthma, the health services used by individuals with physician-diagnosed asthma and the direct health care cost of physician-diagnosed asthma care). Such information is invaluable to health policy makers, stakeholders, clinicians and members of the public wanting to understand the impact of asthma on the population.
Some of the HA definitions were noted to have a relatively low positive predictive value (PPV). This was because the HA definition identified individuals as having physician-diagnosed asthma, but the interpretation of the abstracted chart by the expert panel and/or the PCP chart did not. The main reason we hypothesize these individuals were not identified as having asthma was because full information regarding their respiratory status was not always recorded in their chart. There are several reasons why this may have been the case. For example, an individual may have had another physician (not their PCP) following their asthma or their asthma may have been diagnosed before becoming a patient of their PCP. In support of our hypothesis, further review of the histories of the 84 individuals who were found to be ‘falsely’ identified by our surveillance HA algorithm of two or more ambulatory claims and/or one hospitalization in two years, showed that 59 (75%) of them saw two or more doctors for their asthma and 17 (20%) had not made any asthma claims in the previous five years (data not shown). In support of some of these ‘false-positive’ cases actually being true positive cases, we found that nine (11%) of these individuals had been hospitalized for asthma and that 66 (79%) had three or more ambulatory care visits for asthma (data not shown).
Case verification using chart abstraction is a common study design used in health services research to validate case definitions used in HA databases. Previously, our group used this approach to validate a single ambulatory care claim for asthma in children (3), with an overall observed sensitivity of 91.4% and specificity of 82.9%. In the province of Quebec, a documented diagnosis of asthma in a PCP chart was used as a gold standard to validate asthma diagnosis in the provincial medical services databases. In individuals seen at one academic family practice clinic, two or more primary care outpatient claims for asthma within one year was found to have a PPV of 0.78 in younger adults (16 to 44 years of age) and a PPV of 0.68 in older adults (45 to 80 years of age) (12). In Manitoba, physician claims in the Manitoba Population Health Research Data Repository were compared with survey data regarding the prevalence of asthma and the agreement was found to be moderate (13). Our study expands on these previous studies by validating physician-diagnosed asthma in a wide variety of primary care practices in Ontario. Because individuals in Ontario cannot see a specialist without a referral from a PCP, we are confident that the present study considers and is generalizable to most, if not all, physician-diagnosed asthma in the province.
In addition to validating HA definitions against an expert panel reference standard, they were also validated against the PCP chart. The results differed very little, which further confirms the test characteristics the HA studied as well as the validity of using two or more ambulatory claims and/or one or more hospital claim for asthma in two years to build a surveillance cohort.
The strengths of the present study are its generalizability, its ability to validate an HA algorithm against two clinically relevant standards including one based on expert panel opinion, and the inclusion of comparison groups of patients with diseases that are similar to asthma.
The main limitation of the present study was that, because a true population sample was not used, the estimates of test characteristics were only approximations of the true population test characteristics. There are a few reasons why one may suspect some of the measurements to be higher or lower than those found. For example, using a study sample population heavily weighted toward conditions that resembled asthma may have led to an underestimate of the test characteristics. In addition, the expert panel that forced eligible patients who were in both the asthma and nonasthma categories in the original patient selection, into only one of these designations may have also led to an underestimate of the test characteristics. Alternatively, identifying most patients with asthma, COPD and other respiratory conditions (but not all) through patient records in which OHIP billing claim diagnoses used in the HA definition also came from, may have overestimated the test characteristics. Thus, compromises were made to validate an HA definition that could distinguish between asthma and other common similar respiratory conditions, and that was reflective of real-life clinical practice. Such compromises, which are often used in studies evaluating diagnostic tests, mean that the results are only estimates of the true test properties and they do not change the relative accuracy of the HA definitions studied (3,6).
Validated methods can be used to identify individuals with asthma in HA databases with excellent sensitivity and specificity (8). Future directions include further validating these HA definitions using HA data from other Canadian provinces and building an asthma cohort that can be used to describe and study asthma in the population. This can then be used to generate evidence on which to base effective asthma prevention and management strategies that lessen the burden of physician-diagnosed asthma on individuals and society.
Funding for this project was made available through the Ontario Ministry of Health and Long-Term Care (MOHLTC). Dr Gershon is supported by a research fellowship from the Canadian Institutes of Health Research, the Institute of Population and Public Health and The Public Health Agency of Canada. Dr To is supported by The Dales Award in Medical Research from the University of Toronto (Toronto, Ontario). This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the MOHLTC.
DISCLAIMER: The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.