|Home | About | Journals | Submit | Contact Us | Français|
To test the hypothesis that an association exists between process and outcome measures of the quality of acute asthma care provided to children in the emergency department.
Investigators at 14 US sites prospectively enrolled consecutive children 2 to 17 years of age presenting to the emergency department with acute asthma. In models adjusted for variables commonly associated with the quality of acute asthma care, we measured the association between 7 measures of concordance with national asthma guideline-recommended processes and 2 outcomes. Specifically, we modeled the association between 5 receipt/nonreceipt process measures and successful discharge and the association between 2 timeliness measures and admission.
In this cohort of 1426 patients, 62% were discharged without relapse or ongoing symptoms (successful discharge), 15% were discharged with relapse or ongoing symptoms, and 24% were admitted. The composite score for receipt of all 5 receipt/nonreceipt process measures was 84%, and for timeliness measures, 57% receive a timely corticosteroid and 92% a timely β-agonist. Our adjusted models showed no association between process and outcome measures, with 1 exception: timely β-agonist administration was associated with admission, likely reflecting confounding by severity rather than a true process-outcome association.
We found no clinically significant association between process and outcome quality measures in the delivery of asthma-related care to children in a multicenter study. Although the quality of emergency department care does not predict successful discharge, other factors, such as outpatient care, may better predict outcomes.
Studies of the association between process and outcome measures of the quality of acute asthma care for children have been mixed. These studies are limited by small, single-institution settings or by examining the association at the aggregate level.
This first multicenter analysis of the process-outcome association in acute asthma care for children revealed no association. Because the validity of process measures depends on association with outcomes, further study is needed before implementing existing process measures as performance metrics.
Asthma is the most common chronic illness in children, affecting 9% of US children, and accounting for 593000 emergency department (ED) visits and 155000 hospitalizations annually.1 Evidence-based recommendations for acute asthma management developed by the National Institutes of Health (NIH)2 have been shown to improve acute asthma care.3–5 The quality of care for children with acute asthma, as measured by guideline concordance, varies among ED providers.5–7 For adults, better guideline concordance of ED processes of care has been shown to reduce hospital admissions by 46% in a multicenter study8; however, little is known about how guideline concordance affects acute asthma outcomes in children. Studies of the association between process and outcome measures in ED asthma care for children are either small, single-institution studies or do not examine the association at the patient level.3,4,9,10 One factor limiting these studies is the paucity of quality measures for ED asthma care. We know of only 2 efforts that define process and outcome measures specific to assessment of the ED care for children with acute asthma: an expert panel11 and the Multicenter Airway Research Collaboration (MARC).12,13 By using measures and data from the MARC studies, our objective was to model the association between process and outcome measures in a multicenter ED cohort of children with acute asthma.
Our primary hypothesis was that, among children treated in the ED for acute asthma, the guideline concordance of care is predictive of discharge from ED without subsequent relapse event or ongoing symptoms. Our secondary hypothesis was that, among children treated in the ED after an acute asthma-related encounter, time-sensitive process measures of ED asthma care quality are associated with admission.
We analyzed data from prospective cohort studies performed during fall 1997, spring 1998, and fall 2000, as part of MARC, a division of the Emergency Medicine Network (EMNet). By using a standardized protocol, investigators at 14 EDs in 11 US states provided 24-hour per day coverage for a median of 2 weeks to enroll consecutive patients presenting to the ED for acute asthma. Inclusion criteria were physician diagnosis of acute asthma, age 2 to 17 years, and informed consent of the parent or guardian. The study cohort included children who had no acute asthma symptoms reported on study instruments. Patients were managed at the discretion of the treating physician. Our data collection methods are described elsewhere.12
Eligible subjects underwent a structured interview in the ED that assessed demographic characteristics, asthma history, and details of the current asthma exacerbation. Data on ED management, discharge prescriptions, and disposition were obtained by chart review. Follow-up data were collected by telephone interview 2 weeks later. All forms were reviewed by site investigators before submission to the EMNet Coordinating Center, where they underwent further review by trained personnel and then double data entry.
We used explicit process assessment methods, applying a priori criteria to determine whether the observed outcomes of care are improved when the processes of care are more guideline-concordant or more timely.14 Employing methods similar to those in an EMNet multicenter study of adults with acute asthma,8 we derived 7 process measures of asthma care quality from the NIH asthma guidelines2 (Table 1).
The 5 level A process measures, the same 5 used in the adult EMNet study,8 assess receipt of inhaled β-agonists, inhaled anticholinergics, and systemic corticosteroids; nontreatment with methylxanthines; and prescription of oral corticosteroids at discharge. The 2 level B processes both involve timeliness of medication administration: receipt of a corticosteroid treatment and an inhaled β-agonist in the first hour in the ED. The timeliness measures in the adult EMNet study used different cut-points: 75 minutes for corticosteroids and 15 minutes for β-agonists. We selected the 1-hour threshold for the first β-agonist based on 2 factors: (1) the data collection instrument measured how many β-agonist treatments were given in the first hour rather than the time of administration, and (2) the NIH’s guideline recommendation of “up to 3 doses in [the] first hour.”2 We selected the 1-hour threshold for corticosteroid administration based on a Cochrane review recommending dosing of systemic corticosteroids within the first hour.15
For each process and outcome measure, we defined an eligible population by adapting the eligible population definitions used in the adult EMNet study, which derived them from the NIH guidelines8 (Table 1). For 2 measures, nontreatment with methylxanthines and treatment with a β-agonist in the first hour, the eligible population definitions were identical to those in the adult EMNet study. For 3 measures (treatment with inhaled anticholinergic, treatment with systemic corticosteroids, and prescription of oral corticosteroids at discharge), the primary difference in eligible population definitions was our study’s use of the pulmonary index,16 rather than peak expiratory flow values, to define acute asthma severity. For the measure treatment with a β-agonist, our study’s eligible population definition required acute symptoms (a nonzero pulmonary index), a more restrictive definition than the adult EMNet study’s inclusion of all diagnosed with acute asthma. For the measure treatment with systemic corticosteroids in the first hour, the current study’s eligible population definition was again more restrictive, including only those who received a systemic corticosteroid rather than all meeting criteria to receive a systemic corticosteroid in the ED.
Like the adult EMNet study, our study summarized the 5 level A evidence-based process measures in a composite concordance score, calculated as the sum of guideline-concordant care divided by the patient’s total number of eligible opportunities. The level B measures are not combined into a composite score because there are only 2.
Regarding outcome measures, ED disposition (admission, discharge) was collected by ED chart review. Relapse and ongoing symptoms measures were assessed during the telephone interview conducted 2 weeks after ED discharge. “Relapse” was defined as any urgent visit to an ED or clinic for worsening of asthma during the 2-week follow-up period. An “ongoing symptoms” classification was assigned to patients who reported “severe symptoms” during the 24 hours preceding the telephone interview, who endorsed having asthma symptoms “most of the time” or “severe” discomfort and distress as a result of their asthma, or who stated that their asthma was “about the same” or worse than at the time of their ED presentation.
We selected the outcome “successful discharge”, defined as “discharge from ED without subsequent relapse event or ongoing symptoms” as our primary outcome because it applies to the entire population, it is not as influenced by physician admitting behavior, it has been found to more adequately describe the outcomes of acute asthma care, and it has a clear directionality for defining good quality.17,18 For the secondary hypothesis, we chose “admission” as the primary outcome because it is the most proximate of the candidate outcomes to each timeliness variable, and the relationship between timeliness of care and admission has greater face-validity than, for example, a relationship between timely initial β-agonist administration and a relapse or symptoms 2 weeks later.
Other patient characteristics were included based on variables found in previous studies to have an association with the quality of acute asthma care delivered.12,13,19,20 Demographic information included age, gender, race/ethnicity, parental education (high school graduate, yes/no), median household income, insurance status, and presence of a primary care provider.
A crucial element in retrospective analyses of quality of care is use of appropriate case-mix adjusters; without these, observed differences related to patient characteristics may be falsely attributed to variations in quality.21 To severity-adjust our models, we included both chronic and acute asthma-related factors. Indicators of chronic asthma severity included health care utilization factors, such as number of ED and urgent care visits in the past year, and other known risk factors for ED utilization, including exposure to smoking or pets, history of previous hospitalization or intubation for asthma, comorbid conditions, and presence of an asthma action plan.
Acute asthma severity measures included the duration of symptoms, number of inhaled β-agonist treatments within 6 hours of ED arrival, and the pulmonary index score, calculated according to 4 components, each scored 0 to 3: respiratory rate, accessory muscle use, wheezing, and inspiratory-expiratory ratio.16 Although the models adjusted for all 3 of these acute asthma severity measures, the pulmonary index was selected for categorizing patients’ acute severity because it contains 4 elements recommended by the NIH guidelines for acute exacerbation severity assessment, whereas the other 2 single-item measures are not specifically mentioned with regard to acute severity assessment.2 Because the pulmonary index lacks validated cut-points defining severity categories, we determined cut-points by examining the proportion admitted among patients with each value of the pulmonary index. The resulting histogram (Fig 1) shows natural cut-points, and these were used to define mild (pulmonary index 0–4), moderate (5–7), and severe (8–12) exacerbations. The histogram shows that some patients in the mild category were hospitalized; this may reflect other factors (comorbid diagnoses, variations in admission practice patterns), as well as the imperfect predictive ability of all acute asthma severity scores.2
All analyses were performed by using Stata 10.0 (Stata Corp, College Station, TX). Data are presented as proportions (with 95% confidence intervals [CIs]), means (with SD), or medians (with interquartile range). Imputed values were used to calculate the pulmonary index score when 1 of the 4 physical exam findings was missing. Patients missing more than 1 of the parameters (15%) were not assigned a pulmonary index score and thus were excluded from eligible populations defined by acute severity. Of the 1302 with a pulmonary index score, 64% had all parameters available and 36% had 1 value imputed.
For each patient, we calculated a composite guideline concordance measure score. In calculating the score, we assigned 1 numerator point for each level A guideline-concordant measure in the desired direction. The denominator was the total number of eligible opportunities.
All associations were examined by using the χ2 test, Fisher’s exact test, Student’s t test, and Wilcoxon rank sum test, as appropriate. Age, gender, and race were included in multivariate logistic regression models because of their potential clinical significance. Other variables associated with the outcome of interest at P < .10 in univariate analysis were evaluated for inclusion in multivariate logistic regression models. The multivariate models tested the following associations, corresponding to the 2 study hypotheses: (1) the association between composite level A measures and successful discharge; and (2) the association between individual level B measures and admission.
Unadjusted models used a generalized estimating equation accounting for clustering by site. Adjusted models used a generalized estimating equation accounting for clustering by site, adjusted for age, gender, race/ethnicity; primary care provider; use of asthma medications other than β-agonists, inhaled or systemic steroids, cromolyn, or nedocromil; exposure to cigarette smoke; hospital admission during the past year; corticosteroid use within 4 weeks of ED arrival; duration of symptoms; number of inhaled β-agonists treatments within 6 hours of ED arrival; severity of asthma symptoms during 24 hours before ED arrival; oxygen saturation; pulmonary index score; concomitant medical conditions; number of ED visits during past year; and number of urgent clinic visits during past year. All odds ratios are presented with 95% CIs. All P values are 2-tailed, with P < .05 considered statistically significant.
Among this cohort of 1426 patients presenting to the ED with acute asthma, the median age was 7 years, 40% were girls, 52% African American, 19% Hispanic, and 25% white. Overall, 91% of children had a primary care provider, and 38% had private insurance (Table 2). With regard to asthma history, 57% had a previous hospitalization for asthma, 22% had taken systemic corticosteroids in the previous 4 weeks, and patients had a median of 2 ED visits in the past year. With regard to acute symptoms, 63% had duration of acute symptoms of <1 day, and patients had taken a median of 4 inhaled β-agonist treatments within 6 hours of ED arrival. When acute severity was categorized by using the pulmonary index cut-points, 58% had mild severity, 33% moderate, and 9% severe. The proportion admitted for each severity group was 10%, 31%, and 65%, respectively.
With regard to outcome measures, 62% had successful discharge, 15% had either relapse or ongoing symptoms within 2 weeks, and 24% were admitted (Table 2). Summary statistics for quality measures show that, of level A measures, values for guideline concordance ranged from 63% (for ED use of inhaled anticholinergics) to 99% (for both ED use of inhaled β-agonists and ED nonuse of methylxanthines; Table 3). The composite score for concordance with all 5 level A NIH process measures was 84%. For level B measures, 57% received a systemic corticosteroid in the first hour and 92% received an inhaled β-agonist treatment in the first hour (Table 3). Restricting timeliness of corticosteroid treatment to children recommended to receive ED corticosteroids, 358 of 619 (58%; 95% CI: 54%–62%) received corticosteroid treatment in the first hour, with 27 missing the time of corticosteroid treatment. Restricting the timeliness of inhaled β-agonist treatment to children recommended to receive this therapy, 1093 of 1193 (92%; 95% CI: 90%–93%) received inhaled β-agonist treatment in the first hour, with 12 missing the number of β-agonist treatments in the first hour.
Unadjusted models of the process-outcome measure association revealed 3 significant findings out of 15 comparisons: ED administration of systemic corticosteroids was associated with lower rates of successful discharge, and both ED administration of systemic corticosteroids and timely ED β-agonist administration were associated with higher admission rates (Table 4). In adjusted models, concordance with individual and aggregated level A process measures was not associated with successful discharge (primary hypothesis). With regard to the secondary hypothesis, although timely corticosteroid administration was not associated with admission, timely albuterol admission was associated with a 4.4-fold increased odds of admission (Table 5).
In this study of 1426 patients in 14 EDs, we found, in general, no association between either process or timeliness measures and outcome measures for ED management of acute asthma in children. This differs from the findings of the previously cited adult EMNet study,8 as well as 2 of the 4 previous studies of this issue in children.3,4,9,10 The process measures used in the adult study were the same 5 level A recommendations as in our study; the study revealed that 100% guideline concordance (in 12 level A and B measures) was associated with a 46% lower admission rate.8 Among studies including children, 3 prepost studies, all single-site, evaluated the impact of use of an asthma guideline on the quality of ED asthma care. A US study revealed mixed findings in outcome measures: decreased admissions but no change in revisits.4 An Australian study revealed decreased admissions and revisits.3 An all-ages Canadian study revealed no change in admissions and revisits.10 A fourth, multiinstitutional, Canadian study revealed that the presence of an asthma order sheet, but not the presence of an asthma guideline, was associated with a decreased revisit rate.9 Our negative study is the first multicenter study of the association between guideline-concordant ED care and asthma outcomes in children and the first to use patient-level process measures, rather than the institution-level presence of a guideline, as the predictor of outcomes.
The sole significant process-outcome association found in adjusted models was that between timely albuterol administration and the risk of admission. This finding likely represents insufficient adjustment for confounding by severity because patients with more severe asthma are both more likely to have timely albuterol and more likely to be hospitalized, as also reported in previous MARC studies.19,20
The contrast between our negative study and the positive process-outcome association found by its closest counterpart, the adult EMNet study,8 may reflect 2 confounders.21 First, the 2 studies differed in how they adjusted for patient-level factors, including severity. In our study, we used a 12-point acute severity scale, whereas the authors of the adult EMNet study used the peak expiratory flow absolute value, resulting in a higher percentage of patients categorized as severe in the EMNet study (38% vs 9% in our study) despite a lower admission rate (18% vs 24% in our study, a rate comparable to nationally representative findings1). Because the eligible population definitions for 2 of the 5 level A process measures included acute asthma severity, this may have biased our findings toward the null. The severity assessment used in the adult EMNet study, peak expiratory flow, is not reliably measured among children.22
Second, the 2 studies differed somewhat in measure definition. As an example, the adult EMNet study’s time cutoff for initial β-agonist was 15 minutes compared with 60 minutes in our study (28% vs 92% concordant, respectively). Having fewer patients with nonconcordant care may have limited our study’s power to detect a difference in quality of care.
The other issue raised by our negative study is one of perspective. For a disease process such as asthma, where outcome measures are either related to subjective physician behavior (admission), are not proximate to ED care (relapse, ongoing symptoms), or are exceedingly rare (mortality), our negative study raises the question of the value of using outcome measures to validate or justify the use of process measures. Perhaps process measures are the more sensitive indicator of real variations in quality and can be used without showing a link to outcome measures.21
We regard this argument with caution in that it disregards a central tenet in the field of quality measures: validity of process measures is demonstrated when variations in the attribute they measure lead to differences in outcome and vice versa.14 The importance of careful examination of the process-outcome measure association is illustrated by the adverse consequences of the Joint Commission’s “time to first antibiotic dose” process measure for ED patients with community-acquired pneumonia. Introduction of this performance measure resulted in overuse of antibiotics and no change in the relevant outcome, mortality.23 Thus, we conclude from our negative study that further exploration of the process-outcome link in the quality of ED asthma care is needed, as well as further consideration of appropriate process and outcome measures, before implementing process measures as performance metrics.
Our study has some potential limitations. First, the study derived some measures from chart review, so data quality depended on the accuracy of clinical charting. However, previous studies revealed that the rates of ED assessments and treatments for asthma by retrospective chart abstraction were similar to those achieved by direct observation, with κ coefficients of 0.6 to 0.9.24 Second, we studied only the initial processes of asthma care; several studies revealed that data from the time of ED disposition, rather than from arrival, is more predictive of outcomes.18,20,25 Third, our secondary hypothesis used the outcome admission, a heterogeneous decision based on the clinical opinion of individual providers. The subjectiveness of this secondary outcome is why we selected the composite outcome successful discharge as our primary outcome.18 Fourth, the use of admission to define pulmonary index cut-points may have biased our findings because admission was also a study outcome. As noted above, when compared with the adult EMNet study, the current study’s methods yielded a smaller proportion of exacerbations categorized as severe. It is not clear how this would bias our findings. Fifth, the study’s use of data from noncontinuous time periods may have introduced spectrum bias as the precipitating factors and incidence of acute asthma exacerbations change seasonally, although it is not clear how this would bias our findings. Sixth, this study was a secondary analysis of existing data and it is possible that the available sample size is not sufficient to detect the observed differences in the primary and secondary outcomes (Type II error). Finally, the EDs that compose the study sample were predominantly urban, academic EDs, which may make our results less generalizable to rural or suburban, nonacademic EDs.
We report no clinically significant association between process and outcome quality measures, as defined, in the delivery of asthma-related care to children in a multicenter study of academic EDs. Further exploration of the process-outcome link in the quality of ED asthma care is needed before implementing process measures as performance metrics.
Dr Sills was supported by the Riggs Family/Health Policy grant from the American College of Emergency Physicians, grant 1 R03 HS016418-01A1 from the Agency for Healthcare Research and Quality, the Social Behavioral Research grant from the American Lung Association, and by the Children’s Hospital Research Institute; Dr Ginde was supported by NIH grant KL2 RR025779. The Multicenter Airway Research Collaboration was supported by NIH grant HL-03533 and HL-63253, and by an unrestricted grant from GlaxoSmithKline (Research Triangle Park, NC).
All 4 authors, Drs Sills, Ginde, Clark, and Camargo, meet all 3 of the authorship criteria as follows: (1) substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content; and (3) final approval of the version to be published.
FINANCIAL DISCLOSURE: Dr Camargo has received financial support from a variety of groups for participation in conferences, consulting, and investigator-initiated medical research. Recent industry sponsors with an interest in asthma were AstraZeneca, Dey, GlaxoSmithKline, Merck, Novartis, and Sanofi-Aventis. Drs Sills, Ginde, and Clark have indicated they have no financial relationships relevant to this article to disclose.
Funded by the National Institutes of Health (NIH).