|Home | About | Journals | Submit | Contact Us | Français|
With the use of Centers for Disease Control and Prevention (CDC) immunization recommendations as the gold standard, our objectives were to measure the accuracy (“is this child up-to-date on immunizations?”) and usefulness (“is this child due for catch-up immunizations?”) of the Healthcare Effectiveness Data and Information Set (HEDIS) childhood immunization measures.
For children aged 24 to 35 months from the 2009 National Immunization Survey, we assessed the accuracy and usefulness of the HEDIS childhood immunization measures for 6 individual immunizations and a composite.
A total of 12096 children met all inclusion criteria and composed the study sample. The HEDIS measures had >90% accuracy when compared with the CDC gold standard for each of the 6 immunizations (range, 94.3%–99.7%) and the composite (93.8%). The HEDIS measure was least accurate for hepatitis B and pneumococcal conjugate immunizations. The proportion of children for which the HEDIS measure yielded a nonuseful result (ie, an incorrect answer to the question, “is this child due for catch-up immunization?”) ranged from 0.33% (varicella) to 5.96% (pneumococcal conjugate). The most important predictor of HEDIS measure accuracy and usefulness was the CDC-recommended number of immunizations due at age 2 years; children with zero or all immunizations due were the most likely to be correctly classified.
HEDIS childhood immunization measures are, on the whole, accurate and useful. Certain immunizations (eg, hepatitis B, pneumococcal conjugate) and children (eg, those with a single overdue immunization), however, are more prone to HEDIS misclassification.
Healthcare Effectiveness Data and Information Set (HEDIS) measures are widely used to assess the reliability of pediatric immunization delivery. The extent to which these measures provide accurate (“is this child up-to-date on immunizations?”) and useful (“is this child due for catch-up immunizations?”) information is unclear.
Overall, HEDIS childhood immunization measures are accurate and useful. Users of HEDIS data should be aware, however, that certain immunizations (eg, hepatitis B, pneumococcal conjugate) and children (eg, those with a single overdue immunization) are more prone to HEDIS misclassification.
There is an expanding movement in the United States to systematically measure the quality of health care delivery.1,2 The National Quality Forum, a leader in quality measure appraisal, uses 4 criteria to assess quality measures: importance to measure and report, reliability and validity, usability and relevance, and feasibility to collect.3 The extent to which quality measures score more favorably on these 4 axes theoretically improves the likelihood that their widespread use could lead to improvements in health care quality.
Childhood immunization delivery is a perennial target of quality measurement efforts.4 Childhood immunizations have been deemed the most important pediatric preventive service5 and are clearly important to measure and report. Although integration of practice-based paper records, electronic health records (EHRs), immunization registries, and payer data has been challenging, childhood immunization data have repeatedly been shown to be feasible to collect. Thus, 2 of the 4 National Quality Forum criteria are readily met (importance to measure and report; feasibility). Whether childhood immunization data, as currently measured, are reliable and valid or usable and relevant is less clear.
The National Committee for Quality Assurance has developed childhood immunization quality measures as part of its Healthcare Effectiveness Data and Information Set (HEDIS) program.6,7 HEDIS measures help employers assess the quality of care provided by the health insurance plans they offer; consumers are also occasionally provided with these data. More than 90% of US health plans use HEDIS measures, which allow benchmarking of immunization delivery across health plans. Additionally, federal “meaningful-use” criteria for EHRs include HEDIS childhood immunization measures.8,9
HEDIS childhood immunization measures, however, were not designed to precisely reflect the recommended immunization schedules developed annually by the Advisory Committee for Immunization Practices of the Centers for Disease Control and Prevention (CDC). These CDC schedules are the gold standard to guide immunization delivery for providers, parents, schools, and others.10 The consequences of these inconsistencies on measure reliability/validity (ie, accuracy) and usability/relevance (ie, usefulness) are incompletely understood. By using a nationally representative sample of 2-year-olds, our primary objectives were to (1) calculate the accuracy of the HEDIS immunization measures, compared with CDC recommendations, and (2) determine whether the HEDIS immunization measures provide useful data for providers.
We analyzed data from the 2009 National Immunization Survey (NIS).11 The NIS is conducted annually by the CDC to assess immunization uptake by noninstitutionalized US children aged 19 to 35 months.12 The NIS uses a random-digit–dialing telephone survey to identify households containing children in the target age range and interviews the adult most knowledgeable about the child’s immunizations. With consent of the child’s parent, the NIS also contacts the child’s health care provider(s) to request immunization information from medical records. For the 2009 NIS, household interviews were conducted from January 2009 to February 2010; provider data collection extended through April 2010. NIS incorporates a complex sampling methodology that adjusts for household nonresponse, households with multiple or no telephone lines, and vaccination history nonresponse, yielding data that can be weighted to generate nationally representative estimates.12
We included children residing in the 50 US states or Washington, DC with adequate provider-reported data to assess whether the child received all recommended immunizations, or zero immunizations reported on household interview. Publicly available NIS data categorize participants into 3 age groups: 19 to 23, 24 to 29, and 30 to 35 months. Because our study focused on whether immunizations were up-to-date (UTD) and/or due on the child’s second birthday, we limited our study to children aged 24 to 35 months.
In 2009, the CDC recommended the following immunizations for children in the first 2 years of life: hepatitis B (HepB), rotavirus, diphtheria/tetanus/acellular pertussis (DTaP), Haemophilus influenzae type B (HIB), pneumococcal conjugate (PCV), inactivated poliovirus (IPV), influenza, measles/mumps/rubella (MMR), varicella zoster virus (VZV), and hepatitis A.10 Of these, 7 were included in HEDIS 2009 measures (HepB, DTaP, HIB, PCV, IPV, MMR, and VZV).7 Our study analyzed 6 of these measures, excluding HIB. Production problems led to a global shortage of HIB from December 2007 to September 2009, prompting the CDC to recommend temporary suspension of the booster dose (after age 1 year) of HIB.13,14 Because we were unable to distinguish missed doses from intentionally suspended doses, HIB was excluded from all analyses. Only immunizations given in the first 2 years of life were analyzed.
We compared the performance of the 2009 HEDIS childhood immunization measures7 with the gold standard, CDC-recommended immunization schedule from 200910 (see Supplemental Table 5) for both accuracy and usefulness.
Accuracy was assessed by measuring the proportion of children whose HEDIS-derived UTD status was concordant with their CDC-derived UTD status for the 6 individual immunizations and a composite. The composite measure was evaluated for each child (UTD for all immunizations versus not UTD) with possible concordance of HEDIS and CDC composite results even if individual immunizations were discordant. For the HEDIS measure, a child was UTD if they received the measure-specified number of immunizations on or before their second birthday, regardless of age at administration or dosing intervals. For example, a child receiving 4 doses of PCV by 2 years of age was UTD; a child receiving 3 doses was not. For the CDC measure, a child was considered UTD if, regardless of the number of previously administered immunizations, no catch-up immunizations were recommended either at age 2 years or after. For example, a child receiving 3 doses of PCV at ages 2, 4, and 14 months was UTD (fourth catch-up dose not recommended by CDC); a child receiving 3 doses of PCV at ages 2, 4, and 6 months was not UTD (fourth catch-up dose recommended by CDC). Per CDC guidance, a grace period of 4 days was applied to all minimum intervals and ages for the CDC measure.15 Doses 5 or more days earlier than recommended were not counted; repeat doses were only counted if they were spaced after the invalid dose by the recommended minimum interval. No grace periods were applied to the HEDIS measures, consistent with HEDIS specifications.
We postulated that a childhood immunization measure would be useful to providers if it identified which children required which catch-up immunization(s) based on the CDC recommendations. We therefore calculated the weighted proportion of children due for each immunization and the composite by the HEDIS and CDC standards. Next, we calculated the proportion of children for whom this classification was discordant across the 2 schema, broken down into HEDIS false-positives (HEDIS = due, CDC = not due) and false-negatives (HEDIS = not due, CDC = due), the sum of which represented the overall mismatch rate. In contrast to the accuracy composite, the usefulness composite measure was considered at the immunization level, not the patient level. Therefore, the composite measure was deemed to be mismatched when the HEDIS and CDC measures disagreed on the status of any immunization for a child. For the HEDIS measure, a child was due for immunization on the second birthday if he had had fewer than the measure-specified number of immunizations by that time. By definition, the proportion due for immunizations equaled 1 minus the proportion deemed UTD by the accuracy measure. For the CDC measure, a child was due for immunization if the missing immunization was still recommended for “catch-up” administration at age 2 years and a sufficient interval had passed since administration of the previous dose. For example, a child receiving 3 previous doses of DTaP would be due for immunization by the HEDIS measure. This same child would be due for immunization by the CDC measure only if 6 months had elapsed since the administration of the third dose. In the usefulness analyses, grace periods were applied to the CDC rules only for previous immunizations and not the current immunization. Although the CDC criteria accept previously administered immunizations within the grace period, we felt that most providers would not purposefully offer future immunizations earlier than recommended.
Sampling weights were applied to the data to generate nationally representative estimates. Descriptive analyses were performed by using both unweighted and weighted data. We then calculated the proportion of children UTD according to the HEDIS measure and CDC gold standard for individual immunizations and the composite. We conducted a similar analysis for usefulness and characterized the sources of error when the HEDIS and CDC measures were discordant. Last, we assembled 2 logistic regression models to identify factors associated with the likelihood of a child being accurately assessed by the HEDIS measure as UTD and as due for immunization, compared with the CDC standard. This study of deidentified, publicly available data was approved by the Johns Hopkins University School of Medicine Institutional Review Board.
Of 25241 children for whom household interviews were completed during the 2009 NIS, 12096 met all inclusion criteria and composed the study sample. Children were evenly split by age and gender; half of children were white (Table 1). Interviews were most often with the child’s mother (78%); 15% were conducted in Spanish. Forty-six percent of families comprised 5 or more members; nearly one-third lived below the poverty line. A majority of children had received WIC benefits in their lifetime, and more than 90% were currently insured. A majority of immunization providers reported to an immunization registry and ordered immunizations from a health department.
The proportion of children UTD for each of the 6 immunizations ranged from 78% (PCV) to 91.5% (HepB) according to the HEDIS measures and 79.4% (DTaP) to 90.8% (IPV) according to the CDC gold standard (Table 2). Approximately two-thirds of children (HEDIS, 67%; CDC, 65%) were UTD for the composite measure. The HEDIS measure had >90% accuracy when compared with the CDC gold standard for each of the 6 immunizations (range, 94.3%–99.7%) and the composite (93.8%). The HEDIS measure was most accurate for VZV, IPV, MMR, and DTaP and least accurate for HepB and PCV.
The 2 immunizations most likely to be due at age 2, consistent across both the HEDIS and CDC measures, were DTaP and PCV (Table 3). Approximately one-third of children were due for at least 1 immunization at age 2 (HEDIS, 32.9%; CDC, 34.2%). Across the 6 immunizations, the proportion of children for which the HEDIS measure yielded an incorrect result ranged from 0.33% (VZV) to 5.96% (PCV). The predominant source of mismatch for the 2 immunizations with the highest mismatch rates was different. For HepB, the majority of mismatches derived from HEDIS false-negatives, confirming a finding observed previously by using NIS data.16 This occurred most commonly when the third HepB dose was administered too close to the second dose (minimum interval = 8 weeks) or at too young an age (minimum age = 24 weeks). For PCV, the majority of mismatches derived from HEDIS false-positives (<4 PCV immunizations had been administered, but CDC recommendations were to administer no additional immunizations). Overall, the HEDIS composite result was inaccurate (ie, incorrectly categorizing at least 1 immunization in the composite) for 1 in 9 children (11.7%).
The accuracy and usefulness of the HEDIS immunization measures varied little by patient, household, financial/insurance, or provider attributes (Table 4). The only important predictor of HEDIS measure accuracy and usefulness was the CDC-recommended number of immunizations due at age 2 years. For both accuracy and usefulness, the HEDIS measures performed best at the extremes, when either zero or all 6 immunizations were due. When some, but not all, of the 6 immunizations were due, HEDIS measure accuracy dropped to 86.8% (odds ratio of being correctly classified, compared with zero immunizations due, 0.22 [95% confidence interval 0.16–0.31]) and usefulness to 70.8% (odds ratio, 0.10 [0.08–0.13]). For accuracy, the decrement for the “some but not all immunizations due” category was driven entirely by a drop in performance when only a single immunization was due (Fig 1). For usefulness, the same decrement was driven by drops in performance across the number immunizations due, with a U-shaped distribution.
We used a nationally representative database of child-level, provider-reported immunizations to assess the accuracy and usefulness of the HEDIS childhood immunization measures. Overall, we found the HEDIS measures to be quite accurate. Among the only 2 immunizations with < 98% accuracy, HEDIS measures tended to overclassify children as UTD for HepB and underclassify children as UTD for PCV. These same 2 immunizations were also the 2 most likely to be misclassified with respect to whether a catch-up immunization would be due at age 2. We found the HEDIS measures to be the most accurate when zero or >1 immunization was due and most useful when zero or all 6 immunizations were due.
Systematic measurement of immunization delivery serves a variety of purposes.4,17 First, national, state, and local governments conduct immunization surveillance to identify targets for immunization outreach. Second, payers use immunization delivery data for pay-for-performance (P4P) efforts that reward physicians or organizations for achieving immunization goals. Third, individual providers and practices use immunization measurement to improve the quality of day-to-day delivery of immunizations. Last, individual providers and practices may implement immunization measurement to satisfy meaningful-use requirements for their EHRs. The results of our study have implications for each of these approaches to immunization measurement.
Our data provide reassurance that population-level, HEDIS-based estimates are likely to be quite accurate, contradicting results from 2 smaller studies that used similar methodologies.18,19 Because a substantial fraction of the total misclassification tends to be in opposing directions (HepB overclassified as UTD; PCV underclassified as UTD), population estimates of children UTD for the composite are accurate within 2%, despite an overall HEDIS accuracy of 94%. Additionally, previous research suggests that variability in health plan performance on HEDIS childhood immunization measures may result, in part, from variability in patient-level characteristics (eg, race, poverty, urbanicity) across plans.20,21 Our data suggest that this variability is unlikely to be driven by differences in measure accuracy across these variables. Therefore, observed immunization disparities identified via HEDIS-based immunization surveillance likely reflect true underlying disparities in immunization delivery, and not differential measure accuracy.
Immunization delivery has been a target of P4P efforts.22–28 Payers considering the merits of immunization-related P4P initiatives should be aware that population estimates derived from HEDIS are likely to be quite accurate, whereas estimates at the patient or provider level (in the case of providers with low patient numbers) may or may not be, as has been found for other HEDIS measures.29,30 P4P efforts relying on HEDIS immunization measures should be recognized as specifically incentivizing the timely delivery of immunizations, rather than on achieving an overall UTD population of children. For example, a child that does not present to clinic between ages 6 and 12 months may receive PCV at 2, 4, and 13 months. By CDC criteria, this child, although delayed in receiving immunizations, is fully immunized by age 2. In contrast, HEDIS measures will classify this child as not UTD. Indeed, providers in such a case could have a financial incentive to offer the fourth PCV immunization before age 2, despite the fact that it is not recommended by the CDC, to bring the child into compliance with the HEDIS measure. As a result, more nuanced P4P approaches may be warranted, in particular, at the provider level and potentially involving visit-based (ie, “missed opportunities”) analyses. Such a missed opportunities strategy would assess the proportion of visits at which all due immunizations were delivered, with the CDC recommendations forming the gold standard against which immunization delivery is measured.
For those using HEDIS measures to guide clinical care or for quality improvement purposes, the observed usefulness error rate (1 in 9 children) makes a HEDIS-derived immunization prompt more useful as a screening tool than as definitive decision support. Providers should be especially vigilant for children due for HepB who are not flagged by HEDIS-derived prompts, as well as children flagged by HEDIS-derived prompts for PCV who are not due for catch-up. Although they are technically much more complex to code than the HEDIS measures, direct coding of CDC immunization rules into clinical decision support is a natural next step in the evolution of EHRs and would result in nearly error-free prompting of providers regarding which children are due for which immunizations.31 Nonetheless, federal meaningful-use standards are currently based on the HEDIS immunization measures.8,9 Therefore, practices wishing to meet these standards will still need to measure and report HEDIS immunization measures, even if they use a more refined, CDC-derived decision support tool to guide clinical care.
Our study has several limitations. First, HEDIS measures were not developed for use as point-of-care indicators of whether a child needs catch-up immunizations. Nonetheless, we chose to analyze HEDIS measure usefulness, in addition to accuracy, because this best captures the provider/patient point of view. Second, we analyzed each child only on their second birthday. HEDIS childhood immunization measures are not designed for use at other ages, so their accuracy and usefulness at other ages cannot be assessed. Third, we used CDC-recommended grace periods to determine whether immunizations should be counted. Schools, childcare agencies, and other parties may not accept immunizations that fall within the grace period. Fourth, the NIS data themselves are subject to limitations, including possible residual bias despite adjustments made to account for nonresponse and households without landline telephones.12 Finally, immunization measurement is highly dependent on patient (ie, denominator) selection.17,32–35 HEDIS measures, in particular, examine children who were continuously enrolled in a single health plan during the measurement year.7 Our analyses were independent of health plan enrollment. Results for children meeting other inclusion criteria could be different.
Given the profound health and societal benefits associated with the reliable delivery of childhood immunizations, efforts to measure and improve immunization delivery processes are essential. HEDIS immunization measures have served for years as an important means of comparing immunization delivery across patient populations, providers, and health plans, and will continue to do so. Nonetheless, the HEDIS measures were not designed as a perfect mirror to the CDC recommendations upon which they are based. As a result, users of immunization delivery data must be aware of the strengths and weaknesses of HEDIS-based approaches, until advances in electronic health information technology permit more precise, reliable, and nuanced measurement of immunization delivery.
We thank Drs Tracy King, Lilly Engineer, and Christoph Lehmann for their thoughtful reviews of earlier drafts of this article.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This project was supported, in part, by National Library of Medicine of the National Institutes of Health grant RC1LM010512 to Dr Miller. Funded by the National Institutes of Health (NIH).