|Home | About | Journals | Submit | Contact Us | Français|
The Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA) includes provisions for identifying standardized pediatric care quality measures. These 24 “CHIPRA measures” were designed to be evaluated by using claims data from health insurance plan populations. Such data have limited ability to evaluate population health, especially among uninsured people. The rapid expansion of data from electronic health records (EHRs) may help address this limitation by augmenting claims data in care quality assessments. We outline how to operationalize many of the CHIPRA measures for application in EHR data through a case study of a network of >40 outpatient community health centers in 2009–2010 with a single EHR. We assess the differences seen when applying the original claims-based versus adapted EHR-based specifications, using 2 CHIPRA measures (Chlamydia screening among sexually active female patients; BMI percentile documentation) as examples. Sixteen of the original CHIPRA measures could feasibly be evaluated in this dataset. Three main adaptations were necessary (specifying a visit-based population denominator, calculating some pregnancy-related factors by using EHR data, substituting for medication dispense data). Although it is feasible to adapt many of the CHIPRA measures for use in outpatient EHR data, information is gained and lost depending on how numerators and denominators are specified. We suggest first steps toward application of the CHIPRA measures in uninsured populations, and in EHR data. The results highlight the importance of considering the limitations of the original CHIPRA measures in care quality evaluations.
Several national efforts have developed standardized measures of health care quality for use in quality assessment and reporting. Although most are measures of adults’ care, the Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA), mainly adopted to provide health insurance coverage for an estimated 6 million children,1 includes provisions for identification of an initial core set of recommended pediatric quality measures. The Centers for Medicare and Medicaid Services and the Agency for Healthcare Research and Quality developed this set of measures.
The “CHIPRA core measures” were developed based on pediatric health care quality indicators previously used in public and private health insurance reporting on components of recommended preventive care and treatment of common morbidities.1–5 Measures were included in the initial set based on their validity, feasibility of use by Medicaid and Children’s Health Insurance Programs, and importance to health outcomes.
The measures were intended to adva-nce evaluation of pediatric health care quality in the United States.2,6 Medicaid and states’ Children’s Health Insurance Programs are encouraged to use the recommended measures to assess the quality of services provided to the children they serve. With that application in mind, measures were primarily selected to be evaluated by using claims data from health insurance plan enrollee populations. Thus, most of the CHIPRA measures are annual benchmarks expressed as rates, with denominators specified to include people continuously enrolled in an insurance program during the measurement year.
Ten of the measures are designed to be calculated by using administrative data only; that is, claims data. Two are calculated by using claims data linked to vital records data. Eleven are “hybrid” measures, designed to be calculated by using either claims data or a combination of claims data and systematic review of medical records. The remain-ing measure uses data from a patient survey.7 None of the current CHIPRA measures are designed to be calculated by using automated data from an electronic health record (EHR), presumably because not all health plans currently have access to EHR data and/or have the functional ability to search EHR data.
The initial core measures have some important limitations.8–11 Health insurance claims data include only services provided during a period of coverage under any given plan. Thus, the CHIPRA measures could compare the quality of care received by people continuously enrolled in one plan, or possibly compare longitudinal data from enrollees in different plans. Claims-based measures cannot be used to evaluate care provided to uninsured or discontinuously insured people, however, limiting the measures’ usefulness in assessing care quality in a clinic if patients’ payers change, or if patients lose coverage but are still seen at the clinic.8,12–14 This is unfortunate, as such assessments would be particularly helpful in evaluating how health care reforms initiated through the Patient Protection and Affordable Care Act affect care quality.15 Further, claims data represent bills issued to insurance plans, potentially missing important care information, as not all care is consistently billed; for example, claims may be entered only if providers believe the services will be reimbursed.16,17
Data from EHRs may help to address the limitations of the claims-based CHIPRA measures. The use of EHRs in ambulatory care settings has rapidly expanded in recent years owing to federal mandates and incentives, especially in community health centers (CHCs) and other settings where uninsured children receive care.18–20 As the availability and quality of EHR data continues to improve, these data could replace or augment administrative claims data in care quality assessments. This article discusses the feasibility of applying the CHIPRA measures in EHR data from a network of outpatient CHCs; suggests how these measures could be adapted for application in outpatient EHR data (including both claims and medical record data); and assesses the differences in results seen when applying the original claims-based versus adapted EHR-based specifications, by using 2 measures as examples. This study was reviewed and approved by the Oregon Health & Science University Institutional Review Board.
To determine the feasibility of applying the CHIPRA measures in outpatient EHR data, we collaborated with OCHIN (originally Oregon Community Health Information Network, now just OCHIN), a 501(c)(3) collaboration of CHCs in more than 50 primary care member organizations from several states.21 OCHIN provides and maintains a comprehensive Epic EHR infrastructure for its member clinics. All OCHIN clinics share this EHR; patients have a single health record number linked across all sites. OCHIN clinics provide ambulatory care regardless of patients’ ability to pay. Almost all OCHIN clinic patients are from households below 200% of the federal poverty level; about half of visits are from uninsured people, and half from publicly insured people.
The OCHIN EHR includes administrative data for all member clinics, and clinical data for most of these clinics. The administrative data include appointment information and diagnostic and procedure codes (similar to billing data in insurance claims); the full EHR clinical data contains problem lists, physician notes, prescription records, laboratory results, and so forth. OCHIN has these data for both insured and uninsured patients. The OCHIN clinics’ data can be searched electronically, without manual chart review; are stored in a central repository; and are regularly checked and “cleaned” by OCHIN data staff and clinic staff.
To assess the extent to which clinical data could be used to evaluate each of the CHIPRA measures, and how each meas-ure could be adapted for application in this outpatient EHR dataset, we used an iterative process involving input from researchers, policy makers, informatics experts, and clinicians. We started with the 2011 CHIPRA Initial Core Set Technical Specifications Manual’s numerator and denominator specifications.7 We reviewed each measure’s specifications to determine whether the needed data points could be obtained in OCHIN’s EHR data, and how each measure could be adapted for application in the EHR data while maintaining the integrity of its original specifications.
For 2 CHIPRA measures, we compared the results obtained from the original, claims-based measure specifications, to the results obtained when using our “adapted” specifications, which include electronically abstracted EHR clinical data. We calculated the descriptive differences between the numerators and denominators obtained by using each method. We conducted these analyses among Oregon clinics from OCHIN’s collaborative that had the full EHR in operation before the start of the measurement year (44 clinics in 2009 and 47 clinics in 2010), so that both administrative and clinical data were available for our study population.
This CHIPRA measure assesses the percentage of sexually active young women receiving Chlamydia screening in a given measurement year. In both the original and adapted versions of this measure, the numerator is bas-ed on the CHIPRA specifications’ list of Current Procedural Terminology (CPT) and Logical Observation Identifiers Names and Codes for Chlamydia screening. In this example, we assessed whether adapting the measure for use in EHR data could improve the accuracy of the measure’s denominator. The CHIPRA denominator for this measure is “sexually active women aged 16 to 24 who had ≤1 enrollment gap of up to 45 days during the measurement year.” We identified young women aged 16 to 24 who had ≥1 clinic visit during the measurement year, defining visits by using the CHIPRA measures’ list of “visit type” codes. We then identified which of these women were sexually active, comparing 2 identification methods: (1) the original CHIPRA specifications, based on claims data, and (2) our adapted specifications, based on EHR data.
The original claims-based specifications consider a female patient to be sexually active if, at any point in the measurement year, her claims data contain 1 of a set of CPT, Healthcare Common Procedure Coding System, International Classification of Diseases, Ninth Revision diagnosis, or Logical Observation Identifiers Names and Codes that identify sexually transmitted infections, pregnancy tests or care, or prescriptions for contraceptives. In the “adapted” specifications, we captured data on sexual activity from the OCHIN EHR’s “social history” section, which includes a field for “Sexual Activity,” with response options “Yes,” “No,” “Not currently”, and “Not asked.” We considered a woman ever sexually active if indicated by “Yes” or “Not currently” in the sexual history field.
In both specifications, we considered a person sexually active if the relevant codes (original measure) or sexual activity status in the social history section (adapted measure) were ever seen in the medical record, before or including the first visit in a given measurement year; to better capture women who ever had sex and therefore should receive screening, we did not limit our capture to data only from the measurement year, as the CHIPRA measure specifies.
We calculated rates separately for women who were pregnant during the measurement year. We defined pregnancy according to the EHR data, which uses a complex, validated algorithm to identify pregnancies based on data from myriad sources.
This measure assesses the rate of documented BMI percentile among children aged 3 to 17 in a given measurement year. The original claims-based specifications of this measure limits the denominator to children (aged 3–17) who had ≤1 gap in enrollment of up to 45 days and ≥1 outpatient visits with a primary care provider or obstetrician/gynecologist in the measurement year. Similarly, the denominator in our calculations included children with ≥1 outpatient visits. In this example, we assessed how adapting the BMI measure for use in EHR data could improve the accuracy of the numerator. The numerator in the original claims-based specifications uses International Classification of Diseases, Ninth Revision codes for BMI documentation. Our adapted specifications captured data on BMI percentile as documented in the EHR’s vitals data. We then compared the rates by using each method. As suggested in the original specifications, we present data for pregnant adolescents separately.
We determined that 16 of the 24 original CHIPRA measures could feasibly be evaluated in the OCHIN outpatient EHR dataset (Table 1); the rest required data not available in EHR data from primary care clinics, such as patient-reported, hospital, or dental care data (Table 2).
There were 3 main adaptations necessary to apply the CHIPRA measures in the OCHIN EHR data set. We outline these as follows and include examples of their application.
Because our EHR data are not limited to health plan enrollees, we developed a visit-based denominator relevant to patients who were “active” at the study clinics in the measurement year. Examples are discussed here; for a complete list, see Table 1.
In the original CHIPRA prenatal (Table 1, numbers 1 and 2) and birth outcome measures (numbers 3 and 4), the denominators are live births in an enrolled population. In the EHR data, there was no direct way to identify live births. Thus, we recommend that EHR-based analyses be limited to populations of women with ≥1 prenatal care (PNC) visit and ≥1 infant/first-year well-child visit (WCV) or postpartum visit. We recognize that this specification might bias the results toward women with fewer options about where to receive care, and/or toward those who sought more care; however, including women with only a WCV or postpartum visit (and no PNC visit) could lead to even greater bias, as these women may have received PNC elsewhere. Further, including women who had only a PNC visit (and no WCV or postpartum visit) could also underestimate rates of receipt of appropriate care because some women who received a PNC visit might not have taken the pregnancy to term (and thus might not be expected to continue PNC), or might have transferred PNC, or received WCVs elsewhere.
The original specifications of the childhood preventive care measures (Table 1, numbers 6 and 7, 9–12) are also based on an enrolled population. To calculate these measures for 3- to 6- and 12- to 21-year-olds in EHR data, we suggest defining the denominator as children and youth who have ≥1 clinic visit in a given measurement year and also had ≥1 visit in the year prior, to identify “established” clinic patients. This requirement selects for regular care users, which could lead to falsely elevated performance on this measure. If children with only 1 acute visit are included, however, it could underestimate rates of appropriate WCV receipt, as patients who have only 1 visit may not be “established” at the clinic, and possibly received this care elsewhere.
The original PNC measures (Table 1, numbers 1 and 2) account for month of pregnancy at time of enrollment. Lacking enrollment data, we suggest defining this measure by accounting for month of pregnancy at the date of the first PNC visit. The birth outcome measures (Table 1, numbers 3 and 4) are specified to be calculated by using regional vital statistics data, which include gestational age at delivery, birth weight, parity, and delivery presentation; however, such data often take time to become available. Thus, to assess current care quality, we suggest calculating these measures based on EHR data on birth weight, gestational age, pregnancy end date, delivery mode, singleton status, and parity, as recorded either at postpartum visits or the first WCV.
Several of the original CHIPRA measures involve medication dispense data (Table 1, numbers 5, 13–15). Our EHR data include medication prescriptions, but not dispenses. Medication schedule and dosing data are available, but are currently not standardized, making it difficult to determine how long a patient remained on a medication. The medication dispense data measures can, however, be adapted to the available data. For example, the denominator for the follow-up to attention-deficit/hyperactivity disorder (ADHD) medication prescription measure requires documentation that a patient was prescribed a given ADHD medication and remained on it for ≥210 days. Without medication dispense data in the EHR, we cannot be certain that a patient received the medication, nor whether refills were authorized. Many ADHD medications cannot be issued for more than 1 month at a time, so we could estimate continuity and duration using refill date information, although this could introduce bias toward patients regularly seeking refills. In addition, patients issued prescriptions for longer than a month or those with erratic refill requests may differ from those who need regular refills. In another example, the original Hemoglobin A1c testing measure is calculated among children aged 5 to 17 with a diabetes diagnosis and/or notation of being prescribed insulin or oral hypoglycemic for ≥12 months; we suggest adapting this to children prescribed these medications in the measurement year, regardless of duration of time on the medication.
We cannot assess the number of children who did not receive care in our clinic network; therefore, we cannot measure access to primary care physicians. Few of our clinics offer dental services and we do not have a systematic way of documenting receipt of these services; thus, neither of the dental measures could be assessed by using the OCHIN data. Our EHR data are from outpatient visits only, and we do not now have a systematic way to capture emergency department or inpatient care use, so none of the hospital-based quality measures are feasible. Finally, 1 CHIPRA meas-ure is based on data from the Consumer Assessments of Healthcare Providers and Systems Medicaid 4.0 survey; this self-reported data are not captured in the EHR.
There were 8476 nonpregnant female patients aged 16 to 24 years who had ≥1 clinic visit during 2009, and 9816 in 2010 (Table 3). Among these nonpregnant patients, we identified a total of 6113 (72%) sexually active female patients in 2009 and 7166 (73%) in 2010. We identified 1999 pregnant patients in 2009 and 2098 in 2010.
Most nonpregnant patients identified as sexually active (55% in 2009, 60% in 2010) could be found by using either EHR social history documentation or the original CHIPRA measure specifications (ie, claims codes associated with oral contracep-tives, pregnancy tests, or tests for sexually transmitted infections) (Table 3 and Fig 1). Thirty-seven percent of female patients were identified as sexually active in 2009, and 31% in 2010, by using the original specifications only (no corresponding identification in the EHR). In contrast, 8% of female patients in 2009 and 9% in 2010, could be identified as sexually active in the EHR social history section only (no corresponding identification using the original specification). The resulting rates of annual Chlamydia testing differed depending on how the denominator was identified. Table 3 illustrates how rates might change depending on the method of identifying sexually active women. Also of note, 56 women known to be pregnant in 2009 and 37 known to be pregnant in 2010 were not identified as sexually active by using either the original or the adapted measure specifications.
When BMI percentile documentation was measured based on International Classification of Diseases, Ninth Revision codes, as in the original CHIPRA measure specifications (Table 4), about 1% of the population appeared to have had documented BMI. In contrast, when BMI percentile documentation was identified by using data from the EHR vitals fields, 71% to 73% of children age 3 to 17 years had their BMI percentile documented in the measurement year. In both years, BMI was significantly more likely to be found in the EHR only among persons aged 12 to 17 compared with those 11 and younger (χ2 P < .0001; results not shown).
The CHIPRA measures are an important step toward standardizing the assessment of children’s health care quality in the United States6; however, they are designed to be evaluated by using health plan claims data, limiting their ability to assess population health care quality. Importantly, insurance claims do not include uninsured and discontinuously insured people, potentially making them “invisible” in care quality evaluations. Further, clinic-level quality evaluations based on claims alone will unfairly represent clinics serving discontinuously insured populations, as many of their patients receive services during periods without coverage.13,22,23
To measure population health care quality, including the uninsured and discontinuously insured, we assessed the feasibility of adapting the CHIPRA measures for application in EHR data, concluding that most of the measures could be adapted, with modifications. The most notable adaptation pertains to determining population denominators. Our approach to identifying an “established” patient population was to include patients with ≥1 visit in a measurement year, and in some cases a visit in the year prior. This visit-based approach differed from the original enrollment-based measure specifications. In addition, our suggested methods for adapting some of the measures’ denominators allow for either well visits or urgent outpatient visits. We made this choice on the premise that any visit can be an opportunity to deliver needed care; however, another approach would be to define denominators as persons with ≥1 “well” visits.
We used data from a network of CHCs because these clinics serve many uninsured and discontinuously insured children, who would be missed in insurance claims data. For example, 25% of OCHIN clinics’ pediatric visits in 2009, and 18% in 2010, were self-paid, suggesting the child was uninsured. However, our suggested adaptations are relevant to any practice wanting to use EHR data to measure care quality. This approach is relevant to current policies, which establish Accountable Care Organizations and similar infrastructure, that require providers to measure the quality of care delivered to populations.24 If Accountable Care Organizations are fully realized with defined populations, there may be less need for visit-based denominators; however, the current standard requires this approach for identifying a population denominator in many practices.
Claims data document services billed and associated diagnoses, which does not include all care received, as in the BMI example.17,25–27 Thus, even apparently complete claims data may yield inaccurate rates when used alone. In addition, such data are often not accessible to clinicians in a useable form. In this article, we took the next step and outlined how to operationalize many of the CHIPRA measures for use in EHR data.
The examples presented in this article (Chlamydia testing and BMI percentile documentation) illustrate how quality of care measurements may vary depending on the content of the measure, the specifications, and the availability of appropriate data. For example, <1% of patients were identified as having annual BMI documentation by using claims-based specifications, versus 71% to 73% in the EHR-adapted measurement. The Chlamydia testing example demonstrated substantial variability in rates depending on how the denominator population was identified, similar to findings of Mangione-Smith et al.28
There are limitations to the “adapted” methods we used. When measuring Chlamydia testing, we were unable to access the inpatient codes that are part of the original claims-based algorithm used to identify sexually active women. Had we been able to include inpatient data, the original measure might have identified even more women as sexually active. It may have also been useful to add pregnancy to our definition of sexually active women, as we found 93 who were known to be pregnant, but were not identified as sexually active in the social history section of the EHR. In addition, in OCHIN’s network of CHCs, the sexual activity field is not populated for about 15% of patients. Different health care organizations collect and store data differently in the EHR. Thus, including additional fields to identify sexual activity in the EHR would likely have yielded a larger denominator. The data for these analyses came from up to 47 Oregon clinics, which may have different coding practices; some of the CPT codes used in the original specifications were not found in all of the clinics’ data, likely because those codes are not regularly used at those sites. For example, a CPT code from the original specifications to identify Chlamydia screening (87810) was rarely used by several of the clinics.
Although there are clear limitations to using claims data only, there are also important limitations to EHR data when adapting the CHIPRA measures, as discussed here. First, we could adapt only a subset of the measures because the data needed for some measures (such as data on hospital and dental care) were not available in outpatient EHR data. Claims data linked to EHR data might address this limitation for some populations, but would still exclude people receiving inpatient care during periods of uninsurance. A linked inpatient-outpatient EHR dataset may also help to address this limitation. Second, the lack of medication dispense data required the adaptation of several measures to use prescribing data; however, this adaptation may be beneficial, as prescribing data are commonly used in care quality measures because it reflects providers’ actions.27 Third, visit-based population denominator definitions may introduce bias toward care “users” who receive care at rates greater than the overall patient population. Similarly, this approach is limited to people who are seen by the clinic. When uninsured people are not seen at the clinic, it is difficult to determine whether they received care elsewhere. Fourth, similar to claims data, the completeness of EHR data are reliant on individual clinicians and may not include all of the care provided. Standardization of documentation practices would assist in ensuring complete EHR data. Finally, we conducted the 2 example analyses by using data from a single EHR that is linked across multiple clinics. This kind of data resource is not, at present, commonly available, but will likely become more widely available with the expansion of EHRs. The importance of applying standardized measures of care quality in EHR data will increase with the prevalence of such data systems.
The clinical and policy implications of these findings are important. Care quality assessments based on claims data alone exclude the uninsured, as well as services not commonly billed (eg, BMI documentation); thus, claims-based measures may present a limited view of population care. This work is a first step toward enabling the application of the CHIPRA measures in uninsured and other CHC populations, and in EHR data. Although it is feasible to adapt many of the CHIPRA meas-ures for use in outpatient EHR data, depending on how numerators and denominators are re-specified, some information is gained and some is lost. Immediate next steps must involve assessment of the validity of each adapted measure, similar to our validation of the 2 examples presented here. Next, the use of CHIPRA measures in EHR data should be tested in other outpatient settings. The results highlight the importance of considering the limitations of the original CHIPRA measures when they are used to evaluate care quality.
All authors listed here have made substantive intellectual contributions to this manuscript. Dr Gold led the analysis design and overall manuscript content, and was the lead writer; Ms Angier contributed substantially to the intellectual work involved, wrote first drafts of several sections of the article, and oversaw reference management; Drs Smith and Gallia provided high-level intellectual input, and careful editing; Ms McIntire, Mr Cowburn, and Ms Tillotson were responsible for all data handling and analysis programming; Dr DeVoe contributed major editing to the manuscript and was a primary driver of the intellectual content presented here; and all authors approved this version of the manuscript for submission.
The funding agencies had no involvement in the design and conduct of the study; analysis and interpretation of the data; or preparation, review, or approval of the manuscript.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This project was directly supported by grant 1 R01 HS018569 from the Agency for Healthcare Research and Quality, grant UB2NA20235 from the Health Resources and Services Administration, grant 1RC4LM010852 from the National Library of Medicine, National Institutes of Health, and the Oregon Health & Science University Department of Family Medicine. Funded by the National Institutes of Health (NIH).