|Home | About | Journals | Submit | Contact Us | Français|
The Institute of Medicine proposed recently that, while current pay for performance measures should target multiple dimensions of care, including measures of technical quality, they should transition toward longitudinal and health-outcome measures across systems of care. This article describes the development of the Diabetes Epidemiology Cohorts (DEpiC), which facilitates evaluation of intermediate “quality of care” outcomes and surveillance of adverse outcomes for veterans with diabetes served by the Veterans Health Administration (VHA) over multiple years.
The Diabetes Epidemiology Cohorts is a longitudinal research database containing records for all diabetes patients in the VHA since 1998. It is constructed using data from a variety of national computerized data files in the VHA (including medical encounters, prescriptions, laboratory tests, and mortality files), Medicare claims data for VHA patients, and large patient surveys conducted by the VHA. Rigorous methodology is applied in linking and processing data into longitudinal patient records to assure data quality.
Validity is demonstrated in the construction of the DEpiC. Adjusted comparisons of disease prevalence with general population estimates are made. Further analyses of intermediate outcomes of care demonstrate the utility of the database. In the first example, using growth curve models, we demonstrated that hemoglobin A1c trends exhibit marked seasonality and that serial cross-sectional outcomes overestimate the improvement in population A1c levels compared to longitudinal cohort evaluation. In the second example, the use of individual level data enabled the mapping of regional performance in amputation prevention into four quadrants using calculated observed to expected major and minor amputation rates. Simultaneous evaluation of outliers in all categories of amputation may improve the oversight of foot care surveillance programs.
The use of linked, patient level longitudinal data resolves confounding case mix issues inherent in the use of serial cross-sectional data. Policy makers should be aware of the limitations of cross-sectional data and should make use of longitudinal patient databases to evaluate clinical outcomes.
More than 20 million Americans have diabetes.1 The direct (medical services) and indirect (disability and premature death) costs of treating diabetes exceed $130 billion per year.2 Amputations, renal failure, visual loss, stroke, heart attack, and premature death reduce the length and quality of life of patients.3 However, a number of landmark studies have indicated that risk factor control can markedly decrease microvascular, and possibly macrovascular, complications of type 1 and type 2 diabetes.4–7
Consequently, the federal government has set objectives for increased provision of care to persons with diabetes, as well as decreased rates of hospitalizations and complications. These rates are monitored on a population basis, with an emphasis on reducing disparities.3,8 A number of private sector organizations and health care quality coalitions have developed, adopted, or endorsed process and intermediate outcomes for persons with diabetes for use by physicians and health care plans.9–13
However, discerning actual progress in combating the diabetes epidemic has been difficult. National weighted population level reports from sources such as National Health and Nutrition Examination Surveys and Behavioral Risk Factor Surveillance System are generally based on comparisons of cross-sectional data, with risk adjustment for age and gender.14 Therefore, inferences made on changes in cross-sectional results in “quality” and “outcomes” can be biased by trends in “case mix.” For example, improvement in rates of diabetes-related adverse outcomes or intermediate outcomes of quality of care could result either from greater provision of appropriate care and/or an increase in the denominator of persons with incident onset diabetes (primarily type 2). Furthermore, at the regional level, such inferences are sensitive to assumptions about the estimated denominator of persons with diabetes.15
Evaluations of improvement within health care plans may also differ when results of randomized interventions with longitudinal follow-up16 are compared to operational reports using plan data.17 Some of these biases may result from the technical specifications used by the industry to define the denominator for performance measurement, i.e., a sampling frame for measurement that includes individuals who maintain enrollment with minimal disruption. Consequently, those who die within the year or leave the plan for any reason are not included.10 Thus, trends in outcomes based on serial cross-sectional data may be influenced by both in- and outmigrations of enrollees, in addition to the quality of care provided to enrollees, and inferences regarding performance confounded by selection biases. The Institute of Medicine emphasized that measures and rewards of performance should target multiple dimensions of care, initially including measures of technical quality, patient-centered care, and efficiency but transitioning toward longitudinal and health-outcome measures.18 Current national and industry snapshots are inadequate to meet this challenge; another approach is needed.
The Veterans Health Administration (VHA) provides a national laboratory in which to study trends in intermediate risk factor management and disease outcomes of diabetes on a national population level.19 The VHA provides comprehensive health care services to almost five million veterans annually and has the most advanced electronic health record in the United States.20
Although most veterans who are users of the VHA system maintain some continuity of care within the VHA, more than half these patients are also enrolled in the federal Medicare program because of either age (65 years or older) or disability.21 For veterans with diabetes, this percentage rises to nearly three in four patients. Thus, in order for researchers to be able to obtain a more complete picture of these patients′ health care and outcomes, it is necessary to include Medicare claims with VHA patient data for care in the same period. While Medicare administrative data are not as rich as the clinical information in databases of the VHA, their utilization addresses continuity of follow-up and thus provides more accurate estimates of complication rates and resource utilization.
The Diabetes Epidemiologic Cohort (DEpiC) is a research database of all VHA patients with diabetes from 1997 to 2006.22 The objectives of the DEpiC are to determine diabetes prevalence and incidence, provide surveillance of rates of diabetes-related morbidity and mortality, and evaluate time-varying predictors of diabetes-related morbidity and mortality in the veteran population. The DEpiC has also been used to evaluate variation in the quality of diabetes care and outcomes with variation in race/ethnicity, mental health status, and other comorbidities and to advance methodological issues related to the use of computerized patient data for epidemiology and health services research.23–42 In addition, the DEpiC is unique because of its size (over 1.5 million veterans with diabetes), national scope, and overrepresentation of individuals who are poor or have mental health conditions. This article reports on methodology used to develop the DEpiC and describes findings of methodological and policy importance in the evaluation of glycemic control and lower extremity amputations.
The health information infrastructure of the VHA makes the creation of a research registry possible, but it is necessary to obtain data from a number of sources and integrate it with appropriate quality checks (Figure 1). Available VHA data sources include inpatient and outpatient utilization data, prescription records, and laboratory test results, vital status, and patient surveys.43 Most of these data are collected on an encounter/order/transactional basis at individual facilities with abstracts transferred to central repositories. In the past, Medicare data on veterans enrolled in the VHA were obtained directly from the Centers for Medicaid and Medicare Services [previously Health Care Finance Administration (HCFA)] with individual data use agreements and significant work required for accurate linkages and data processing. Currently, these data are obtained by the Veterans Affairs (VA) Information Resource Center and are made available to investigators under interagency data use agreements. Data elements include Beneficiary Files (Denominator File—one file for each year with demographics; Institutional Claims File (Part A); Non-Institutional Claims File-Part B (Physician/Supplier DME) Medicare Enrollment database (EDB); Group Health Master File—Medicare managed care enrollees; Provider Files; and typical claims data, collected on two forms UB-92 (also known as HCFA 1450)—used by institutional providers (hospitals in- and outpatients, home health agencies, hospices); and HCFA 1500—used by physicians and laboratories.
Reusing data from merges of all VHA patients with Medicare utilization has an additional advantage. The Medicare utilization files based on a national merge with VHA patient identifiers can be searched for patients who meet study criteria but who would not be identified in VHA files. For example, VHA patients with diabetes diagnoses in their Medicare claims do not all have such diagnoses in VHA data and may not get their prescriptions filled in the VHA. Searching the national files of Medicare utilization by VHA patients for the appropriate codes adds value to studies of specific conditions. Similar advantages may pertain to other study population definitions.
When merging VHA and Medicare data sets, it is advisable to check their face validity. The proportion of duplicate matches is important. For the VHA patient population, the distribution of the matches by age (e.g., the percent of those 65 and older who match and the number of matches for age less than 18) is also a potential indicator of problems.
Once the study sample has been confirmed, the next step is constructing data elements from VHA and Medicare files that are similar enough to allow them to be used in analyses. Although this will largely depend on the nature of the study, most studies of utilization will include hospitalization and outpatient utilization data. The first step any study must accomplish is reconciling fiscal years and calendar years. Because the VHA is a federal agency that utilizes fiscal years, it is necessary to convert Medicare calendar year files to fiscal years. In addition, because veterans enrolled in Medicare health maintenance organizations (HMO) do not have utilization data available, it is necessary to consider whether to include such individuals in individual analytic data sets.
When the mortality status of all VHA patients is ascertained from four sources [the Beneficiary Identification Records Locator System, the Medical SAS Inpatient Datasets (for veterans who died during an inpatient stay), the Medicare Vital Status File, and the Social Security Administration's Death Master File], results are highly concordant with data from the National Death Index.43
VHA pharmacy and laboratory data are entered automatically and directly into the electronic health record (Veterans Health Information Systems and Technology Architecture) at the facility level, and there are monthly quality control evaluations. Nonetheless, it is possible that missing VHA and laboratory data can result from an incomplete transmission of facility VHA data collection or random nonentry of data, which can be addressed by the use of stochastic analyses.29
There are also limitations in ascertaining non-VHA laboratory results and medication prescriptions that must be acknowledged. Although VHA data capture intermediate outcomes for most clinical users cross-sectionally, chart abstraction adds about 10% to cross-sectional adherence in the performance of low-density lipoprotein cholesterol (over 2 years) and hemoglobin A1c tests (over 1 year).44 Consequently, inferences regarding processes of care are made most reliably using VHA fee-for-service Medicare data sets that capture whether tests are performed, although results are not available. Because the majority of veterans are of lower socioeconomic circumstances, we also assume that, prior to the implementation of the Medicare Part D benefit in 2006, we have captured medication usage on almost all veteran clinical users unless they are in a Medicare HMO for an extended period of time.
Prior to 2004, patient race/ethnicity in the VHA information system was recorded based on an administrative or clinical employee's observation. Since 2004, the VHA has collected self-reported race in compliance with a new federal guideline from the Office of Management and Budget. Results from the VA Information Resource Center suggest that observer-recorded and self-reported data for whites and African-Americans can be used interchangeably across years without creating serious bias, but that observer-recorded race was less reliable for non-African-American minorities.44
We have found it useful to improve the completeness and accuracy of race in three ways. Most patients have used VHA care for many years, and we have made use of the multiple entries for race over the years of data available. Furthermore, race/ethnicity in Medicare is self-reported and relatively complete. Combining Medicare data with enhanced VHA data on race/ethnicity decreased the proportions of missing data to 6.6% for race, <0.1% for gender (<0.1%), and 1.1% for marital status for patients prior to 2004. We have also demonstrated that evaluation of racial differences in the veteran population is sensitive to the inclusion of dual system use.34
A third source for race/ethnicity data is the large national VHA surveys. For example, we have survey data from nearly 25% of the VHA patient population in 1999. Use of these data further enhances our race/ethnicity in significant subsets of the population. They also permit evaluation of the accuracy of known data and potential biases of unknown data. Consequently, we are able to compare VHA estimates of diabetes prevalence to the U.S. adult population by standardizing age, sex, and race (Figure 2).
Although the control of hemoglobin A1c levels is a major governmental priority in the health care of persons with diabetes, national trends in hemoglobin A1c values have been conducted using cross-sectional study designs in the absence of weighted, longitudinal population samples.14 Public reporting from accrediting organizations also utilizes cross-sectional data.10 However, hemoglobin A1c values are also related to patient level factors extraneous to health care systems, especially duration of diabetes and comorbid conditions. For example, even in the high-quality environment of landmark clinical trials,5 mean hemoglobin A1c levels rose over 10 years in study subjects with incident diabetes. Furthermore, these studies excluded most individuals with comorbid physical and mental health conditions that could minimize benefit and increase the risk of glycemic control. In addition, even in short-term (26–52 week) clinical studies with intention to treat to <7%, it is difficult to achieve this target in patients receiving insulin. In a meta-analysis of trials comparing glargine versus neutral protamine Hagedorn insulin, the mean achieved improvement in hemoglobin A1c was 7.58% in both groups, with about one-third in each reaching less than 7%.46 Also, long-term follow-up is not available to evaluate whether the percent of “adherent” patients increased or decreased over time.
Consequently, multiple measures on individuals over time, termed “longitudinal data,” are crucial for assessing change over time within patients. Longitudinal data enable differentiation of cohort effects (patient population changes over time) from within-subject time effects. Longitudinal methods may give a better sense of how a patient's health care system is treating the disease over time than cross-sectional data, which overestimate improvement. This is illustrated in Figure 3.
In order to evaluate whether reductions seen in adherence to poor hemoglobin A1c following the introduction of performance measurement in the mid-late 1990s were possibly a consequence of the influx of new veterans,47 we used a growth curve approach to fit longitudinal data of all veterans utilizing the VA from 1998 to 2000.30,48 The deviation of patient level change in hemoglobin A1c from the average hemoglobin A1c growth curve for a given health care system is accounted for by random effects in the model. These random effects incorporate the correlation of multiple hemoglobin A1c measurements made on each patient. This use of growth curve methodology is attractive because it should result in meaningful assessments of quality of care as reflected in how health care systems influence patient hemoglobin A1c levels over time or, in other words, the patient response to the clinical setting.
We utilized this approach not only to evaluate overall population trends, but to evaluate facility level trends in the quality of care among diabetic veterans in 125 of the VHA facilities nationwide. The models ascribed to each facility an expected trajectory (“growth curve”) of hemoglobin A1c values for patients utilizing that facility as their home facility. A key methodological advance was the demonstration that hemoglobin A1c exhibits seasonality (Figure 4).27 Our cohort was large enough to demonstrate that a winter–summer contrast was observed consistently in all age, sex, race, insulin use groups, and climate regions. We used the estimated facility level growth curve slopes to compare outcomes of diabetes care among the facilities, both with and without case mix adjustment.
We reported that case mix-adjusted glycosylated hemoglobin values among veterans with diabetes decreased by -0.31% (range -1.90 to +1.03) over 2 years, indicating that improved glycemic control over time is not an artifact of changes in the population served by VA. These findings provide a convincing demonstration of the overall effectiveness of VA diabetic care management and suggest that with wider availability of the electronic health record, longitudinal data could eventually be used in larger health care plans.
Finally, our results on the evaluation of VHA facilities have demonstrated that risk adjustment of cross-sectional hemoglobin A1c using either administrative data37 or self-reported health and socioeconomic status, as well as health behaviors,39 has little predictive or explanatory value. Nonetheless, they significantly alter identifications of “best” and “worse” performing facilities.
Consequently, in the lack of a validated model to control for bias that could result from variation in case mix among facilities, the presence of seasonality that can lead to potential gaming, and inability to control for in- and outmigration, we have proposed that all or none cross-sectional threshold measures for optimal glycemic control are likely to be misleading for the purposes of public reporting and pay for performance.49
Diabetes-related foot complications inflict an enormous individual and societal burden. Persons with diabetes have a 15% lifetime risk of a lower extremity complication, and such individuals are at increased risk of hospitalization, infection, amputation, and death.50 Minorities and seniors incur a disproportionate number of amputations. Several U.S. governmental agencies utilize total amputation rates as a key indicator of foot health on a population basis.3 Healthy People 2010 have set a target of a 55% reduction in total amputation rates during the current decade, with a reduction in racial disparities,3 after amputation rates in 1990–2000 trended in the opposite direction from the target goals set for Healthy People 2000.
More recently, the Agency for Healthcare Quality and Research has utilized lower extremity amputations as prevention quality indicators for communities.51 However, the Organization for Economic Cooperation and Development52 has recommended major amputation rates for this purpose, in part because of the marked functional difference between minor and major amputations and in part because a minor amputation may precede a major one in the hopes of salvaging the limb.
However, any inferences regarding rates of amputations are less meaningful without individual level data. For example, the use of total amputation rates to assess population rates overestimates amputation rates because they are derived from the National Hospital Discharge Survey and the National Health Interview Survey. Because the National Hospital Discharge Survey samples hospital discharges, it does not necessarily reflect individual level results; persons hospitalized more than once a year will be overcounted.53 Finally, because of recent increasing secular trends in incident diabetes, as well as increasing recognition of undiagnosed diabetes, the population risk for lower extremity complications may not be stable over multiple years.
However, in contrast to communities, health care systems are in a position to make improvements in foot health care delivery for their population of enrolled individuals, if provided with actionable information.54 Individual level data permit definitions of persons with diabetes accurately and allow for determination of the maximal amputation level (major vs minor) per episode when multiple amputations were performed on a patient. Our methodology enabled us to evaluate present risk adjusted observed to expected amputation rates simultaneously for 22 regional units of the VHA using a four-quadrant map that could facilitate the management of foot care programs based on minor and major amputations (Figure 5).
Our proposed classification schema demonstrates that health care systems with similar total amputation rates can be differentiated further by observed to expected rates of major and minor amputations. We suggest that this detailed information can enable health care systems with access to individual level data to target the evaluation of foot health care delivery for persons with diabetes more accurately. For example, regions that are low outliers by major and minor amputation rates (first quadrant) could be benchmarked as best performers, thus warranting case study and dissemination of best practices. In contrast, those regions having higher rates of both major and minor amputation rates may need to be targeted immediately for a comprehensive program review and quality improvement interventions. Regions that are ranked as having high major but low minor rates (quadrant 4) could consider evaluating access and quality of their foot salvage care, as a low incidence of amputations could actually reflect an inappropriately conservative approach to nonhealing ulcers.55 Finally, those with high minor rates and low major rates could assess their foot care systems to implement those that provide patient-centered care56 and possibly improved outcomes.57
The Diabetes Epidemiology Cohorts is a unique research resource used to study outcomes of veterans with diabetes, many of whom are older and Medicare enrolled. Through linked pharmacy, laboratory, and medical encounter data over multiple years, it is possible to evaluate determinates of intermediate outcomes and complications of diabetes, as well as to control for case mix issues inherent in the use of cross-sectional data.
This approach permits the use of special methods in epidemiologic and health services research to model dynamic cohorts with time-varying covariates, including physiological parameters (such as seasonality), risk adjustment (such as prevalent and incident diabetes), longitudinal analyses, and better definition of outcomes (such as amputation).
Consequently, the DEpiC has potential importance to federal policy makers in constructing and validating quality measures that are consistent with the challenges posed by the Institute of Medicine. It is a matter of concern that quality report cards structured around cross-sectional data may provide incentives for disenrolling the sickest patients and those with a longer duration of disease.18 The DEpiC provides a unique opportunity to construct less “gameable” quality measures that assess within-patient longitudinal care through multiple repeated measures on the same individual, enabling distinction of cohort effects (patient population changes over time) from within-subject time effects, which are often used in comparative quality assessments. It also may inform federal policy makers as to the impact of such factors as provision of preventive care and care coordination upon adverse diabetes-related outcomes of Medicare beneficiaries. Finally, although regional health maintenance organizations may have similar electronic health record capacity, the national scope of the VHA clinical population, and its potential for the evaluation of chronic complex illnesses, may be unique.