|Home | About | Journals | Submit | Contact Us | Français|
Instrumental variable (IV) methods can correct for unmeasured confounding when using administrative (claims) data for cardiovascular outcomes research, but difficulties identifying valid IVs have limited their use. We evaluated the safety and efficacy of drug-eluting coronary stents (DES) compared to bare metal stents (BMS) for Medicare beneficiaries with acute coronary syndromes (ACS) using the rapid uptake of DES in clinical practice as an instrument. We compared results from IV to those from propensity score matching (PSM) and multivariable regression models.
Retrospective cohort study involving 62,309 fee-for-service beneficiaries aged 66 and older treated with coronary stenting between May 2003 and February 2004. Outcomes were measured for 46 months after revascularization using claims data.
DES recipients were younger, had lower prevalence of myocardial infarction, and had fewer comorbidities compared to BMS recipients. Use of DES was associated with lower rates of mortality by PSM (HR 0.80, CI: [0.77, 0.83]) but not by IV (HR 0.99, CI: [0.87, 1.11]). IV models estimated a larger reduction in repeat revascularization (HR 0.76, CI: [0.63, 0.89]) than did PSM (HR 0.90, CI: [0.87, 0.93]).
Based on IV analysis, the increased utilization of DES relative to BMS among Medicare beneficiaries with ACS is associated with reduced rates of repeat revascularization and no difference in mortality. IV approaches provide a useful complement to conventional approaches to cardiovascular outcomes research with administrative data.
The cardiovascular community has demonstrated considerable enthusiasm for outcomes research employing observational data.1, 2 Observational analyses evaluate real-world usage patterns and populations and can be performed at significantly lower cost than randomized trials; these advantages ensure that such approaches will remain an important complement to trials. However, because treatments are not randomly assigned in observational data sets, failure to adjust for factors that affect treatment choice can bias study results (i.e., confounding). Conventional regression modeling approaches including propensity score methods can adjust for confounding from factors observed in the data, but they cannot adjust for confounding due to unobserved factors. Such unmeasured confounding has been proposed as an explanation for notable discrepancies between observational evidence and trials in cardiovascular outcomes research, such as that of hormone replacement therapy for postmenopausal women.3 Observational analyses using administrative data from insurance claims are popular due to data availability and large sample sizes, particularly data provided by the United States Centers for Medicare and Medicaid Services. However, such studies may be particularly susceptible to confounding given the lack of detail provided on claims.4
Randomized trials have not demonstrated a survival benefit of drug-eluting coronary stents (DES) over bare metal stents (BMS) despite dramatic reductions in rates of restenosis and repeat revascularization.5, 6 In contrast, multiple observational analyses suggest a survival benefit from DES.7–10 It is possible that DES improve survival among high-risk subpopulations receiving them in clinical practice but excluded from randomized trials, including high risk groups such as elderly patients with acute coronary syndromes (ACS). However, it is also possible that observational analyses of DES effectiveness may be limited by unobserved confounding factors, as DES are used preferentially for patients with less acute coronary disease.11
Instrumental variable (IV) analysis is an established technique that can address situations in which there is potential confounding due to unmeasured variables. IV approaches depend on the use of valid instruments, which are observed factors that affect the treatment a patient receives but, after adjustment for other observed factors, do not affect patient outcome through a pathway other than treatment selection. In other words, instruments are factors that identify populations of patients which would receive treatment A with the instrument at one value and treatment B given another value, and in which the difference in treatment choice can be treated as a “natural experiment”. While difficulties identifying valid instruments can limit the utility of this method in cardiovascular outcomes research and the estimated effect pertains only to the sub-population whose choice is affected by the instrument rather than the whole population, IV methods have been used successfully in previous cardiovascular outcomes studies.12, 13
Using administrative data, we used the rapid uptake of DES in the United States after initial approval in April 2003 as an instrument to evaluate the effect of DES on mortality and additional revascularization procedures for older patients with ACS, and compared results to those obtained by conventional techniques. During the period of rapid increase in the utilization of DES, the month-to-month variation was hypothesized to represent a potential instrument to appropriately adjust for treatment choice.
We used 2003 and 2004 Medicare Provider Analysis and Review (MedPAR) records to identify fee-for-service beneficiaries age 66 or older who were admitted for ACS, defined by International Classification of Diseases (ICD) diagnosis codes 410.xx and 411.1. Each patient’s first hospitalization for ACS was considered his or her index admission. For identified patients, we obtained all MedPAR, Carrier, Outpatient, and Denominator files for 2002 through 2007. Using claims from the 12 months prior to the index admission and the index admission itself, patients were excluded if they had a history of ACS, cardiac surgery, or percutaneous coronary intervention (PCI) prior to the index admission; these exclusions were applied given the limitations of the Medicare data in determining whether PCI was performed at the site of an existing stent or graft. Patients aged 65 years were excluded due to the need for prior Medicare claims data to evaluate the other exclusions. Patients were also excluded if they opted out of Medicare Part A or B at any time between 2002 and 2007, as outcomes could not be assessed for these patients.
We used Carrier and MedPAR records to identify whether patients received coronary revascularization during the 30 days following their index admission date, based on ICD procedure (PCI: 36.06, 36.07; CABG: 36.1x), Healthcare Common Procedure Coding System (PCI: 92980, 92981; CABG: 33510-33523, 33530, 33533-33536) and Diagnosis-Related Grouping (PCI: 526, 527, 555-558; CABG: 106-109, 547-550) codes. Patients were included if their initial revascularization procedure was PCI with stent insertion, conducted between May 2003 and February 2004. This time period includes all complete months in which the first approved form of DES (sirolimus-eluting) was the only DES available in the United States. We categorized patients as DES recipients based on whether they received any DES during the initial PCI associated with ACS admission (based on procedure code 36.07).
Outcomes included all-cause mortality and repeat coronary revascularization. All-cause mortality was obtained from the Denominator file. Repeat coronary revascularization (PCI or coronary artery bypass grafting) was obtained from MedPAR and Carrier records. Staged PCI procedures were not considered repeat revascularization. A patient was considered to have had a staged PCI if a second PCI was performed within 30 days of the initial procedure without an ACS-related diagnosis code included on the claim. Follow-up data were available until death or 46 months following the initial revascularization procedure.
We conducted multivariate model adjustment using Cox proportional hazards models. Covariates included patient demographic characteristics such as age, gender, race, and socioeconomic status as well as clinical characteristics including medical comorbidities. Because of the seasonal variation in myocardial infarction (MI) incidence, models controlled for specific ACS diagnosis.14, 15 ZIP code-level median income data were obtained from the 2000 Census, and a measure of rural and urban status was obtained by linking ZIP code to Rural-Urban Commuting Areas.16 The 2003 and 2004 Medicare Provider of Services files were used to determine hospital characteristics (ownership type and major medical school affiliation). Comorbidities were obtained by applying the Elixhauser comorbidity criteria to claims from the months prior to the index admission.17 To account for miscoding, comorbidities on Outpatient and Carrier claims were included if documented on two or more claims at least 30 days apart.18 All analyses were performed using complete case analysis.
We predicted the propensity to receive DES using the covariates above using generalized boosted modeling based on logistic regression to create a flexible prediction model.19 Generalized boosted modeled is a data-driven technique drawn from machine learning techniques. Propensity score matching (PSM) was used to pair DES patients to BMS recipients with a similar predicted probability of receiving DES. We matched without replacement using a caliper of 0.25 times the standard deviation of the propensity score.20, 21 Treatment effects were obtained using Cox proportional hazards models, with stratification by pair.22
We used the month the patient was treated with PCI (specified as a series of binary indicator variables) as an instrument to evaluate DES use. Time as an instrument has been used previously for other rapidly-adopted technologies, including highly-active antiretroviral therapy for HIV disease.23 In order to serve as a valid instrument, the month a patient was treated with PCI must strongly influence whether or not they received DES, but not their outcome through unobserved pathways in the data. We tested the strength of the instrument by evaluating the joint test of significance of the time indicator variables in the adjusted model predicting receipt of DES. The assumption regarding unobserved factors is by definition untestable, but we did evaluate whether observed characteristics were correlated with the instrument by developing models to predict the risk of outcomes based on observed covariates (stratifying by ACS diagnosis due to well-known seasonal trends) and evaluating whether monthly variation existed in the predicted event rate.
IV analysis was conducted with linear regression models, and the results were approximated as hazard ratios.12 We conducted several tests to evaluate the robustness of IV findings. To evaluate whether unobserved variation in hospital quality affected our results, hospital-level fixed effects were tested.24 Most previous reports implementing IV have used linear models, and traditional IV regression approaches do not allow for one event (e.g., mortality) to act as a competing risk for another (e.g., repeat revascularization). To evaluate whether our results were sensitive to these limitations, we also constructed IV models using two-stage residual inclusion models, using logistic regression to predict DES use and Cox models for outcome, and also tested a competing risk proportional hazards model for repeat revascularization.25, 26 Standard errors for these models were obtained by bootstrapping with 500 iterations.27
Univariate comparisons of DES and BMS recipients were conducted using t-tests and chi-square tests. All analyses were conducted at the patient level; inferential statistics were considered significant at a two-sided alpha level of 0.05 and were conducted in Stata/IC, version 12.1. The University of North Carolina Public Health-Nursing Institutional Review Board approved this study. This study was funded by the National Institute on Aging (R01AG025801 and T32AG000272), the National Institute of General Medical Sciences (T32GM008719), and the National Heart, Lung, and Blood Institute (F30HL110483), no funding agency had any role in the design, conduct, or dissemination of the study. The authors are solely responsible for the design and conduct of this study, all analyses, the drafting and editing of the paper and its final contents.
The cohort included 62,309 beneficiaries treated with PCI after presentation with ACS. Of these patients, 28,653 (46.0%) received at least one DES (Table I), of which 2,843 (7.8%) also received at least one BMS. Patients receiving DES were younger than BMS recipients and lived in ZIP codes with higher median incomes. They were more likely to be female, have unstable angina rather than MI, undergo a multiple vessel PCI, and have diabetes mellitus. DES recipients were less likely to also be receiving state Medicare buy-in benefits and had lower rates of most comorbidities. Rates of death and repeat revascularization were higher in BMS recipients than DES recipients (Figure 1), with DES and BMS event curves differentiated by log-rank tests (p<.001 for both).
Over the study period, DES utilization increased from 28.1% to 57.8% (Figure 2). Patients presenting with either ST-elevation MI (STEMI) or non ST-elevation MI (NSTEMI) were less likely to receive DES than those presenting with unstable angina (relative risk 0.62 and 0.87, respectively, p<0.001), but increasing temporal trends were noted in all three types of ACS (data not shown). In adjusted models, the month of PCI procedure strongly predicted whether a patient received DES (joint test of significance: F = 292, p<.001) ; this finding is important to demonstrate because weak IVs can bias estimates.28 As expected due to known seasonal variation in MI rates, the relative incidence of ACS diagnoses varied over the study interval, with MI-related ACS more common in the cooler months (Figure 3). However, after stratification by ACS diagnosis, predicted event rates based on all other observed patient and facility characteristics varied little over the study period (Table II).
Propensity scores ranged from 0.07 to 0.89 for DES recipients and 0.07 to 0.84 for BMS recipients, with 62,296 patients (>99.9%) in the region of common support. Matching identified controls for 25,989 DES recipients (90.7%). The 2,664 unmatched DES recipients were younger, less likely to have MI, and had fewer comorbidities than matched recipients. After propensity-based matching, the DES and BMS cohorts appeared similar across observed covariates.
IV analysis produced results that differed from those using conventional approaches. DES utilization was associated with lower rates of mortality using both multivariate adjustment and propensity-based matching, with hazard ratios ranging between 0.80 and 0.81. This difference was eliminated using the IV approach, with the hazard ratio estimated at 0.99 (confidence interval: 0.87, 1.11) (Table III). Conversely, DES was estimated to be more effective at reducing repeat revascularization by IV analysis than by conventional approaches. Multivariate modeling and propensity score matching produced hazard ratios in the range of 0.90 to 0.91, while the IV estimated the hazard ratio at 0.76 (confidence interval: 0.63, 0.89). Results obtained by IV were robust to variations in modeling approach, including the addition of hospital-level fixed effects (correcting for unobserved differences in the facility providing PCI), use of two-stage residual inclusion approach with Cox proportional hazard models rather than linear IV models, and the use (with two-stage residual inclusion) of a competing risk proportional hazard model rather than the standard Cox model for repeat revascularization.
In this retrospective cohort study of Medicare beneficiaries receiving PCI, we compared results using conventional modeling approaches to an instrumental variable approach leveraging the rapid uptake of DES over time. Using conventional methods, DES was associated with a strong (20%) relative risk reduction (RRR) in all-cause mortality and only a modest 10% RRR effect in repeated revascularization at 46 months post-PCI. In contrast, IV methods produced estimates showing no association between DES and all-cause mortality, but a much stronger (25% RRR) effect on repeat revascularization.
The findings from this study of Medicare beneficiaries with ACS are consistent with those of other observational studies of stent outcomes. Multiple investigators have suggested a survival benefit of DES using conventional methods, such as propensity scores.7, 9, 29, 30 In contrast, others using IV and related methods have generally shown no difference in mortality and stronger impacts on repeated revascularization than is observed with propensity scores.31–35 A natural question that emerges from these disparate lines of evidence is, “which one should be believed?” While the extensive trial data available for the DES versus BMS comparison are consistent with the IV results in this analysis, reliance on clinical trial results to validate any particular methodology (IV or PS in this case) is not necessarily appropriate in most applications.36 Both propensity scores and IV methods rely on untestable assumptions – for propensity scores, it is the lack of confounding between treatment assignment and outcome after adjustment for propensity score, while for IV it is the lack of confounding between instrument and outcome after adjustment. We believe that in this circumstance, the latter is the more defensible assumption.
In order for our time-based instrument approach to be a valid, it must be plausible that potential bias from other secular trends, such as unobserved changes in patient characteristics, improvements in other medical treatments provided as part of care, and data coding quality, are modest.37 While changes in unobserved characteristics are, by definition impossible to test, we show in Table II that after stratification for an observed characteristic known to exhibit seasonal variation (ACS diagnosis), based on all other observed characteristics, there is no evidence that patients differ in predicted event rate over the study window. Another possibility is that trends in the characteristics of facilities using DES are correlated with differences in outcome. However, correcting for all static, unobserved facility-level characteristics through fixed effects modeling did not alter our results, suggesting that the effect of any such bias is modest. It is possible that a time-based IV could be affected by seasonal differences in outcome, such as potentially worse outcomes for patients treated by relatively inexperienced cardiology fellows in July, but a sensitivity analyses (results not shown) restricting the cohort to only those treated in non-teaching facilities showed comparable results. While improvements in ACS care or data coding are possible, we believe the potential impact of these threats very limited due to the limited time period (10 months) under study.
IV approaches have important limitations. As with most IV techniques, the estimates we produce are local average treatment effects and are estimated from the population of “marginal patients” whose treatment is affected by the IV.38 The local average treatment effect will only equal the overall average treatment effect if DES work equally well in all patients (i.e., if the treatment effect is homogenous) or if treatment choice is not affected by expected treatment effect. While these assumptions are not plausible in the case of DES, we still believe the local average treatment effects to be useful estimates given that we have a broad population of marginal patients (utilization doubled from 30% to 60% over the study period). As with most observational analyses employing Medicare claims, we did not have access to patients enrolled in Medicare Advantage plans, who are in general younger and healthier than Medicare Part A and B beneficiaries; the generalizability of our results to this subset of the Medicare population is thus also limited.39
In this evaluation of drug-eluting stents for Medicare patients with ACS, instrumental variable methods suggested no mortality benefit and a significant reduction in revascularization procedures. These findings are generally consistent with randomized clinical trials, and suggest this particular subgroup may not receive “additional” benefit of DES therapy beyond reduction in repeated revascularization. Although observational analyses are a critically important component of outcomes research, confounding due to unmeasured factors (e.g., angiographic characteristics, socio-economic status, etc.) is a major threat to validity when assessing treatment effectiveness. Analyses with administrative data are particularly vulnerable to this threat. In those circumstances in which clinical practice changes rapidly after introduction of a new technology, IV analysis can provide a complementary mechanism to assess outcome.
We thank Marisa E. Domino, PhD and Alan R. Ellis, MSW (The Cecil G. Sheps Center for Health Services Research) for their suggestions regarding the methodology used in this paper.