|Home | About | Journals | Submit | Contact Us | Français|
The generalizability of clinical trial findings (efficacy) to routine care (effectiveness) may be limited. The present study found similar first year virologic and CD4 outcomes among antiretroviral-naïve patients treated through routine care vs. those participating in clinical trials.
The generalizability of clinical trial findings (efficacy) to routine care (effectiveness) may be limited due to study eligibility criteria and volunteer bias. While well chronicled in many conditions, the efficacy vs. effectiveness of antiretroviral therapy (ART) remains understudied.
A retrospective study of the UAB 1917 Clinic Cohort evaluated naïve patients starting ART between 1/1/00–12/31/06. Patients received ART through clinical trials or routine care. Multivariable logistic and linear regression models were fit to evaluate factors associated with virologic failure (VF=VL>50 copies/mL) and change from baseline CD4 count 6 and 12 months after ART initiation. Sensitivity analyses evaluated the impact of missing data on outcomes.
Among 570 patients starting ART during the study period, 121 (21%) enrolled in clinical trials vs. 449 (79%) receiving ART via routine care. ART receipt through routine care was not associated with VF at either 6 (OR=1.00;95%CI=0.54–1.86) or 12 (OR=1.56;95%CI=0.80–3.05) months in primary analyses. No significant differences in CD4 count responses at 6 and 12 months were observed.
Though marked differences in efficacy vs. effectiveness have been observed in the therapeutic outcomes of other conditions, our analyses found no evidence of such divergence among our patients initiating antiretroviral therapy for HIV.
Randomized clinical trials (RCTs) are the cornerstone of Level I evidence-based medicine treatment recommendations and provide the highest level of evidence . However, some RCT-tested interventions have not performed as well when implemented in routine care settings [2–5]. Factors such as selection bias introduced by trial eligibility criteria and volunteer bias among participants choosing to participate in research studies have been linked to this discrepancy [2, 4–10]. Selected patient samples may show improved treatment outcomes in trials (efficacy) when compared to the more heterogeneous population treated through routine care (effectiveness), raising concerns about the generalizability of RCT findings to routine care settings.
Efforts to characterize differential efficacy versus effectiveness of treatments have been undertaken in many medical conditions [2, 4–6, 9, 10], yet this relationship regarding antiretroviral therapy (ART) for HIV/AIDS has been notably understudied, particularly in the contemporary ART era . While numerous studies have separately evaluated either the efficacy or the effectiveness of initial ART regimens when used in RCTs and routine care, respectively, relatively few have studied the comparative effectiveness of treatment modality (RCT vs. routine care) on outcomes among patients starting ART in the same clinical setting. Therefore, we conducted a retrospective study to evaluate the impact of receiving initial ART through a clinical trial vs. routine care on short-term viral load and CD4 outcomes among ART-naïve individuals initiating therapy. Because treatment-naïve ART studies are commonly available, are ingrained in the culture of HIV care at many treatment centers, and provide a means to access medications and laboratories at little to no cost to patients, we hypothesized that volunteer bias would be less apparent in an HIV-infected cohort relative to other diseases. Accordingly, we posited the sociodemographic composition of those treated through clinical trials would be reflective of the larger clinic population and mirror the characteristics of those receiving ART through routine care. We further hypothesized that similar virologic and CD4 outcomes would be observed between patients treated in clinical trials and routine care due to the similarities in the patient populations.
Since 1988, the University of Alabama at Birmingham (UAB) 1917 HIV/AIDS Clinic (1917 Clinic) has provided HIV care for over 6,000 HIV-infected individuals. The UAB 1917 HIV/AIDS Clinic Cohort Database Project (UAB 1917 Clinic Cohort), recently recognized for excellence in information integrity (http://www.eiiaward.org/2007_winners), is a 100% quality controlled, IRB-approved prospective clinical cohort study that includes detailed sociodemographic, psychosocial and clinical information from HIV-infected patients receiving primary HIV and subspecialty care at the clinic (http://www.uab1917cliniccohort.org). The 1917 Clinic utilizes a locally programmed electronic medical record (EMR) that imports laboratory values from the central UAB laboratory, requires electronic prescriptions for all medications and contains detailed encounter notes. Both the UAB 1917 Clinic Cohort and local EMR have been described in detail elsewhere [12–14].
A dedicated clinical trials program and staff has been part of the 1917 Clinic since inception. At our center, RCTs for antiretroviral-naïve patients are frequently available and open for enrollment. Prior to study enrollment, providers ascertain patients’ willingness to learn more about clinical trial participation and refer interested patients to clinical trial study nurses who screen patients and begin the informed consent process. Once enrolled in a research study, patients receive additional follow-up from study personnel (nurses, mid-level providers and physicians) as determined by specific study protocols, in addition to regular outpatient care at the clinic. Patients initiating ART through routine care meet with a clinic pharmacist to discuss their regimen. Otherwise, no specific treatment protocol is in place and all clinic and laboratory follow-up is at the discretion of the primary provider (a nurse practitioner or infectious diseases fellow) and attending physician.
Here we present a retrospective study of the UAB 1917 Clinic Cohort evaluating antiretroviral-naïve patients initiating ART between January 1, 2000 and December 31, 2006. Patients were categorized into 2 groups: those initiating ART through a clinical trial and those starting treatment through routine care. A comparison of viral load and CD4 outcomes between these groups, efficacy in RCTs vs. effectiveness in routine care, was the primary focus of this study. Patients whose initial ART regimen lasted longer than 14 days were included.
Independent variables previously reported [15, 16] to impact virologic outcomes were chosen a priori and included socio-demographic characteristics (age, sex, race, HIV risk factor, and health insurance status), psychosocial information (history of affective mental disorder depression, anxiety or bipolar disease; alcohol abuse; and substance abuse), and baseline laboratory values [CD4 cell count and plasma HIV viral load (VL, HIV RNA in copies/mL)]. Outcome measures included plasma HIV virologic failure (VL>50 copies/mL) and change from baseline CD4 cell count following ART initiation at 6- and 12-month time points (measure closest to time point in a ± 90 day window was utilized).
Study variables were evaluated using descriptive statistics to determine the distributions of variables among patients who were treated through routine care vs. those who received ART through a clinical trial. Bivariate analyses were utilized to identify independent variables associated with clinical trial enrollment. Student’s t tests and χ2 tests were applied for continuous and categorical variables, respectively. Univariate and multivariable logistic regression models were fit to determine factors associated with virologic failure at 6 and 12 months after ART initiation. Univariate and multivariable linear regression models evaluated factors associated with change from baseline CD4 value after 6 and 12 months of therapy. Primary analyses included only patients with available lab measures at the 6- and 12-month time points, and those with missing data were excluded analytically (missing=missing).
To investigate the potential impact of missing data on study outcomes, sensitivity analyses were conducted for VL and CD4 endpoints at both 6 and 12 months. For those with missing viral load values, single imputation methods were employed to assign outcomes . Missing viral load outcomes were based upon predicted probabilities of virologic failure derived from a multivariable model that included patients with available measures. A cut-point for assignment of virologic failure was selected erring on the side of mis-classification of patients with missing viral load data as treatment failures (>50 copies/mL). For missing CD4 results, the last value recorded was carried forward for sensitivity analyses. All statistical analyses were performed using SAS Software, version 9.1.3 (SAS Institute) and statistical significance was defined as p<0.05.
Among 570 ART-naïve patients starting therapy between January 1, 2000 and December 31, 2006, 21% (n=121) were treated through a clinical trial and 79% (n=449) through routine care. Patients participated in 13 clinical trials during the study period, including four Adult AIDS Clinical Trial Group (ACTG) studies, which enrolled 86 of the 121 patients (71%) treated through RCTs (Table 1). Overall, most patients were between the ages of 30 and 49 (66%), male (77%), black (54%), had no health insurance (37%), and were men who have sex with men (51%). Baseline CD4 values were <200 cells/mm3 in 56% of patients, while a baseline VL <100,000 copies/mL was found in 63% of individuals. Patient histories included diagnoses of affective mental health disorders in 47%, substance abuse in 23%, alcohol abuse in 16% and opportunistic infections in 31%. The most commonly used third drug was a non-nucleoside reverse transcriptase inhibitor (66%) (Table 2).
In bivariate analysis, clinical trial enrollment was more common in patients with higher baseline CD4 values (CD4>200 cells/mm3; clinical trial 61% vs. routine care 40%). Black patients were significantly less likely to participate in clinical trials (p<0.001). HIV risk factor impacted study enrollment as well; clinical trial MSM (61%), heterosexual (31%) vs. routine care MSM (49%), heterosexual (44%); p=0.04. However, patient age, sex, baseline VL value, insurance status, affective mental health disorders, substance abuse and alcohol abuse were not associated with clinical trial enrollment (Table 2).
Among patients with available viral load measures at 6 months, 66% of those treated through routine care and 71% of those treated through clinical trials achieved virologic suppression (VL<50 copies/mL); at 12 months, 67% and 73% achieved virologic suppression, respectively. In primary multivariable analysis (missing=missing; Table 3a), a statistically significant association between method of ART receipt (routine care vs. clinical trial) and virologic failure was not observed at either time point [routine care vs. clinical trial (referent) 6-month OR=1.00, 95%CI=0.54–1.86; 12-month OR=1.56, 95%CI=0.80–3.05]. Six- and 12-month virologic failure was associated with Black race (6-month OR=1.73, 95%CI=1.07–2.82; 12-month OR=2.11, 95%CI=1.27–3.53) and baseline VL>100,000 copies/mL (6-month OR=2.51, 95%CI=1.58–4.01; 12-month OR=1.65, 95%CI=1.01–2.71). Compared to patients with private health insurance, those with public health insurance had higher odds of virologic failure at 6 months (OR=2.06; 95%CI=1.07–3.95), but not at 12 months (OR=1.29; 95%CI=0.66–2.55). When compared to NNRTIs, only un-boosted protease inhibitors had higher odds of 12-month virologic failure (OR=5.24, 95%CI=2.30–11.92). No other study variables were significantly associated with 6- or 12-month virologic failure in primary analyses.
Sensitivity analyses utilizing imputation to assign virologic outcomes to patients with missing values were performed (Table 3b). In multivariable sensitivity analysis, method of ART receipt (routine care vs. clinical trial) was not associated with virologic failure at 6 months (OR=1.22; 95%CI=0.68–2.19). Though not statistically significant, patients receiving ART through routine care had a trend toward increased odds of virologic failure at 12 months (OR=1.77; 95%CI=0.98–3.23). Additional sensitivity analyses using a missing=failure approach yielded largely consistent findings, although relative to the primary sensitivity analyses, slightly higher (and statistically significant) odds of virologic failure (OR=2.10; 95%CI=1.21–3.66) were observed in the routine care group at 12 months owing to a higher proportion of patients with missing values (data not shown).
The increased odds of virologic failure associated with Black race as well as the use of an unboosted PI (vs. NNRTI ) as a third drug, and the lack of statistically significant associations with age, gender, and history of mental health disorder, substance abuse, or alcohol abuse observed in primary analyses were consistent in sensitivity analyses (Table 3b).
Finally, univariate and multivariable linear regression analyses of factors associated with 6- and 12-month change from baseline CD4 count value were modeled (missing=missing; Table 4a). Baseline VL>100,000 copies/mL was associated with a significantly greater increase in CD4 count (6-month p<0.001, 12-month p=0.03). Twelve months after initiation of ART, no other factors were associated with a difference in CD4 response. Notably, similar CD4 count responses were observed in patients treated through a clinical trial and those treated through routine care. Sensitivity analyses (last value carried forward, Table 4b) of CD4 outcomes yielded findings similar to primary analyses.
Among HIV-infected patients receiving care at an academic HIV clinic in the Southeastern US, our primary analysis revealed similar virologic suppression (<50 copies/mL) and CD4 count responses in ART-naïve patients initiating treatment through a clinical trial versus routine care. Though the efficacy vs. effectiveness relationship has been examined thoroughly in cardiac care [2, 4, 5], substance abuse programs , and psychotherapy [9, 10, 19], it has been notably understudied in HIV/AIDS therapy . A comparison of viral load suppression, CD4 responses and mortality among patients receiving the same protease inhibitor regimens through the Danish Protease Inhibitor Study clinical trial and routine care showed trial participants had better responses to ART . In contrast, we found that 6- and 12-month virologic failure and CD4 count response were not significantly different between patients receiving ART through a clinical trial and those receiving treatment through routine care in our study (Tables 3a and and4a4a).
This study also sought to characterize factors associated with clinical trial enrollment in an HIV-infected cohort. Consistent with prior findings in other specialties [7, 8, 20, 21], and with earlier studies in HIV , we found that Blacks were less likely to participate in clinical trials compared to Whites (p<0.001; Table 2). Previously-identified factors that may contribute to these findings include mistrust of physicians and researchers [20, 22–27], patient fears (being treated as guinea pigs, purposeful infection, past history such as the Tuskegee syphilis study, etc.) [22–24, 26–29], and inequality in requests for research participation among racial/ethnic minorities [21, 24, 29–32]. In addition to underrepresentation in clinical trial participation, racial disparities in viral load outcomes were also observed. Black race was associated with increased odds of virologic failure in our population at both 6 and 12 months in primary and sensitivity analyses (Tables 3a and 3b). Bivariate comparisons of sociodemographic and clinical characteristics among patients with missing vs. available VL and CD4 values in both the routine care and clinical trial groups showed a statistically significant increase in the frequency of missing data in Black patients receiving ART through routine care at both 6 and 12 months (data not shown). It has been proposed that limited access to health care and increased frequency of missed clinic appointments may contribute to the poor clinical outcomes observed in Black patients with HIV [11, 13, 33, 34]; these factors may also impact the availability of laboratory measures.
We found individuals with public health insurance were more likely than those with private insurance to experience 6-month virologic failure. These findings identify another vulnerable and underserved group at risk for worse health outcomes. Consistent ART receipt and adherence in this group of lower socioeconomic status may be complicated by gaps in coverage imposed by public insurance programs  and the need to balance the costs of therapy for an initially asymptomatic illness with other economic priorities and competing needs. Health care system reforms facilitating the acquisition and consistent receipt of therapy in vulnerable populations with limited access to health care are an important prerogative.
Regimen and clinical characteristics associated with virologic failure were also identified. Patients with drug regimens including un-boosted protease inhibitors had a higher rate of virologic failure, a result not surprising given the multitude of data illustrating the poor outcomes of un-boosted protease inhibitors compared to other ART strategies (Tables 3a and 3b) [36–38]. Elevated baseline VL has also been linked to increased risk of subsequent virologic failure [11, 39, 40], a finding echoed by our study. With regards to analyses concerning change from initial CD4 value, only baseline VL>100,000 copies/mL was associated with a statistically significant CD4 count change at 12 months (Tables 4a and 4b).
In sensitivity analyses of virologic outcomes using imputation methods, significant differences in 6-month virologic failure were not observed between patients treated in clinical trials vs. routine care, in accordance with primary analyses (Table 3b). However, at 12 months, ART receipt through routine care was associated with a trend toward increased odds of virologic failure (OR=1.77; 95%CI=0.98–3.23). We suspect this trend may reflect the greater frequency of missing viral load values in the routine care group [routine care 28% (n=126); clinical trial group 11% (n=13)], which may relate to several factors. Volunteer and selection bias for clinical trial participation may result in a sample that is more likely to attend clinic appointments and have laboratory measures obtained relative to the routine care population. Study selection criteria are known to contribute to differences in clinical trial enrollment rates among different groups [2, 5, 6, 8, 9, 22], and may have played a role in the current study. Participation in a clinical trial also entails close follow-up with study personnel. Such close monitoring and aggressive rescheduling after missed study visits is beyond the capacity of our clinic for all patients in routine clinical care. In summary, regarding efficacy vs. effectiveness in HIV therapy, 6-month virologic outcomes were consistent in primary and sensitivity analyses, though a trend toward differences in viral load outcomes appeared at 12 months in sensitivity analyses. By utilizing two strategies to evaluate the impact of missing data on virologic outcomes, a more complete understanding of the efficacy-effectiveness gap surfaces, underscoring the importance of a comprehensive approach.
Our findings should be interpreted with respect to the limitations of our study. As a retrospective study from a single HIV cohort, our findings may not be generalizable to other national or international settings, though our analysis may provide insights applicable to such settings. As with all observational studies, we were able to identify associations but cannot attribute causality. While we controlled for measured confounders using multivariable models, there is potential for unmeasured confounding, inherent to observational studies, which may impact outcomes interpretation. Other studies have implicated patient education level in contributing to clinical trial participation [7, 8, 20, 22–26, 28, 29], but we were unable to systematically ascertain this variable in our sample. Because of our modest sample size, we were able to assess treatment modality (clinical trial vs. routine care) but had insufficient numbers to assess efficacy vs. effectiveness at the regimen level. Such analyses are on-going through larger, multi-site cohort collaborations.
A notable strength of this study is the use of multiple strategies to analyze the impact of missing data on outcomes, enabling a more comprehensive understanding of the efficacy vs. effectiveness relationship within the constraints of the measurements available. Many prior studies of HIV outcomes have neither explicitly stated the handling of missing data nor evaluated the impact of missing data on outcomes interpretation.
In conclusion, clinical research studies have played a vital role in the improvement of HIV treatment and outcomes. However, it is critical to evaluate both the efficacy and effectiveness of therapy to ensure that the results obtained from clinical trials are generalizable to other populations treated through routine care. In primary analyses evaluating patients with available measures, we found similar 6- and 12-month virologic failure and CD4 count responses among antiretroviral-naïve patients treated through routine care vs. those participating in clinical trials. These findings provide insight into the efficacy-effectiveness relationship of antiretroviral therapy for HIV infection, suggesting similar first year responses are observed in treatment-naïve patients starting ART in clinical trials vs. routine care in the contemporary treatment era.
The authors would like to thank the UAB 1917 HIV/AIDS Clinic Cohort Observational Database project, the UAB 1917 Clinic medical records department (Robin, Juwata, Christine) without whose assistance this project would not have been possible, the UAB Center for AIDS Research (grant P30-AI27767), CFAR-Network of Integrated Clinical Systems, CNICS (grant 1 R24 AI067039-1), and the Mary Fisher CARE Fund for their assistance and support of this project.
Sources of Support: The UAB 1917 HIV/AIDS Clinic Cohort Observational Database project receives financial support from the following: UAB Center for AIDS Research (grant P30-AI27767), CFAR-Network of Integrated Clinical Systems (CNICS; grant 1 R24 AI067039-1), and the Mary Fisher CARE Fund. Michael Mugavero is supported by Grant Number K23MH082641 from the National Institute of Mental Health. James Willig is supported by the Ruth L. Kirschstein National Research Service Award (grant 5T32AI52069). The content herein is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health, the National Institutes of Health, or any other agency providing support for this study.
Michael S. Saag, Michael J. Mugavero, James H. Willig, James L. Raper, Paul Goepfert, Jeroan J. Allison, Mirjam-Colette Kempf, Joseph E. Schumacher, Inmaculada B. Aban
Hui-Yi Lin, Maria Pisu, Linda Moneyham, David Vance, Susan L. Davies, Eta Berner, Edward Acosta, Jennifer King, Richard A. Kaslow, Eric Chamot, Andrew O. Westfall
Research Support Team
Karen Savage, Christa Nevin, Frances B. Walton, Malcolm L. Marler, Sarah Lawrence, Barbara Files-Kennedy, D. Scott Batey
Manoj A. Patil, Mohit Varshney, Eugene Gibson, Suneetha Thogaripally, Alfredo Guzman, Dustin Rinehart, Ridha T. Bagana
Justin S. Routman, James McKinnell, Paula Seal, Noah Godwin, Mary Orr, Michael Kozak, Tyler Tate, Sarah Abroms
Data Presented in part at: 15th Conference on Retroviruses and Opportunistic Infections, Boston, Massachusetts, USA, from February 3–6, 2008.
Disclosure Statement: J.H.W. has received research funding and/or consulted for: Bristol-Myers Squibb, Gilead, Merck and Tibotec. M. J. M. has received recent research funding and/or consulted for: Tibotec Therapeutics, Bristol-Myers Squibb and Gilead. M.S.S. has received recent research funding or consulted for: Adrea Pharmaceuticals, Avexa, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead, GlaxoSmithKline, Merck, Monogram Biosciences, Panacos, Pfizer, Progenics, Roche, Serono, Tanox, Tibotec, Trimeris, and Vertex. All other authors: no conflicts of interest related to this manuscript.