Comparative efficacy versus effectiveness of initial antiretroviral therapy in clinical trials versus routine care

doi:10.1086/650004

Clin Infect Dis. Author manuscript; available in PMC 2011 February 15.

Published in final edited form as:

Clin Infect Dis. 2010 February 15; 50(4): 574–584.

doi: 10.1086/650004

PMCID: PMC2848720

NIHMSID: NIHMS159641

Comparative efficacy versus effectiveness of initial antiretroviral therapy in clinical trials versus routine care

Justin S. Routman, BA,¹^* James H. Willig, MD,²^* Andrew O. Westfall, MS,³ Sarah R. Abroms, BA,² Mohit Varshney, MD,² Sunil Adusumilli, MD,² Jeroan J. Allison, MD, MSc,⁴ Karen G. Savage, BSN, CCRC,² Michael S. Saag, MD,² and Michael J. Mugavero, MD, MHSc²

¹ University of Alabama School of Medicine, University of Alabama at Birmingham

² Division of Infectious Diseases, Department of Medicine, University of Alabama at Birmingham

³ School of Public Health, Department of Biostatistics, University of Alabama at Birmingham

⁴ Division of Preventive Medicine, Department of Medicine, University of Alabama at Birmingham

Corresponding Author: James H. Willig, MD, CCB 178, 908 20^th St So, Birmingham, AL, 35294-2050; Phone: (205) 996-5753; Fax: (205) 975-8273; Email: jwillig/at/uab.edu

^*Both authors contributed equally to the elaboration of this manuscript.

The publisher's final edited version of this article is available free at Clin Infect Dis

See other articles in PMC that cite the published article.

Summary

The generalizability of clinical trial findings (efficacy) to routine care (effectiveness) may be limited. The present study found similar first year virologic and CD4 outcomes among antiretroviral-naïve patients treated through routine care vs. those participating in clinical trials.

Background

The generalizability of clinical trial findings (efficacy) to routine care (effectiveness) may be limited due to study eligibility criteria and volunteer bias. While well chronicled in many conditions, the efficacy vs. effectiveness of antiretroviral therapy (ART) remains understudied.

Methods

A retrospective study of the UAB 1917 Clinic Cohort evaluated naïve patients starting ART between 1/1/00–12/31/06. Patients received ART through clinical trials or routine care. Multivariable logistic and linear regression models were fit to evaluate factors associated with virologic failure (VF=VL>50 copies/mL) and change from baseline CD4 count 6 and 12 months after ART initiation. Sensitivity analyses evaluated the impact of missing data on outcomes.

Results

Among 570 patients starting ART during the study period, 121 (21%) enrolled in clinical trials vs. 449 (79%) receiving ART via routine care. ART receipt through routine care was not associated with VF at either 6 (OR=1.00;95%CI=0.54–1.86) or 12 (OR=1.56;95%CI=0.80–3.05) months in primary analyses. No significant differences in CD4 count responses at 6 and 12 months were observed.

Conclusions

Though marked differences in efficacy vs. effectiveness have been observed in the therapeutic outcomes of other conditions, our analyses found no evidence of such divergence among our patients initiating antiretroviral therapy for HIV.

Keywords: Efficacy, effectiveness, ART, HIV

Introduction

Randomized clinical trials (RCTs) are the cornerstone of Level I evidence-based medicine treatment recommendations and provide the highest level of evidence [1]. However, some RCT-tested interventions have not performed as well when implemented in routine care settings [2–5]. Factors such as selection bias introduced by trial eligibility criteria and volunteer bias among participants choosing to participate in research studies have been linked to this discrepancy [2, 4–10]. Selected patient samples may show improved treatment outcomes in trials (efficacy) when compared to the more heterogeneous population treated through routine care (effectiveness), raising concerns about the generalizability of RCT findings to routine care settings.

Efforts to characterize differential efficacy versus effectiveness of treatments have been undertaken in many medical conditions [2, 4–6, 9, 10], yet this relationship regarding antiretroviral therapy (ART) for HIV/AIDS has been notably understudied, particularly in the contemporary ART era [11]. While numerous studies have separately evaluated either the efficacy or the effectiveness of initial ART regimens when used in RCTs and routine care, respectively, relatively few have studied the comparative effectiveness of treatment modality (RCT vs. routine care) on outcomes among patients starting ART in the same clinical setting. Therefore, we conducted a retrospective study to evaluate the impact of receiving initial ART through a clinical trial vs. routine care on short-term viral load and CD4 outcomes among ART-naïve individuals initiating therapy. Because treatment-naïve ART studies are commonly available, are ingrained in the culture of HIV care at many treatment centers, and provide a means to access medications and laboratories at little to no cost to patients, we hypothesized that volunteer bias would be less apparent in an HIV-infected cohort relative to other diseases. Accordingly, we posited the sociodemographic composition of those treated through clinical trials would be reflective of the larger clinic population and mirror the characteristics of those receiving ART through routine care. We further hypothesized that similar virologic and CD4 outcomes would be observed between patients treated in clinical trials and routine care due to the similarities in the patient populations.

Methods

Sample and procedure

Since 1988, the University of Alabama at Birmingham (UAB) 1917 HIV/AIDS Clinic (1917 Clinic) has provided HIV care for over 6,000 HIV-infected individuals. The UAB 1917 HIV/AIDS Clinic Cohort Database Project (UAB 1917 Clinic Cohort), recently recognized for excellence in information integrity (http://www.eiiaward.org/2007_winners), is a 100% quality controlled, IRB-approved prospective clinical cohort study that includes detailed sociodemographic, psychosocial and clinical information from HIV-infected patients receiving primary HIV and subspecialty care at the clinic (http://www.uab1917cliniccohort.org). The 1917 Clinic utilizes a locally programmed electronic medical record (EMR) that imports laboratory values from the central UAB laboratory, requires electronic prescriptions for all medications and contains detailed encounter notes. Both the UAB 1917 Clinic Cohort and local EMR have been described in detail elsewhere [12–14].

A dedicated clinical trials program and staff has been part of the 1917 Clinic since inception. At our center, RCTs for antiretroviral-naïve patients are frequently available and open for enrollment. Prior to study enrollment, providers ascertain patients’ willingness to learn more about clinical trial participation and refer interested patients to clinical trial study nurses who screen patients and begin the informed consent process. Once enrolled in a research study, patients receive additional follow-up from study personnel (nurses, mid-level providers and physicians) as determined by specific study protocols, in addition to regular outpatient care at the clinic. Patients initiating ART through routine care meet with a clinic pharmacist to discuss their regimen. Otherwise, no specific treatment protocol is in place and all clinic and laboratory follow-up is at the discretion of the primary provider (a nurse practitioner or infectious diseases fellow) and attending physician.

Here we present a retrospective study of the UAB 1917 Clinic Cohort evaluating antiretroviral-naïve patients initiating ART between January 1, 2000 and December 31, 2006. Patients were categorized into 2 groups: those initiating ART through a clinical trial and those starting treatment through routine care. A comparison of viral load and CD4 outcomes between these groups, efficacy in RCTs vs. effectiveness in routine care, was the primary focus of this study. Patients whose initial ART regimen lasted longer than 14 days were included.

Independent variables previously reported [15, 16] to impact virologic outcomes were chosen a priori and included socio-demographic characteristics (age, sex, race, HIV risk factor, and health insurance status), psychosocial information (history of affective mental disorder depression, anxiety or bipolar disease; alcohol abuse; and substance abuse), and baseline laboratory values [CD4 cell count and plasma HIV viral load (VL, HIV RNA in copies/mL)]. Outcome measures included plasma HIV virologic failure (VL>50 copies/mL) and change from baseline CD4 cell count following ART initiation at 6- and 12-month time points (measure closest to time point in a ± 90 day window was utilized).

Statistical analyses

Study variables were evaluated using descriptive statistics to determine the distributions of variables among patients who were treated through routine care vs. those who received ART through a clinical trial. Bivariate analyses were utilized to identify independent variables associated with clinical trial enrollment. Student’s t tests and χ² tests were applied for continuous and categorical variables, respectively. Univariate and multivariable logistic regression models were fit to determine factors associated with virologic failure at 6 and 12 months after ART initiation. Univariate and multivariable linear regression models evaluated factors associated with change from baseline CD4 value after 6 and 12 months of therapy. Primary analyses included only patients with available lab measures at the 6- and 12-month time points, and those with missing data were excluded analytically (missing=missing).

To investigate the potential impact of missing data on study outcomes, sensitivity analyses were conducted for VL and CD4 endpoints at both 6 and 12 months. For those with missing viral load values, single imputation methods were employed to assign outcomes [17]. Missing viral load outcomes were based upon predicted probabilities of virologic failure derived from a multivariable model that included patients with available measures. A cut-point for assignment of virologic failure was selected erring on the side of mis-classification of patients with missing viral load data as treatment failures (>50 copies/mL). For missing CD4 results, the last value recorded was carried forward for sensitivity analyses. All statistical analyses were performed using SAS Software, version 9.1.3 (SAS Institute) and statistical significance was defined as p<0.05.

Results

Among 570 ART-naïve patients starting therapy between January 1, 2000 and December 31, 2006, 21% (n=121) were treated through a clinical trial and 79% (n=449) through routine care. Patients participated in 13 clinical trials during the study period, including four Adult AIDS Clinical Trial Group (ACTG) studies, which enrolled 86 of the 121 patients (71%) treated through RCTs (Table 1). Overall, most patients were between the ages of 30 and 49 (66%), male (77%), black (54%), had no health insurance (37%), and were men who have sex with men (51%). Baseline CD4 values were <200 cells/mm³ in 56% of patients, while a baseline VL <100,000 copies/mL was found in 63% of individuals. Patient histories included diagnoses of affective mental health disorders in 47%, substance abuse in 23%, alcohol abuse in 16% and opportunistic infections in 31%. The most commonly used third drug was a non-nucleoside reverse transcriptase inhibitor (66%) (Table 2).

Table 1

Treatment-naive trials and number of patients enrolled at the University of Alabama at Birmingham 1917 HIV/AIDS Clinic between January 1, 2000 and December 31, 2006.

Table 2

Baseline characteristics and bivariate analysis of factors associated with clinical trial participation among 570 ART-naive patients initiating therapy at the University of Alabama at Birmingham 1917 HIV/AIDS Clinic between January 1, 2000 and December (more ...)

In bivariate analysis, clinical trial enrollment was more common in patients with higher baseline CD4 values (CD4>200 cells/mm³; clinical trial 61% vs. routine care 40%). Black patients were significantly less likely to participate in clinical trials (p<0.001). HIV risk factor impacted study enrollment as well; clinical trial MSM (61%), heterosexual (31%) vs. routine care MSM (49%), heterosexual (44%); p=0.04. However, patient age, sex, baseline VL value, insurance status, affective mental health disorders, substance abuse and alcohol abuse were not associated with clinical trial enrollment (Table 2).

Among patients with available viral load measures at 6 months, 66% of those treated through routine care and 71% of those treated through clinical trials achieved virologic suppression (VL<50 copies/mL); at 12 months, 67% and 73% achieved virologic suppression, respectively. In primary multivariable analysis (missing=missing; Table 3a), a statistically significant association between method of ART receipt (routine care vs. clinical trial) and virologic failure was not observed at either time point [routine care vs. clinical trial (referent) 6-month OR=1.00, 95%CI=0.54–1.86; 12-month OR=1.56, 95%CI=0.80–3.05]. Six- and 12-month virologic failure was associated with Black race (6-month OR=1.73, 95%CI=1.07–2.82; 12-month OR=2.11, 95%CI=1.27–3.53) and baseline VL>100,000 copies/mL (6-month OR=2.51, 95%CI=1.58–4.01; 12-month OR=1.65, 95%CI=1.01–2.71). Compared to patients with private health insurance, those with public health insurance had higher odds of virologic failure at 6 months (OR=2.06; 95%CI=1.07–3.95), but not at 12 months (OR=1.29; 95%CI=0.66–2.55). When compared to NNRTIs, only un-boosted protease inhibitors had higher odds of 12-month virologic failure (OR=5.24, 95%CI=2.30–11.92). No other study variables were significantly associated with 6- or 12-month virologic failure in primary analyses.

Table 3

Sensitivity analyses utilizing imputation to assign virologic outcomes to patients with missing values were performed (Table 3b). In multivariable sensitivity analysis, method of ART receipt (routine care vs. clinical trial) was not associated with virologic failure at 6 months (OR=1.22; 95%CI=0.68–2.19). Though not statistically significant, patients receiving ART through routine care had a trend toward increased odds of virologic failure at 12 months (OR=1.77; 95%CI=0.98–3.23). Additional sensitivity analyses using a missing=failure approach yielded largely consistent findings, although relative to the primary sensitivity analyses, slightly higher (and statistically significant) odds of virologic failure (OR=2.10; 95%CI=1.21–3.66) were observed in the routine care group at 12 months owing to a higher proportion of patients with missing values (data not shown).

The increased odds of virologic failure associated with Black race as well as the use of an unboosted PI (vs. NNRTI ) as a third drug, and the lack of statistically significant associations with age, gender, and history of mental health disorder, substance abuse, or alcohol abuse observed in primary analyses were consistent in sensitivity analyses (Table 3b).

Finally, univariate and multivariable linear regression analyses of factors associated with 6- and 12-month change from baseline CD4 count value were modeled (missing=missing; Table 4a). Baseline VL>100,000 copies/mL was associated with a significantly greater increase in CD4 count (6-month p<0.001, 12-month p=0.03). Twelve months after initiation of ART, no other factors were associated with a difference in CD4 response. Notably, similar CD4 count responses were observed in patients treated through a clinical trial and those treated through routine care. Sensitivity analyses (last value carried forward, Table 4b) of CD4 outcomes yielded findings similar to primary analyses.

Table 4

Discussion

Among HIV-infected patients receiving care at an academic HIV clinic in the Southeastern US, our primary analysis revealed similar virologic suppression (<50 copies/mL) and CD4 count responses in ART-naïve patients initiating treatment through a clinical trial versus routine care. Though the efficacy vs. effectiveness relationship has been examined thoroughly in cardiac care [2, 4, 5], substance abuse programs [18], and psychotherapy [9, 10, 19], it has been notably understudied in HIV/AIDS therapy [11]. A comparison of viral load suppression, CD4 responses and mortality among patients receiving the same protease inhibitor regimens through the Danish Protease Inhibitor Study clinical trial and routine care showed trial participants had better responses to ART [3]. In contrast, we found that 6- and 12-month virologic failure and CD4 count response were not significantly different between patients receiving ART through a clinical trial and those receiving treatment through routine care in our study (Tables 3a and and4a4a).

This study also sought to characterize factors associated with clinical trial enrollment in an HIV-infected cohort. Consistent with prior findings in other specialties [7, 8, 20, 21], and with earlier studies in HIV [22], we found that Blacks were less likely to participate in clinical trials compared to Whites (p<0.001; Table 2). Previously-identified factors that may contribute to these findings include mistrust of physicians and researchers [20, 22–27], patient fears (being treated as guinea pigs, purposeful infection, past history such as the Tuskegee syphilis study, etc.) [22–24, 26–29], and inequality in requests for research participation among racial/ethnic minorities [21, 24, 29–32]. In addition to underrepresentation in clinical trial participation, racial disparities in viral load outcomes were also observed. Black race was associated with increased odds of virologic failure in our population at both 6 and 12 months in primary and sensitivity analyses (Tables 3a and 3b). Bivariate comparisons of sociodemographic and clinical characteristics among patients with missing vs. available VL and CD4 values in both the routine care and clinical trial groups showed a statistically significant increase in the frequency of missing data in Black patients receiving ART through routine care at both 6 and 12 months (data not shown). It has been proposed that limited access to health care and increased frequency of missed clinic appointments may contribute to the poor clinical outcomes observed in Black patients with HIV [11, 13, 33, 34]; these factors may also impact the availability of laboratory measures.

We found individuals with public health insurance were more likely than those with private insurance to experience 6-month virologic failure. These findings identify another vulnerable and underserved group at risk for worse health outcomes. Consistent ART receipt and adherence in this group of lower socioeconomic status may be complicated by gaps in coverage imposed by public insurance programs [35] and the need to balance the costs of therapy for an initially asymptomatic illness with other economic priorities and competing needs. Health care system reforms facilitating the acquisition and consistent receipt of therapy in vulnerable populations with limited access to health care are an important prerogative.

Regimen and clinical characteristics associated with virologic failure were also identified. Patients with drug regimens including un-boosted protease inhibitors had a higher rate of virologic failure, a result not surprising given the multitude of data illustrating the poor outcomes of un-boosted protease inhibitors compared to other ART strategies (Tables 3a and 3b) [36–38]. Elevated baseline VL has also been linked to increased risk of subsequent virologic failure [11, 39, 40], a finding echoed by our study. With regards to analyses concerning change from initial CD4 value, only baseline VL>100,000 copies/mL was associated with a statistically significant CD4 count change at 12 months (Tables 4a and 4b).

In sensitivity analyses of virologic outcomes using imputation methods, significant differences in 6-month virologic failure were not observed between patients treated in clinical trials vs. routine care, in accordance with primary analyses (Table 3b). However, at 12 months, ART receipt through routine care was associated with a trend toward increased odds of virologic failure (OR=1.77; 95%CI=0.98–3.23). We suspect this trend may reflect the greater frequency of missing viral load values in the routine care group [routine care 28% (n=126); clinical trial group 11% (n=13)], which may relate to several factors. Volunteer and selection bias for clinical trial participation may result in a sample that is more likely to attend clinic appointments and have laboratory measures obtained relative to the routine care population. Study selection criteria are known to contribute to differences in clinical trial enrollment rates among different groups [2, 5, 6, 8, 9, 22], and may have played a role in the current study. Participation in a clinical trial also entails close follow-up with study personnel. Such close monitoring and aggressive rescheduling after missed study visits is beyond the capacity of our clinic for all patients in routine clinical care. In summary, regarding efficacy vs. effectiveness in HIV therapy, 6-month virologic outcomes were consistent in primary and sensitivity analyses, though a trend toward differences in viral load outcomes appeared at 12 months in sensitivity analyses. By utilizing two strategies to evaluate the impact of missing data on virologic outcomes, a more complete understanding of the efficacy-effectiveness gap surfaces, underscoring the importance of a comprehensive approach.

Our findings should be interpreted with respect to the limitations of our study. As a retrospective study from a single HIV cohort, our findings may not be generalizable to other national or international settings, though our analysis may provide insights applicable to such settings. As with all observational studies, we were able to identify associations but cannot attribute causality. While we controlled for measured confounders using multivariable models, there is potential for unmeasured confounding, inherent to observational studies, which may impact outcomes interpretation. Other studies have implicated patient education level in contributing to clinical trial participation [7, 8, 20, 22–26, 28, 29], but we were unable to systematically ascertain this variable in our sample. Because of our modest sample size, we were able to assess treatment modality (clinical trial vs. routine care) but had insufficient numbers to assess efficacy vs. effectiveness at the regimen level. Such analyses are on-going through larger, multi-site cohort collaborations.

A notable strength of this study is the use of multiple strategies to analyze the impact of missing data on outcomes, enabling a more comprehensive understanding of the efficacy vs. effectiveness relationship within the constraints of the measurements available. Many prior studies of HIV outcomes have neither explicitly stated the handling of missing data nor evaluated the impact of missing data on outcomes interpretation.

In conclusion, clinical research studies have played a vital role in the improvement of HIV treatment and outcomes. However, it is critical to evaluate both the efficacy and effectiveness of therapy to ensure that the results obtained from clinical trials are generalizable to other populations treated through routine care. In primary analyses evaluating patients with available measures, we found similar 6- and 12-month virologic failure and CD4 count responses among antiretroviral-naïve patients treated through routine care vs. those participating in clinical trials. These findings provide insight into the efficacy-effectiveness relationship of antiretroviral therapy for HIV infection, suggesting similar first year responses are observed in treatment-naïve patients starting ART in clinical trials vs. routine care in the contemporary treatment era.

Acknowledgments

The authors would like to thank the UAB 1917 HIV/AIDS Clinic Cohort Observational Database project, the UAB 1917 Clinic medical records department (Robin, Juwata, Christine) without whose assistance this project would not have been possible, the UAB Center for AIDS Research (grant P30-AI27767), CFAR-Network of Integrated Clinical Systems, CNICS (grant 1 R24 AI067039-1), and the Mary Fisher CARE Fund for their assistance and support of this project.

Sources of Support: The UAB 1917 HIV/AIDS Clinic Cohort Observational Database project receives financial support from the following: UAB Center for AIDS Research (grant P30-AI27767), CFAR-Network of Integrated Clinical Systems (CNICS; grant 1 R24 AI067039-1), and the Mary Fisher CARE Fund. Michael Mugavero is supported by Grant Number K23MH082641 from the National Institute of Mental Health. James Willig is supported by the Ruth L. Kirschstein National Research Service Award (grant 5T32AI52069). The content herein is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health, the National Institutes of Health, or any other agency providing support for this study.

1917 Clinic Cohort Team

Steering Committee

Michael S. Saag, Michael J. Mugavero, James H. Willig, James L. Raper, Paul Goepfert, Jeroan J. Allison, Mirjam-Colette Kempf, Joseph E. Schumacher, Inmaculada B. Aban

Faculty Investigators

Hui-Yi Lin, Maria Pisu, Linda Moneyham, David Vance, Susan L. Davies, Eta Berner, Edward Acosta, Jennifer King, Richard A. Kaslow, Eric Chamot, Andrew O. Westfall

Research Support Team

Karen Savage, Christa Nevin, Frances B. Walton, Malcolm L. Marler, Sarah Lawrence, Barbara Files-Kennedy, D. Scott Batey

Informatics Team

Manoj A. Patil, Mohit Varshney, Eugene Gibson, Suneetha Thogaripally, Alfredo Guzman, Dustin Rinehart, Ridha T. Bagana

Current Trainees

Justin S. Routman, James McKinnell, Paula Seal, Noah Godwin, Mary Orr, Michael Kozak, Tyler Tate, Sarah Abroms

Footnotes

Data Presented in part at: 15^th Conference on Retroviruses and Opportunistic Infections, Boston, Massachusetts, USA, from February 3–6, 2008.

Disclosure Statement: J.H.W. has received research funding and/or consulted for: Bristol-Myers Squibb, Gilead, Merck and Tibotec. M. J. M. has received recent research funding and/or consulted for: Tibotec Therapeutics, Bristol-Myers Squibb and Gilead. M.S.S. has received recent research funding or consulted for: Adrea Pharmaceuticals, Avexa, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead, GlaxoSmithKline, Merck, Monogram Biosciences, Panacos, Pfizer, Progenics, Roche, Serono, Tanox, Tibotec, Trimeris, and Vertex. All other authors: no conflicts of interest related to this manuscript.

References

1. Phillips B, Ball C, Sackett D, et al. Oxford Centre for Evidence-based Medicine levels of evidence. Oxford: Centre for Evidence-based Medicine; Mar2009. [accessed July 5, 2009]. http://www.cebm.net/levels_of_evidence.asp.

2. Bahit MC, Cannon CP, Antman EM, et al. Direct comparison of characteristics, treatment, and outcomes of patients enrolled versus patients not enrolled in a clinical trial at centers participating in the TIMI 9 Trial and TIMI 9 Registry. Am Heart J. 2003 Jan;145(1):109–17. [PubMed]

3. Eg Hansen AB, Gerstoft J, Kirk O, et al. Unmeasured confounding caused slightly better response to HAART within than outside a randomized controlled trial. J Clin Epidemiol. 2008 Jan;61(1):87–94. [PubMed]

4. Hordijk-Trion M, Lenzen M, Wijns W, et al. Patients enrolled in coronary intervention trials are not representative of patients in clinical practice: results from the Euro Heart Survey on Coronary Revascularization. Eur Heart J. 2006 Mar;27(6):671–8. [PubMed]

5. Steg PG, Lopez-Sendon J, Lopez de Sa E, et al. External validity of clinical trials in acute myocardial infarction. Archives of internal medicine. 2007 Jan 8;167(1):68–73. [PubMed]

6. Bauer MS, Williford WO, Dawson EE, et al. Principles of effectiveness trials and their implementation in VA Cooperative Study #430: ‘Reducing the efficacy-effectiveness gap in bipolar disorder’ J Affect Disord. 2001 Dec;67(1–3):61–78. [PubMed]

7. Bozzette SA, Berry SH, Duan N, et al. The care of HIV-infected adults in the United States. HIV Cost and Services Utilization Study Consortium. N Engl J Med. 1998 Dec 24;339(26):1897–904. [PubMed]

8. Hankins C, Lapointe N, Walmsley S. Participation in clinical trials among women living with HIV in Canada. Canadian Women’s HIV Study Group. Cmaj. 1998 Dec 1;159(11):1359–65. [PMC free article] [PubMed]

9. Nathan PE, Stuart SP, Dolan SL. Research on psychotherapy efficacy and effectiveness: between Scylla and Charybdis? Psychol Bull. 2000 Nov;126(6):964–81. [PubMed]

10. Seligman ME. The effectiveness of psychotherapy. The Consumer Reports study. Am Psychol. 1995 Dec;50(12):965–74. [PubMed]

11. Lucas GM, Chaisson RE, Moore RD. Highly active antiretroviral therapy in a large urban clinic: risk factors for virologic failure and adverse drug reactions. Ann Intern Med. 1999 Jul 20;131(2):81–7. [PubMed]

12. Chen RY, Westfall AO, Mugavero MJ, et al. Duration of highly active antiretroviral therapy regimens. Clin Infect Dis. 2003 Sep 1;37(5):714–22. [PubMed]

13. Mugavero MJ, Lin HY, Allison JJ, et al. Racial disparities in HIV virologic failure: do missed visits matter? J Acquir Immune Defic Syndr. 2009 Jan 1;50(1):100–8. [PMC free article] [PubMed]

14. Mugavero MJ, Lin HY, Willig JH, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis. 2009 Jan 15;48(2):248–56. [PMC free article] [PubMed]

15. Egger M, May M, Chene G, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. Lancet. 2002 Jul 13;360(9327):119–29. [PubMed]

16. Paredes R, Mocroft A, Kirk O, et al. Predictors of virological success and ensuing failure in HIV-positive patients starting highly active antiretroviral therapy in Europe: results from the EuroSIDA study. Archives of internal medicine. 2000 Apr 24;160(8):1123–32. [PubMed]

17. Little RJA, Rubin DB. Statistical analysis with missing data. 2. Hoboken, N.J: Wiley; 2002.

18. Hallfors D, Cho H, Sanchez V, Khatapoush S, Kim HM, Bauer D. Efficacy vs effectiveness trial results of an indicated “model” substance abuse program: implications for public health. Am J Public Health. 2006 Dec;96(12):2254–9. [PubMed]

19. Mintz J, Drake RE, Crits-Christoph P. Efficacy and effectiveness of psychotherapy: Two paradigms, one science. American Psychologist. 1996 Oct;:1084–5.

20. el-Sadr W, Capps L. The challenge of minority recruitment in clinical trials for AIDS. Jama. 1992 Feb 19;267(7):954–7. [PubMed]

21. Stone VE, Mauch MY, Steger K, Janas SF, Craven DE. Race, gender, drug use, and participation in AIDS clinical trials. Lessons from a municipal hospital cohort. J Gen Intern Med. 1997 Mar;12(3):150–7. [PMC free article] [PubMed]

22. Gifford AL, Cunningham WE, Heslin KC, et al. Participation in research and access to experimental treatments by HIV-infected patients. N Engl J Med. 2002 May 2;346(18):1373–82. [PubMed]

23. Corbie-Smith G, Thomas SB, Williams MV, Moody-Ayers S. Attitudes and beliefs of African Americans toward participation in medical research. J Gen Intern Med. 1999 Sep;14(9):537–46. [PMC free article] [PubMed]

24. Freimuth VS, Quinn SC, Thomas SB, Cole G, Zook E, Duncan T. African Americans’ views on research and the Tuskegee Syphilis Study. Soc Sci Med. 2001 Mar;52(5):797–808. [PubMed]

25. Gao X, Nau DP, Rosenbluth SA, Scott V, Woodward C. The relationship of disease severity, health beliefs and medication adherence among HIV patients. AIDS Care. 2000 Aug;12(4):387–98. [PubMed]

26. Sengupta S, Strauss RP, DeVellis R, Quinn SC, DeVellis B, Ware WB. Factors affecting African-American participation in AIDS research. J Acquir Immune Defic Syndr. 2000 Jul 1;24(3):275–84. [PubMed]

27. Green BL, Maisiak R, Wang MQ, Britt MF, Ebeling N. Participation in health education, health promotion, and health research by African Americans: Effects of the Tuskegee Syphilis Experiment. J Health Educ. 1997;28:196–201.

28. Klonoff EA, Landrine H. Do blacks believe that HIV/AIDS is a government conspiracy against them? Prev Med. 1999 May;28(5):451–7. [PubMed]

29. Thomas SB, Quinn SC. The Tuskegee Syphilis Study, 1932 to 1972: implications for HIV education and AIDS risk education programs in the black community. Am J Public Health. 1991 Nov;81(11):1498–505. [PubMed]

30. Garber M, Hanusa BH, Switzer GE, Mellors J, Arnold RM. HIV-infected African Americans are willing to participate in HIV treatment trials. J Gen Intern Med. 2007 Jan;22(1):17–42. [PMC free article] [PubMed]

31. Madge S, Mocroft A, Wilson D, et al. Participation in clinical studies among patients infected with HIV-1 in a single treatment centre over 12 years. HIV Med. 2000 Oct;1(4):212–8. [PubMed]

32. Wendler D, Kington R, Madans J, et al. Are racial and ethnic minorities less willing to participate in health research? PLoS Med. 2006 Feb;3(2):e19. [PMC free article] [PubMed]

33. Gulick RM, Ribaudo HJ, Shikuma CM, et al. Three- vs four-drug antiretroviral regimens for the initial treatment of HIV-1 infection: a randomized controlled trial. Jama. 2006 Aug 16;296(7):769–81. [PubMed]

34. Hartzell JD, Spooner K, Howard R, Wegner S, Wortmann G. Race and mental health diagnosis are risk factors for highly active antiretroviral therapy failure in a military cohort despite equal access to care. J Acquir Immune Defic Syndr. 2007 Apr 1;44(4):411–6. [PubMed]

35. Pence BW, Ostermann J, Kumar V, Whetten K, Thielman N, Mugavero MJ. The influence of psychosocial characteristics and race/ethnicity on the use, duration, and success of antiretroviral therapy. J Acquir Immune Defic Syndr. 2008 Feb 1;47(2):194–201. [PubMed]

36. Chandwani A, Shuter J. Lopinavir/ritonavir in the treatment of HIV-1 infection: a review. Ther Clin Risk Manag. 2008 Oct;4(5):1023–33. [PMC free article] [PubMed]

37. Cooper CL, van Heeswijk RP, Gallicano K, Cameron DW. A review of low-dose ritonavir in protease inhibitor combination therapy. Clin Infect Dis. 2003 Jun 15;36(12):1585–92. [PubMed]

38. Lichterfeld M, Wohrmann A, Schmeisser N, et al. Superior virological efficacy of ritonavir-boosted protease inhibitor regimens compared to single protease inhibitor therapy. Eur J Med Res. 2003 Feb 21;8(2):56–60. [PubMed]

39. Paris D, Ledergerber B, Weber R, et al. Incidence and predictors of virologic failure of antiretroviral triple-drug therapy in a community-based cohort. AIDS Res Hum Retroviruses. 1999 Dec 10;15(18):1631–8. [PubMed]

40. Tuboi SH, Harrison LH, Sprinz E, Albernaz RK, Schechter M. Predictors of virologic failure in HIV-1-infected patients starting highly active antiretroviral therapy in Porto Alegre, Brazil. J Acquir Immune Defic Syndr. 2005 Nov 1;40(3):324–8. [PubMed]