|Home | About | Journals | Submit | Contact Us | Français|
Measuring retention in HIV primary care is complex as care includes multiple visits scheduled at varying intervals over time. We evaluated six commonly used retention measures in predicting viral load (VL) suppression and the correlation among measures.
Clinic-wide patient-level data from six academic HIV clinics were used for 12-months preceding implementation of the CDC/HRSA Retention in Care intervention. Six retention measures were calculated for each patient based upon scheduled primary HIV provider visits: count and dichotomous missed visits, visit adherence, 6-month gap, 4-month visit constancy, and the HRSA HAB retention measure. Spearman correlation coefficients and separate unadjusted logistic regression models compared retention measures to one another and with 12-month VL suppression, respectively. The discriminatory capacity of each measure was assessed with the c-statistic.
Among 10,053 patients, 8,235 (82%) had 12-month VL measures, with 6,304 (77%) achieving suppression (VL<400 c/mL). All six retention measures were significantly associated (P<0.0001) with VL suppression (OR;95%CI, c-statistic): missed visit count (0.73;0.71–0.75,0.67), missed visit dichotomous (3.2;2.8–3.6,0.62), visit adherence (3.9;3.5–4.3,0.69), gap (3.0;2.6–3.3,0.61), visit constancy (2.8;2.5–3.0,0.63), HRSA HAB (3.8;3.3–4.4,0.59). Measures incorporating “no show” visits were highly correlated (Spearman coefficient=0.83–0.85), as were measures based solely upon kept visits (Spearman coefficient=0.72–0.77). Correlation coefficients were lower across these two groups of measures (Range=0.16–0.57).
Six retention measures displayed a wide range of correlation with one another, yet each measure had significant association and modest discrimination for VL suppression. These data suggest there is no clear gold standard, and that selection of a retention measure may be tailored to context.
In recent years, increased attention has focused on expanding HIV adherence beyond antiretroviral medications to include engagement in medical care.1–3 Following HIV diagnosis, timely linkage and subsequent retention in care is imperative to allow access to antiretroviral therapy (ART), which requires uninterrupted receipt and a high-level of adherence to achieve and sustain plasma viral load (VL) suppression.4–6 A recent meta-analysis suggests that 69% (95% confidence interval: 66–71%) of newly diagnosed individuals have timely entry into HIV medical care (included studies defined care entry using a range of 3–6 months from HIV diagnosis date), 59% (95% confidence interval: 53–65%) are retained in care (included studies used varying definitions of “retained” and observation periods), and only 28% of persons living with HIV/AIDS in the US, including the undiagnosed, have achieved VL suppression.7–9 Clearly, sub-optimal linkage and retention in HIV care are formidable barriers to fully realizing the individual and public health benefits of VL suppression afforded by ART.4–6,10,11 The importance of this issue is highlighted by the emphasis placed on maximizing retention in care are in the US National HIV/AIDS Strategy,12 and by mounting enthusiasm for antiretroviral treatment as prevention approaches,13 which have been bolstered by recent research findings.14
Despite the recognized clinical and programmatic importance of retention in care, there is no recognized standard measure for retention in care. In contrast to measuring linkage to care, a dichotomous event, measuring retention is more complex as it includes multiple visits, scheduled at varying time intervals, and occurring across time. Indeed, there are many ways to operationalize retention in care, and studies have used a wide range of approaches.15 Most published studies have focused their analyses on a single retention measure, demonstrating factors associated with retention and its impact on HIV biomarker and clinical outcomes.16–19 However, the degree to which different measures of retention are related to outcomes, as well as to one another, among the same sample of HIV-infected patients is largely unexplored.20 Here we compare six commonly used measures of retention in HIV care to VL suppression and to one another in a large sample inclusive of all clinic patients participating in a multi-site study. Our aim was to evaluate the prognostic value of retention measures in predicting VL suppression as well as the correlation among measures to inform their use in future research and quality improvement initiatives.
Clinic-wide, patient-level data were used from six academically-affiliated HIV medical clinics participating in a CDC and HRSA sponsored Retention in Care (RIC) intervention study, which has previously been described in detail.21,22 The six study sites include HIV clinics affiliated with Baylor College of Medicine, Houston, TX; Boston University Medical Center, Boston, MA; Johns Hopkins University, Baltimore, MD, State University of New York, Downstate Medical Center, Brooklyn, NY; University of Alabama at Birmingham, Birmingham, AL; and University of Miami, Miami, FL. The current study used systematically captured, de-identified, socio-demographic, clinical and medical visit patient-level data from a 2-year period preceding implementation of the intervention. The analysis included patients at the six study sites who had: (1) at least one scheduled primary HIV care appointment during the first 6 months (189 days) of a 12-month observation period (1 May 2008 – 30 April 2009) and (2) attended at least 1 primary HIV medical provider visit in the year preceding the observation period (1 May 2007 – 30 April 2008). These criteria were employed to identify established clinic patients in whom retention in care could be measured. The RIC research protocol received Institutional Review Board approval at all study sites.
Our principal focus was upon retention in HIV medical care, using six commonly used measures that have been described in detail previously (Table 1).15 In accordance with prior studies, only scheduled clinic visits with a primary HIV medical provider with medication prescribing privileges (i.e., MDs, CRNPs, PAs) were included in calculating retention measures; we did not include walk-in, urgent care, subspecialty, or supportive services visits. Three of the retention measures incorporate data on missed clinic visits, and include only no-show visits that were not canceled in advance of the scheduled appointment by the patient, provider or clinic.15,23 Missed visits were recorded as a count and dichotomous measure (zero vs. ≥1 no-show visits), and visit adherence was calculated as a proportion of kept to scheduled appointments (range=0–100%), with kept visits being those attended by the patient and with the denominator excluding canceled visits. An additional three retention measures were calculated based solely upon kept visits. A 4-month constancy measure calculated the number of 4-month intervals in which a patient had at least one kept visit (range=0–3). A 6-month gap measure captured whether a patient had >189 days between sequential kept visits, and the Health and Resources Services Administration HIV/AIDS Bureau (HRSA HAB) measure calculated whether a patient had 2 kept visits separated by >90 days during the 12-month observation period. With the exception of the missed visit count measure, the other five retention measures were formulated such that a higher value indicated better retention in order to align the directionality and facilitate the interpretation of study results.
Plasma VL suppression at the end of the 12-month observation period (12-month VL) was the primary outcome of interest. As VL measures were obtained through routine clinical care and not at a specific study visit, a window of ± 120 days around the study end date of 30 Apr 2009 was used. If multiple VL measures were available in this time window, the measure closest to the study end date was used. A value <400 c/mL was considered suppressed.
Descriptive statistics and plots were generated to evaluate distributions of all study variables, and to generate summary statistics including means, standard deviations, frequencies and percentages. Spearman rank correlation was used to evaluate the associations among retention measures. Logistic regression was used to examine the association between each retention measure and 12-month VL suppression. The c-statistic (range=0.5–1.0), an estimate of the area under the receiver operator characteristic (ROC) curve plotting sensitivity by 1-specificity, was used to evaluate the discriminatory capacity of each retention measure in predicting VL suppression. The c-statistic captures the prognostic value of each retention measure to correctly assign patients to the observed 12-month VL state (suppression vs. failure), with 0.5 indicating a measure is no better than chance, and 1.0 indicating perfect discriminatory capacity. In addition, the sensitivity and specificity of each retention measure in predicting 12-month VL suppression were calculated. In primary analyses, patients with missing 12-month VL measures were excluded. Sensitivity analyses were conducted considering those with missing 12-month VL measures to be detectable (>400 c/mL, missing=failure). All statistical analyses were conducted using SAS version 9.3.
Among 10,053 study patients, the mean age was 46 ± 10 years, 65% were male, 64% were black race, 19% were of Hispanic ethnicity and 49% reported heterosexual sex (Table 2). Patients averaged 1.5 ± 1.7 no show visits during the 12-month observation period, with two-thirds of patients having at least one no-show visit. A wide range of visit adherence was observed, with nearly half of patients attending less than 75% of their scheduled clinic appointments. Fifty percent of patients attended visits in all three 4-month intervals (constancy measure), one-third had a 6-month gap, and roughly one-quarter were not retained according to the HRSA HAB measure. Twelve month VL measures were available in 8,235 patients (82%), among whom 6,304 (77%) were suppressed at 12-months (<400 c/mL, Table 2).
When comparing retention measures to one another a wide range of association was observed, with Spearman correlation coefficients ranging from 0.16 to 0.85 (Table 3). In general, measures that incorporate missed visits (missed visit count, missed visit dichotomous measure, and visit adherence) were more highly correlated with each other (0.83–0.85), as were measures based solely on kept visits (4-month constancy, 6-month gap and HRSA HAB measure: 0.72–0.77). Across these two broad categories, the count and dichotomous missed visit measures demonstrated lower correlation (0.16–0.26) with the kept visit group of measures, relative to visit adherence, which had more moderate correlations (0.51–0.57, Table 3).
All six retention measures demonstrated strong and statistically significant (P<0.001) associations with 12-month VL suppression (Table 4a). As anticipated, dichotomous measures (dichotomous missed visit, 6-month gap and HRSA HAB measure) had less discriminatory capacity for VL suppression as indicated by lower c-statistic values (0.59–62) relative to non-dichotomous measures (missed visit count, visit adherence, 4-month constancy: 0.63–0.69). This is demonstrated graphically as the area under the receiver operator characteristic curve, which is estimated by the c-statistic and plotted as sensitivity by 1-specificity (Figure 1). The performance of retention measures was largely consistent across study sites (data not shown).
Notably, among the dichotomous retention measures the 6-month gap and HRSA HAB measure demonstrated sensitivity for VL suppression of 82% and 91%, respectively, with lower specificity (39% and 27%), whereas the dichotomous missed visit measure displayed specificity of 82%, with lower sensitivity (42%). Thresholds to define retention for other measures were selected to provide a balance of sensitivity and specificity (Table 4), although other cut-points may be applied with an associated tradeoff of sensitivity for specificity and vice versa. Additional analyses using a missing equals failure approach for patients lacking 12-month VL measures (18%) yielded consistent findings with primary analyses, albeit with larger parameter estimates and c-statistic values for retention measures, particularly for kept visit based measures (Table 4b).
Our study is the first to evaluate the prognostic value of six commonly used measures in predicting VL suppression as well as the correlation among HIV retention in care measures in the same patient sample. A recent study from the HIV Research Network identified strong correlations (concordance correlation coefficients range=0.67–0.88) among retention measures calculated based upon kept visits only, which was corroborated by our analyses.20 We extend this work by further examining an additional three measures that incorporate no-show visits, and additionally comparing the prognostic value of the six measures for HIV VL suppression. Overall, considerable variability was observed among these measures in categorizing patients as being retained, ranging from one-third of patients with no missed visits (perfect visit adherence) to over three-quarters of patients meeting the HRSA HAB retention measure. While these varying definitions translated to a broad range of correlations across retention measures (Spearman coefficients range=0.16–0.85), each measure demonstrated a strong and statistically significant (P<0.001) relationship with VL suppression. Accordingly, our data suggest there is no clear gold standard to measure retention in care, and that any of the evaluated measures may have a role depending on visit data availability, the questions being addressed, and the principal rationale and goals of measuring “retention.” Moreover, there may be merit to using multiple retention measures, particularly in research settings, and using one measure that incorporates missed visits and another based solely on kept visits.
As anticipated, improved prognostic value for VL suppression was observed for multi-level retention measures compared with dichotomous measures. By allowing more granular categorization of patient retention, the missed visit count measure, visit adherence, and 4-month constancy measures allowed for better discriminatory capacity in predicting VL suppression, as indicated by higher c-statistic values. For example, among the two-thirds of patients categorized as “not retained” by the dichotomous no-show visit measure, a broad range of counts of missed visits and of visit adherence was observed. The enhanced variability captured by these latter two measures translates into improved prognostic capacity for VL suppression, and is perhaps best visualized by the multiple points incorporated into the ROC curves, in contrast to the single point employed for dichotomous measures (Figure 1).
However, dichotomous measures clearly have value, offering advantages including face validity and less complex programming, computational and analytic demands.15 Moreover, the dichotomous missed visit, 6-month gap and HRSA-HAB measures were all strongly associated with VL suppression, albeit with reduced discriminatory capacity. An interesting study finding was the variability in defining patients as retained across dichotomous retention measures, which translated into robust differences in the sensitivity and specificity of these measures in predicting VL suppression. The dichotomous no-show measure categorized only one-third of the sample as retained, resulting in high specificity (82%) of this measure in relation to VL suppression. In other words, 82% of persons without viral suppression had at least one no show visit during the year. In contrast, the 6-month gap and HRSA HAB measures categorized 68% and 77% of patients as retained, respectively. This translated into high sensitivity of these measures (82% and 91%, respectively) in predicting VL suppression. In other words, 8 or 9 of 10 persons with viral suppression met these standards for retention, respectively. No measure, however, had both high sensitivity and high specificity. These fascinating relationships highlight the potential to use multiple retention measures; one including missed visits and the other based on kept visits only, as they appear to provide complementary information regarding measurement of retention, yet are each significantly associated with VL suppression with large effect sizes. Future studies should evaluate the prognostic value of composite measures of retention integrating two or more of the measures examined here.
Our findings are germane to contemporary clinical and public health issues related to HIV treatment and prevention. In recent years, the importance of retention as a key step on the HIV treatment continuum has received heightened attention.2,4 Our findings indicate that the operational definition chosen to measure retention can have far-reaching implications in assessing this component, with subsequent downstream implications for estimates of persons on ART and achieving VL suppression. The US National HIV/AIDS Strategy set 80% retention among HRSA Ryan White CARE Act clients as a goal by 2015, with retention measured using the HRSA HAB measure.12 Among our sample, 77% of patients achieved retention according to this measure, which would have been widely variable, ranging from 33–68%, if other measures had been employed. This observation is of particular importance when comparing retention in care across settings and studies, as the measure employed, as well as the duration of the observation period can have a dramatic impact on interpretation of findings and the inference that may be drawn. It is imperative to ensure consistency of measures when evaluating similarities or differences in retention across settings.
Prior studies have established significant associations between the retention measures under study and HIV biomarker and clinical outcomes.2,16–19 The current study extends this work by evaluating VL suppression across retention measures among the same study sample, showing strong and statistically significant associations for each measure. The large parameter estimates observed are in accordance with prior studies, and underscore the critical role of retention in care as a key step along the treatment continuum that ultimately leads to VL suppression. However, the retention measures studied demonstrated only modest discriminatory capacity (c-statistic=0.59–0.69) for VL suppression. It is anticipated that including other steps in the treatment continuum, notably ART adherence, to retention measures as an additional independent variable in statistical models would improve overall prognostic value for VL suppression. Notably, a significant association between retention and ART adherence was observed in a previous study that compared visit constancy with pharmacy refills.16 Future research should examine the discriminatory capacity of models including both measures of retention and ART adherence in relation to HIV biomarker and other clinical outcomes.
Beyond individual health outcomes, considerable improvement in retention in care at the population level is essential to achieving the potential success of ART treatment as prevention initiatives.4,10,13 There is great need for substantial improvement in retention in care, along with other steps across the treatment continuum, if we are to meaningfully increase the proportion of HIV-infected Americans with suppressed VL levels from current estimates of 19–28%.4,8,9 The impact of treatment as prevention approaches is predicated upon the collective success of public health, medical and supportive service providers and affected communities in dramatically increasing these estimates.
Our study has limitations. By focusing on established clinic patients to allow for a sample with a comparable observation period we cannot comment on retention measures in persons newly establishing HIV medical care, which has been evaluated in other studies.6,17,24,25 Established patients may be more likely to be adherent to ART regardless of their retention status, and this may place a limit on the discriminatory ability of these measures. We are unable to systematically capture and account for patients who may have transferred their medical care during the one-year observation period, which could impact calculation of retention measures. Similarly, deaths during the 1-year observation period were not systematically captured, which may have resulted in under-reporting of retention, although it is unclear this would introduce systematic bias when making comparisons across retention measures. Moreover, patients who died likely had missing 12-month VL values and were excluded from primary analyses. Our examination of retention measures was for a relatively short period of time. Additional research should evaluate these measures over longer time intervals. While our six sites serve diverse populations across the US, our findings might not translate to other domestic and international treatment settings or to non-academically affiliated clinics. We also note study strengths, including the clinic-wide capture of high quality patient-level visit utilization, socio-demographic and clinical data. In addition, the evaluation of measures incorporating missed clinic visits in addition to those based solely on kept visits is novel, and extends recent work comparing only this latter group of retention measures.20
In summary, six commonly used measures of retention in care demonstrated considerable variability in categorizing retention, translating to a wide range of correlations among these measures. In general, stronger associations were observed among measures incorporating missed visits and among those based solely on kept visits, with potentially complementary information provided when using measures from these two groups. Despite the observed heterogeneity across retention measures, each demonstrated strong and statistically significant relations with VL suppression, albeit with variable discriminatory capacity. Taken together, our findings suggest that as for ART adherence, there is no clear gold standard to measure retention in HIV care, and each measure studied may have value and utility according to setting and circumstance.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the US Centers for Disease Control and Prevention or Health Resources and Services Administration. We thank the patients, providers, clinical and research personnel at the six study sites as well as the CDC and HRSA administrative and data management teams. We also thank the Retention in Care Study Group:
Boston University Medical Center
Mari-Lynn Drainoni (PI)
Meg Sullivan (PI)
SUNY Downstate Medical Center
Tracey Wilson (PI)
University of Alabama-Birmingham
Michael Mugavero (PI)
Michael Saag (PI)
Baylor College of Medicine
Tom Giordano (PI)
Johns Hopkins University
Jeanne Keruly (PI)
Richard Moore (PI)
University of Miami
Lisa Metsch (PI)
Allan Rodriguez (PI)
Laura Cheever, HRSA
Faye Malitz, HRSA
Robert Mills, HRSA
Jason Craw, CDC/ICF-Macro
Lytt Gardner, CDC
Sonali Girde, CDC/ICF-Macro
Gary Marks, CDC
Mountain Plains AETC
Source of Funding: MJM has received consulting fees (advisory board) from Bristol-Myers Squibb, Gilead Sciences and Merck Foundation, and grant support from Bristol-Myers Squibb, Pfizer, Inc, Tibotec Therapeutics, and Definicare, LLC.
This study was supported by the Centers for Disease Control and Prevention and the Health Resources and Services Administration through CDC contracts 200-2007-23685, 200-2007-23690, 200-2007-23689, 200-2007-23687, 200-2007-23684, 200-2007-23692. MJM was also supported by 5K23MH082641-05.
Conflict of Interest: All other authors report no potential conflicts of interest.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.