|Home | About | Journals | Submit | Contact Us | Français|
Substance use can have major consequences among HIV patients. Interviewer- or self-administered modalities are widely used to measure drug use frequency. This often involves Timeline Follow-Back (TLFB) interviewer-administered measures, or self-administered computerized questions assessing similar information via Audio Computer-Assisted Self Interview (A-CASI). Little is known about agreement between these two modalities on drug use frequency in HIV-infected samples.
Prior to randomization into a trial of brief interventions to reduce drug use, 240 HIV patients completed a baseline A-CASI assessment battery that included questions on drug use frequency, followed by an interviewer-administered TLFB. Each measure generated number of days patients used their primary drug in the prior 30 days. Agreement between TLFB and A-CASI modalities on days using primary drug was determined using intraclass correlation coefficients (ICC). Regression analysis tested the association of patient characteristics with discrepancies between TLFB and A-CASI modalities.
Overall agreement was excellent (ICC=.80), with little variation by primary drug, education, race, current drug treatment, binge drinking or years since HIV diagnosis. Gender, ethnicity (Hispanic vs. non-Hispanic) and age predicted differences in days used (p<0.05); the A-CASI modality reflected more days used than TLFB.
Measures of days used primary drug showed high agreement whether assessed by interviewer-administered TLFB or by questions self-administered via the A-CASI modality. Differences by gender, ethnicity and age suggest some caution in using the TLFB, although additional studies are needed. However, findings generally indicate that studies based on one assessment method or the other can be compared with reasonable confidence.
Substance use among HIV-infected individuals is associated with many adverse consequences, including poor treatment outcomes (Pacek et al., 2014), low antiretroviral treatment (ART) initiation (Kapadia et al., 2005), HIV sexual risk behaviors, STI transmission (Khan et al., 2013) and hepatitis C virus (HCV) (Hermanstyne et al., 2012). Substance use is prevalent among HIV-infected persons (Mimiaga et al., 2013; Pacek et al., 2014); in a recent U.S. national survey, 33.1% of those with HIV/AIDS used an illegal drug in the prior 30 days (Pacek et al., 2014). Further, non-injecting drug use is increasingly associated with HIV infection (Des Jarlais et al., 2007). Thus, developing and disseminating effective interventions to reduce drug use in this population is a public health priority. In order to test the efficacy of such interventions, or methods of disseminating them, researchers and clinicians must often measure frequency of substance use.
Measuring frequency of drug use poses significant challenges. Studies often use participant self-reports, which are relatively easy to administer, but can be vulnerable to under-reporting due to two main factors: participants’ inability to accurately recall drug use (forgetting), and social desirability bias (responding inaccurately in an interview due to the desire to appear favorably to the interviewer). Forgetting may occur when the assessment timeframe is longer than a day or two, including the common timeframe of the prior 30 days. Social desirability bias may occur because drug use is illegal, stigmatizing and potentially embarrassing to report (Rosenbaum et al., 2006), problems that are compounded in HIV-infected individuals who often suffer from multiple stigmatized conditions (Eaton et al., 2015; Katz et al., 2013). Consistent with this, HIV-infected persons consistently report distrust of healthcare institutions (Beer et al., 2009; Eaton et al., 2015), poor communication with providers (Flickinger et al., 2013; Messer et al., 2013), and fears about loss of confidentiality (Beer et al., 2009). In research in a healthcare setting, these issues can all contribute to under-reporting of drug use frequency. Therefore, in HIV patients, drug use frequency could vary by the measurement modality used (Beck et al., 2014; Grucza et al., 2007; Rosenbaum et al., 2006), depending on whether the modality primarily focuses on reducing forgetting, or on reducing social desirability bias.
To assess self-reported drug use frequency, two widely-used modalities are commonly used, interviewer-administered assessments and self-administered assessments. A commonly-used interviewer-administered assessment is the Timeline Follow-Back (TLFB) (Sobell and Sobell, 1992). Self-administered assessments, which may consist of simple questions about frequency of drug use, often use the Audio Computer-Assisted Self Interview (A-CASI) modality. Each method has a differential emphasis on reducing forgetting or social desirability bias as a source of under-reporting, as well as other advantages and disadvantages.
In an interviewer-administered TLFB, interviewers work actively to address forgetting by assisting patients to recall events. At the beginning of a TLFB interview, an interviewer uses a calendar with the participant as a recall and memory aid (Carey et al., 2004; Richter and Johnson, 2001; Robinson et al., 2014) reviewing events that occurred in the participant’s life during the timeframe of interest (e.g., past 30 days). After noting these events, the interviewer then uses unstructured questions to help the participant recall days of substance use, which are noted on the calendar and tallied later to produce measures such as drug use frequency. A major assumption about the interviewer-administered TLFB modality is that the face-to-face, interviewer-administered, semi-structured format establishes trust and rapport between interviewer and participant (Rosenbaum et al., 2006), overcoming social desirability bias. An advantage of interviewer-administered TLFBs is that interviewers can work to ensure that patients attend to each question, and can also provide additional instructions and clarifications on the procedure for patients as needed. Thus, the TLFB is seen as advantageous for diverse patients, including those with mild cognitive impairments or other characteristics potentially affecting attention and recall (Sacks et al., 2003). The major disadvantage of the TLFB is that respondents may minimize their drug use reports to present themselves in a socially desirable manner. In addition, although TLFB interviewers are trained to administer the TLFB in a standardized way, they may unintentionally offer patients varying levels of non-standardized communication and feedback (Richter and Johnson, 2001). Despite these potential influences on reporting, the TLFB has shown strong test-retest reliability across several substances (Robinson et al., 2014) and across diverse populations (Carey et al., 2004; Fals-Stewart et al., 2000; Levy et al., 2004; Sacks et al., 2003) . The TLFB has also shown strong concurrent validity using biological measures(Hjorthøj et al., 2012), other interviewer-administered instruments (Dennis et al., 2004; Fals-Stewart et al., 2000) and self-report instruments (Fals-Stewart et al., 2000; Sacks et al., 2003).
In the A-CASI modality, questions about drug use frequency (simple or complex) are self-administered without an interviewer, allowing patients to respond privately and thus reducing the threat of social desirability bias (Estes et al., 2010; Richter and Johnson, 2001). This privacy could result in more accurate and honest responses, yielding greater reports of drug use. For example, patients may feel uncomfortable reporting substance use behaviors to an interviewer due to fear of stigmatization. The confidentiality of the A-CASI format is assumed to eliminate this (Dolezal et al., 2012; Estes et al., 2010). Another A-CASI advantage is the consistency in administration of questionnaire items within and across patients. The same program and questions are always used, ensuring a standardized protocol. In addition, the program used to administer the A-CASI questionnaire automatically generates a data file for analysis, eliminating the need for post-interview data entry and tallying. However, A-CASI administration leaves the study team reliant on patients’ willingness and ability to read questionnaire items carefully and enter their data accurately, with no way of identifying data entry errors and outlier responses. This could yield some responses that appear implausible, with no obvious way to resolve them (Rosenbaum et al., 2006; Wright et al., 1998). Drug use frequency questions administered in the A-CASI modality have demonstrated good test-retest reliability (McNeely et al., 2014) and validity (Qian et al., 2014; Simões and Bastos, 2004) to measure drug use in primary care patients and in drug users (Islam et al., 2012).
Past research has typically demonstrated that respondents report higher rates of drug and alcohol use with a self-administered questionnaire than a face-to-face interview (Bjarnason and Adalbjarnardottir, 2000; Rosenbaum et al., 2006). However, little is known about the agreement between the interviewer-administered TLFB and other questions on frequency of drug use administered via A-CASI to HIV-infected patients who are drug users. Substantial disagreement would indicate that studies using one method should not be compared to similar studies that used the other method, impairing the field’s ability to aggregate data across studies. Further, knowledge is limited on whether or not disclosure of drug use between the two modalities varies by patient characteristics. Previous studies suggest that in general, demographic characteristics such as age (Ledgerwood et al., 2008; Vigil-Colet et al., 2015; Welte and Russell, 1993), race (Johnson and Fendrich, 2005; Ledgerwood et al., 2008; White et al., 2014) gender (Bjarnason and Adalbjarnardottir, 2000; Dolezal et al., 2012) and socioeconomic status (Welte and Russell, 1993) may be related to differential reporting of sensitive health behaviors. Whether these characteristics differentially affect reporting of drug use frequency in interviewer-administered vs. A-CASI-administered assessments among HIV positive patients is presently unknown.
The main purpose of this study was therefore to test the agreement on drug use frequency in the prior 30 days between questions administered in an A-CASI battery and in data tallied from interviewer-administered TLFB interviews. We also examined whether the agreement between A-CASI and TLFB data was associated with patient demographic and clinical characteristics.
The current study uses baseline data from an ongoing randomized trial of brief interventions to reduce non-injection drug use (NIDU) in HIV primary care patients (Aharonovich et al., 2012). Participants were recruited from two large HIV primary care clinics in New York City between 2011 and 2014. The HIV clinics mainly serve low-SES, minority patients from all five boroughs of New York City. Participants were referred by clinic staff to a bi-lingual study coordinator, who screened patients for eligibility and obtained informed consent. Following this, patients completed an A-CASI assessment battery that included questions on drug use frequency as well as other scales and measures. After completing the A-CASI battery, a baseline TLFB interview focused on use of the patients’ primary drug was conducted. All interviews and self-administered assessments were administered, in English or Spanish, according to participants’ preference. After completing the A-CASI battery and the TLFB, participants (N=240) were randomized to treatment conditions. Institutional review boards at the New York State Psychiatric Institute, St. Luke’s Roosevelt Hospital and Mt. Sinai Medical Center approved all study procedures.
Participants were eligible if they were at least 18 years old, able to give informed consent; had at least four days of non-injection use of cocaine, methamphetamine or heroin during the prior 30 days, one of which was identified by the patient as their primary drug, and were not in need of detoxification. Exclusion criteria included any intravenous drug use in the past six months, being psychotic, homicidal or suicidal, having gross cognitive impairments, having hearing or vision impairment that precluded telephone use, or definite plans to leave the greater New York metropolitan area within the study period. For the present analysis, we excluded participants whose self-reported drug use data at baseline had a technical error (n=1) or were treated preventatively at the HIV clinic without being HIV positive (n=6). Thus, the present analysis includes 233 participants.
During screening, the study coordinator ascertained patients’ demographic variables, including gender, race (African American vs. other), ethnicity (Hispanic vs. non-Hispanic), age and education status (years of education). Patients also self-reported their primary drug (amphetamines, cocaine/crack or heroin), self-defined as the drug that currently gave them the most problems. Patients reported whether they were currently in alcohol or drug treatment, years since receiving an HIV diagnosis, and frequency and quantity of alcohol consumption in the prior 30 days in the A-CASI assessment battery (see below). Patients who consumed at least 4 (for women) or 5 (for men) standard drinks (NIAAA) on at least one occasion in the past 30 days were defined as binge drinkers (NIAAA, 2004).
Participants completed the A-CASI assessment battery prior to receiving any other assessment. Patients were given a paper calendar of the prior 30 days to use for reference while completing the A-CASI assessment battery, but received no instructions on what to do with this calendar other than to use it to help them answer the questions, if needed. The A-CASI assessment battery began with questions on drug use, including the number of days each type of drug was used during the prior 30 days. To assess drug use frequency, patients were asked, “On how many days since [date 30 days prior] did you use your primary drug?” Patients responded to A-CASI questions by keying in their responses using the computer keyboard. The A-CASI provided an audio function for patients who wanted to hear the questions and response options. A-CASI data were imported into Statistical Analysis Software (SAS Inc, 2014) for analysis, including a variable representing days used each type of drug in the prior 30 days. The drug identified by the patient as his/her primary drug was the one analyzed in this report. The A-CASI was programmed using Questionnaire Design Software (QDS)(NOVA, 2013).
Following standard TLFB procedures (Sobell and Sobell, 1992), patients were oriented to the timeframe by use of a calendar showing the prior 30 days. To enhance recall, patients were shown a calendar of the prior 30 days, and asked to indicate any birthdays, holidays, events, reunions, paychecks etc. that occurred within the prior 30 days. The interviewers recorded this information on the calendar. Next, working with this information, interviewers probed for the days patients used their primary drug in the prior 30 days, and recorded these days on the calendar. Data from the TLFB calendars were entered into Excel spreadsheets by trained research assistants, and exported into SAS files, from which the variable representing days used in the prior 30 days was generated.
Chance-corrected agreement between TLFB and A-CASI measures of frequency of drug use (days used primary drug in the prior 30 days) was determined using intraclass correlation coefficients (ICC) (Shrout and Fleiss, 1979), computed with STATA software, version 13.0 (StataCorp, 2013). Values of ICC can range from −1.0 to 1.0, with values ≥ 0.75 indicating excellent agreement. We conducted the analysis in the full sample as well as in the subsamples identified in Table 1. Then, for each participant, we calculated the absolute difference in days the patient used his/her primary drug as reported in the TLFB and A-CASI measures. Using this difference as an outcome variable, we assessed characteristics associated with it using a generalized linear regression model with a negative binomial distribution (SAS Inc, 2014). Individual characteristics included in the regression model were: primary drug (methamphetamine, heroin, cocaine/crack), age, gender, education (completed high school or higher vs. all others), Hispanic origin vs. all others, race, years since HIV diagnosis, current substance or alcohol treatment, and binge drinking. All characteristics were included simultaneously in the model.
Study notes indicated that 18 patients requested assistance to enable them to self-administer the A-CASI assessment battery. We explored whether these patients might have had an impact on the results by excluding them from the sample and then re-running the ICC analysis in the remaining patients.
As shown in Table 1, most patients were male (82.8%) and African American (66.5%). Participants were primarily in mid-adulthood (mean age: 46.6 years) and had at least a high school education (68.1%). Many identified cocaine or crack as their primary drug (70.2%), about a fifth reported receiving alcohol or drug treatment outside the study (17.6%), and about one-third were binge drinkers (34.3%).
Participants reported using their primary drug an average of 10.0 days in past 30 days with the A-CASI, and an average of 8.72 days in past 30 days with the TLFB (Table 2). Agreement between the TLFB and A-CASI assessments on days used primary drug was excellent in the full sample (ICC = 0.80) and in virtually all subsamples (ICC Range = 0.75-0.88) (Table 2).
When the nine characteristics of between-measure differences in days used were examined using regression analysis, gender, ethnicity and age were significantly associated with the difference between measures in number of days primary drug was used. For females, younger patients, and non-Hispanic participants, the A-CASI ascertained a greater number of days used than the TLFB. Compared to men, the measure of days used was 45% larger among women in the A-CASI measure than in the TLFB measure. The absolute difference between the A-CASI compared to TLFB was 4.56 (s.d.=4.4) days among women, compared to an absolute difference of 2.84 (s.d.=3.3) days among men. For age, the measure of days used was 49% greater in the A-CASI measure than in the TLFB measures among younger adults (40 years or less) compared to older adults (55 years or more). The absolute difference was 3.53 days (s.d.=3.9) in the youngest age group, 3.12 days (s.d.=3.5) in the middle age group (41-54 years) and 2.74 days (s.d. = 3.2) in the oldest age group (55 years and older). Among non-Hispanic respondents, the measure of days used was 19% larger in the A-CASI measure than in the TLFB measures than among Hispanic respondents. The absolute difference was 3.38 (s.d. = 3.7) in non-Hispanic respondents, and 2.43 (s.d. = 3.1) among Hispanic respondents.
In the subsample of patients that did not request assistance with the ACASI program (n=215), the ICC representing agreement on days used primary drug in the prior 30 days was excellent (ICC=0.80, 95% CI (0.73, 0.86), similar to the full sample.
The aims of this study were to assess the agreement between two widely used methods to assess frequency of illicit drug use, i.e., days used primary drug in the prior 30 days. Each of these measures attempts to reduce under-reporting, but focuses on different ways of accomplishing this. One measure consisted of structured questions self-administered via computer (A-CASI), a modality that provides participant privacy to reduce under-reporting due to social desirability. The other measure, the interviewer-administered TLFB, focuses on memory aids and interviewer probing to reduce under-reporting due to forgetting. Results indicated that patients reported the frequency of days used their primary drug quite similarly in the two modalities, with an average of 10.2 days reported in the A-CASI modality, and 8.7 days in the interviewer-administered TLFB. Chance-corrected agreement between the assessments was consistently excellent in the full sample and in all subsamples.
Although previous work has examined interviewer-administered and computer-assisted modalities in heterogeneous samples of patients (Dolezal et al., 2012; Islam et al., 2012; Rosenbaum et al., 2006; Wright et al., 1998), this is the first study directly determining agreement between these two modalities in measuring days of substance use among HIV-infected individuals. Given the high prevalence of substance use among HIV-infected individuals, the many adverse consequences of such use, and the widespread use of both types of measures across studies, this study provides important information on the agreement between the two types of measures in this population. The high chance-corrected agreement between TLFB and A-CASI suggests that both modalities serve as useful options for measuring non-injection drug use in this population, and that studies based on either type of measure can be compared to each other with a reasonable degree of confidence.
Out of numerous patient characteristics, three predicted significant discrepancies between the A-CASI modality and the interviewer-administered TLFB in days used primary drug. These characteristics were gender, ethnicity and age. In these sub-groups, the A-CASI modality reflected a greater number of days used than the interviewer-administered TLFB. The discrepancy between A-CASI and TLFB was greater among women than men, younger participants than older ones, and non-Hispanics than Hispanics. The other demographic and clinical characteristics, including primary drug (cocaine/crack vs. meth vs. heroin), education status (high school vs. no high school), race (African American vs. other), years since HIV diagnosis, treatment at time of participation and binge drinking in 30 days prior to participation were not significantly associated with differences in days used primary drug reported in A-CASI and TLFB. These results warrant replication in HIV and other samples, and in studies that can address the differential validity of the two measures.
Previous studies show that under-reporting of drug use and other stigmatized behaviors due to social desirability bias tends to increase among older respondents (Vigil-Colet et al., 2015; Welte and Russell, 1993). Thus, our finding that discrepancies were greater among younger participants, who reported greater days used in the A-CASI modality than in the TLFB, is inconsistent with these previous studies. However, the earlier studies were largely conducted in general population samples. In this sample of HIV patients, younger patients had their diagnoses of HIV for fewer years, and so had less years of treatment for HIV to build trust in personnel encountered in medical settings, perhaps leading them to report more days of drug use in the privacy of the A-CASI modality. Additional studies are needed to determine if this finding will replicate in other samples.
Our finding that women reported days used primary drug more discrepantly than men in the A-CASI and the TLFB is consistent with previous research suggesting that women are more vulnerable to social desirability bias than men. Women may generally be more concerned about judgmental attitudes of others, or feel more stigmatized in HIV settings than men, or may be wary about reporting substance use in a face-to-face interview such as the TLFB due to fear of losing custody of their children.
In this study, the reasons that Hispanic participants reported drug use frequency more consistently than non-Hispanic participants in A-CASI than in the interviewer-administered TLFB are unclear. The fact that study staff members were all of Hispanic ethnicity (so that study procedures could be offered to all participants in Spanish or English) may have enhanced rapport between the Hispanic patients and their TLFB interviewers. Further studies utilizing both Hispanic and non-Hispanic study personnel are needed to clarify this issue.
Notwithstanding these group differences, the main study results suggest that the magnitude of discrepancy between the A-CASI and interviewer-administered TLFB measures is not substantial enough to cause investigators to chose one measure exclusively instead of the other. This study warrants replication in other HIV and in non-HIV samples, as well as studies that can address the differential validity of the two measures, for example, via biological samples.
When a choice is required, investigators may choose the A-CASI modality over the interviewer-administered TLFB, or vice versa, for many reasons. Often, the instruments selected to measure substance use are limited by availability of clinic and research team resources. Especially if a lengthy battery is required, computer-assisted self-report measures can require time and resources to develop and program. A-CASI questionnaires can also be somewhat tiresome to self-administer, which could affect engagement in the assessment process, and eventual study retention if not addressed by the study team. However, once developed, A-CASI questionnaires can be administered without a trained interviewer, and thus cost-effective in a variety of settings. In contrast, the TLFB administered by an interviewer requires training study or clinic personnel to administer the assessment, which may be expensive. Additionally, administration of the TLFB necessitates a private space where patients can respond to the interviewer in confidence. However, the TLFB can be used to develop rapport with participants, potentially increasing study engagement in the measurement process, and study retention. For some investigators, the TLFB increases confidence that the assessment of substance abuse has benefited from the involvement of a skilled member of the research team.
Study limitations are noted. Data were collected from HIV primary care patients who subsequently received brief interventions for non-injection substance use in New York City. Further studies are needed to determine the generalizability of findings to patients in other settings and geographical locations. In addition, only patients who had already acknowledged drug use to be eligible for participation in this study were included, so differences in reporting due to social desirability bias might be more pronounced in a more representative sample of HIV primary care patients, or in samples other than HIV primary care patients, warranting further research. Studies should also address whether findings are consistent in assessments administered at follow-up intervals or at study conclusion. In addition, neither modality was compared to biological measures of substance use, which could serve as a validation of one measure or the other. Ideally, the order of the A-CASI drug use frequency questions and the interviewer-administered TLFB would have been randomly assigned to address any potential of “learning” from answering the A-CASI questions on responses in the TLFB interview. However, we think that little such learning actually occurred, since the A-CASI questions covered multiple types of illicit drugs, and were followed by an extensive battery of other scales and questionnaires. If the A-CASI questions on frequency of drug use had been followed immediately by the TLFB, then such learning could have occurred because respondents would remember what they reported just minutes before. However, completing the additional A-CASI scales and questionnaires (which took about 45 minutes) is likely to have interfered with recall of specific responses to the A-CASI questions on frequency of drug use. Further, computerized versions of TLFB measures have been developed (Rueger et al., 2012; Sobell et al., 1996) and used by some investigators. Little research exists on how these measures compare to interviewer-administered TLFB or other A-CASI procedures; future studies in this area would offer useful information. Further, the comparative performance of the TLFB and A-CASI to measure more complex aspects of substance use could be studied.
These limitations are offset by strengths of the study, which include a large clinical sample and comparison of two widely-used modalities for ascertaining self-reported substance use. Further, we provide three analyses summarizing reporting between both measures. We descriptively summarize days used substance as assessed by each modality, calculate the ICC for agreement between the measures, and model discrepancies regressed on patient characteristics. These three methods give a comprehensive depiction of differences in reporting across measures.
Overall, the study provides data to support that studies based on TLFB or A-CASI assessments can be compared with reasonable confidence. Specifically, the TLFB and A-CASI substance use assessment have excellent agreement in a sample of HIV-infected primary care patients. However, the findings do suggest some degree of caution in using the TLFB when age, gender and ethnicity are salient considerations, as they often are in substance abuse research. However, future research should attempt to replicate these findings in other samples, across additional time-points, and across additional drug use outcomes.