|Home | About | Journals | Submit | Contact Us | Français|
Little is known about the consistency of information provided by people who inject drugs (PWID) during quantitative and qualitative interviews in mixed methods studies.
We illustrate the use of the intraclass correlation coefficient, descriptive statistics, and regression to assess the consistency of information provided during a mixed methods study of PWID living in Los Angeles and San Francisco, California, USA.
Age of first use of heroin, methamphetamine, marijuana, powder cocaine, and crack cocaine and first injection of heroin, methamphetamine, and powder cocaine were collected during an interviewer administered computer-assisted personal interview followed by an in-depth qualitative interview (N=102).
Participants were 63% male, racially/ethnically diverse. 80.4% between the ages of 40 and 60 years old, 89% US-born, and 57% homeless. Consistency of self-reported data was adequate for most drug use events. Exact concordance between quantitative and qualitative measures of age of onset ranged from 18.2% to 50%. Event ordering was consistent across qualitative and quantitative results for 90.2% of participants. Analyses indicated that age of onset for heroin use, heroin injection, and injection of any drug was significantly lower when assessed by qualitative methods as compared to quantitative methods.
While inconsistency will emerge during mixed method studies, confidence in the timing and ordering of major types of events such as drug initiation episodes appear to be warranted.
Mixed methods designs – study approaches that use both qualitative and quantitative techniques --- are increasingly used in substance use research (1,2). These techniques provide new insights, yet also challenge researchers to interpret complimentary datasets that may contain contradictory information. While some researchers have provided guidance on how to reconcile contradictory results across qualitative and quantitative studies (3,4), there exists little guidance on how to approach interpreting discrepancies in raw data from qualitative and quantitative portions of the same study. This study examines consistency of measurement of age of initiation to drug use in a mixed methods study guided by life course theory's emphasis on timing and trajectories of drug use events (5,6).
Data triangulation techniques have been designed to encourage the use of multiple research methods with complementary strengths and weaknesses to arrive at research findings with reduced methodological biases (7,8). These techniques are commonly used to incorporate qualitative and quantitative results into a cohesive conclusion about the data(1). Thus, data triangulation generally assesses if the results of analyses from multiple methods are the same (or explain one another) as opposed to asking if the raw data collected from the different methods is consistent, which is our present aim. The multitrait-multimethod matrix was proposed by Campbell and Fiske to assess convergent and discriminant validity (9). While this method can be used to assess the agreement of raw data from multiple methods, it uses correlations and cannot detect differences in methods which use the same scale where it may be important to detect a systematic arithmetic difference in data collected using multiple methods. Additionally, it was developed for use with data from uncorrelated or minimally correlated variables, such as unrelated personality traits, and may not be applicable to substance use behaviors which are often associated (10).
In this paper we present ways to find discrepancies in data and potential reasons for the discrepancies using a mixed methods study of people who inject drugs (PWID). Our interest in these issues grew out of our use of life course theory (6) to understand patterns of drug injection initiation among PWID (11,12). Life course theory's focus on the time and timing, trajectories and transitions, critical periods, and accumulated risk at which important events occur within a person's life makes accurate collection of data on when crucial events occur particularly important (6,13). Past studies of drug use patterns have illustrated the usefulness of examining the trajectories of drug use (5,14). This research area will only be fruitful if accurate drug use histories are obtained. Exploring the consistency of data and potential methodological reasons for inconsistencies may aid in improvement of data collection methods and/or provide validation for current methodologies.
Research examining the consistency of age of initiation to illicit drug use is sparse. However, age of initiation has important clinical implications and is associated with increased risk for drug abuse and dependence (15,16). Some studies have assessed consistency of report of onset of tobacco, alcohol, and marijuana use and have described phenomena associated with discrepant reports (17–19). Therefore, we briefly review literature concerning age of initiation to alcohol, tobacco, and marijuana use.
Common issues related to recall bias and consistency between qualitative and quantitative data collection include forward telescoping and event clumping (17). Forward telescoping refers to reporting that an event occurred more recently than it did in actuality, such as reporting an older age of onset for drug use than is true (19). This may occur simply due to a perception that the event occurred more recently or to a changed understanding of the research question as age increases (17). For example, a child may consider their first alcohol use to be their first sip of alcohol, while a teenager or young adult may consider their first alcohol use to be when they first drank heavily (17). The events did not change, but the participants' understanding of what their first alcohol use was had changed, resulting in inconsistently reported age of onset.
Event clumping refers to reporting events as all occurring around the same time due to a rounding of age or perception that events all occurred around one important event in a person's life (17). For example, a participant may report that they began using 3 different drugs at the age of 18 when they first entered college, when in actuality their ages of first use may be more spread out. Clumping may affect both estimates of mean age of onset as well as ordering of drug use events. It is not known if data collected using qualitative or quantitative methodologies may be more susceptible to forward telescoping or clumping.
Studies of the consistency of age of onset of drug use of adolescents and young adults suggest that age of onset may not be consistently reported (17,18). In a longitudinal repeated-measures study of age of onset in adolescents, age of onset was not reported consistently, particularly for tobacco and alcohol use (18). However, consistency was observed in the ordering of drug use events (18). For example, participants' first use of alcohol is consistently reported as occurring prior to their first use of marijuana, even though both events on average are reported to have occurred at a later age in follow-up measurements (18). Therefore, while age of onset of drug use may not be consistently reported, participants generally report the order in which drug use events occurred consistently.
This study examines discrepancies in self-reported age of initiation to drug use collected using qualitative and quantitative methodologies. Due to limited research on this topic, we do not propose any hypotheses concerning discrepancies we may observe.
We present data from across-sectional study that used community outreach and targeted sampling methods (20,21) to identify and recruit PWIDs in Los Angeles and San Francisco, California, USA. The overall goal of the parent study is to conduct an exploratory qualitative and quantitative study of late initiation to injection drug use to better understand the circumstances, motivations, and social environments of injection initiation later in life (after turning 30 years old)(11,12,22). Eligibility criteria for the study was being 18 years of age or older and having physical evidence of recent drug injection (at least one injection episode in the last 30 days and visible signs of recent venipuncture)(23). After obtaining informed consent, we collected drug use, HIV risk behavior, and demographic data among other domains during a 30-minute survey using computer assisted personal interviewing involving a standardized questionnaire administered in a one-on-one interview session (24). Participants who began injecting at or after 30 years of age were invited to participate in an in-depth qualitative interview lasting 60 to 90 minutes. Eligibility criteria for participation in the qualitative study were masked to reduce the likelihood of participants providing false information during the quantitative interview. A comparison group of participants who began injecting at a younger age, but who were also over 30 years of age at the time of the quantitative interview, were also invited to participate in a qualitative interview. Qualitative interviews were conducted using a qualitative guide that contained open ended questions and follow-up probes that addressed key aspects of the research questions of interest. This type of guide was used to maintain a balance between systematic data collection in pertinent topic areas and exploration of emergent themes (25). The guide was modified over the course of the study as we learned about and explored unanticipated areas of analytic interest. Participants were paid US$20 for completing the quantitative portion of the study and an additional US$25 for completing the qualitative portion. The Institutional Review Boards at the University of Southern California and at RTI International approved all study procedures. Only participants who completed both the quantitative and the qualitative interviews are included in these analyses.
Transcribed qualitative interviews and quantitative data were used to create timelines detailing events in participants' lives, separate per participant, following a method described by Friedman et al. using Timeline Maker software (26,27). All events available from qualitative and quantitative data with specific dates or general time periods provided were entered onto the timelines. Each event was included as a separate entry which contained a short description of the event, start/end dates, event category, and data source (qualitative or quantitative). On occasions where participants stated two different ages for age of onset in the qualitative interview, the ages were averaged. Illustrative examples are a participant who stated first using marijuana at age 16 or 17, without specifying which age exactly (averaged to 16.5) or a participant who stated that they first used marijuana at age 16 in one dialogue then stated they first used marijuana at age 18 in a later portion of the interview, without acknowledging that they reported two different ages or stating that they were correcting past information provided. Data was extracted from the timelines for analyses. Creating the timelines highlighted discrepancies in age of onset from qualitative and quantitative data and inspired this study. Data from additional participants (detailed below) were entered directly from the quantitative dataset and qualitative interview into the study dataset in order to obtain a larger sample to assess the study aims while by-stepping the timeline creation process in order to have a larger sample size while limiting data processing unnecessary for the present study.
We selected 9 items to assess for consistency from the quantitative interview that elicited age of onset of use or injection for 5 drugs. The items were: (1) “The first time you injected drugs, how old were you?”; (2) “How old were you when you first used crack?”; (3) “How old were you when you first used powder cocaine?”; (4) “Age at first powder cocaine injection”; (5) “How old were you when you first used methamphetamine?”; (6) “Age at first methamphetamine injection?”; (7) “How old were you when you first used heroin?”; (8) “Age at first heroin injection?”; and (9) “How old were you when you first used marijuana?”. Skip patterns were built into the interview such that age of onset was only assessed if the participant indicated that they had used/injected the specified drug in a previous question. Crack cocaine injection was assessed in the quantitative interview but is not included for analysis due to the rarity of occurrence in our sample.
We assessed the data for discrepancies in quantitative and qualitative measures of mean age of first use of the nine substance use items. Items that the participant had not experienced or did not provide data for in both the quantitative and the qualitative interviews were dropped from the dataset. One hundred and eight participants' data was assessed for inclusion in the analytic sample (61 timelines and 47 additional participants' data entered directly into dataset). One participant was dropped from the analysis because they violated study protocols. Out of the remaining 107 participants, 102 provided data during both qualitative and quantitative interviews on at least 1 item. These 102 participants constitute the analytic sample. Refer to Table 1 for specifics on sample size.
Descriptive statistics for the age of onset were examined. Preliminary analysis to assess if there was a statistically significant difference between the quantitative and qualitative measures of age of on set was completed with paired t-tests. These were followed with linear regression models which controlled for time elapsed in years since age of onset and time elapsed in days between qualitative and quantitative interview. Separate models were computed for each event of interest. Normality of the distribution of age of onset was assessed for all 9 item; skewness ranged from -0.43 to 1.25 and kurtosis ranged from 1.80 to 4.47. Due to a large range in the difference scores calculated between quantitative and qualitative data, data was analyzed both with outliers and without outliers. Outliers were determined by taking the mean of the absolute values of the differences between quantitative and qualitative measures of all reported items per participant and identifying those participants whose average differences were at or above the 95th percentile for the sample.
Intraclass correlation coefficients (ICCs) were calculated using the single score ICC (A,1) formula printed below and as described in Table 4 of McGraw and Wong(28).
This formula is appropriate for two-way mixed effects models where the raters (in this case, qualitative and quantitative measurement) are fixed and we are interested in the absolute agreement in the data as opposed to simply the consistency of the data, where a systematic arithmetic difference could be present between the raters without detection. In this formula, n= number of subjects, k= number of observations per subject, MSR = mean square rows, MSE= mean square error, and MSC= mean square column. Conceptually, variance due to the column (due to the raters) refers to variance attributed to using qualitative or quantitative methods and variance due to the rows (due to the unit of measurement) is attributed to differences among the participants. Therefore, the ICC is the percent of the variance in a variable that is due to differences between the participants (28). ICCs closer to one indicate high consistency, and ICCs closer to zero indicate low consistency. The Stata ICC command with options absolute and mixed in Stata 12.1 was used for analyses, and was checked by running the ANOVA command and calculating the ICC using the ICC (A,1) formula(29).
Concordance of age reported was assessed by presenting the percentage of participants who reported the same age of onset in qualitative and quantitative interviews and the percentage of participants who reported ages of onset within 1 year of each other in qualitative and quantitative interviews. Consistency in ordering of events was also assessed. The ordering of drug use events was considered consistent if participants reported the same sequence of onset of the various drugs in both the qualitative and quantitative data, regardless of the ages at which the events occurred. For example, a participant may report marijuana use at age 16 and first injection drug use at age 40 in the qualitative study and marijuana use at age 18 and injection drug use at age 30 in the quantitative study. This observation would be considered consistently ordered because the events occurred in the same order, albeit at different ages. However, if in this example the participant reported that injection drug use occurred prior to marijuana use in the quantitative study the observation would be considered inconsistently ordered.
Sample demographics are presented in Table 2. Briefly, the sample was 63% male and racially/ethnically diverse. The majority of participants was between the ages of 40 and 60 years old, US-born, and had obtained at least a high school education. Slightly over half of the participants considered themselves homeless.
Descriptive statistics and ICC of the drug use items are presented in Table 3. Note that there is a different sample size for each drug use item; refer to Table 1 for detailed information addressing sample size by item. For all drug use items, the participants were on average younger at time of first drug use/injection when assessed using qualitative methods as opposed to quantitative methods, with the exception for crack cocaine use. Analyses indicated that age of onset for heroin use, heroin injection, and injection of any drug was significantly lower when assessed by qualitative methods as compared to quantitative methods; findings remained after controlling for time elapsed between interviews and time elapsed since age of onset. Five outliers were identified. These participants had discrepancies with average magnitudes ranging from 9.2 to 18.9 years. Findings did not change when these outliers were excluded.. ICCs ranged from 0.69 to 0.96. ICCs for drug use items were lower than ICCs for drug injection items. Marijuana had the lowest ICC, at 0.69. All other items had ICCs of 0.79 or higher. Exact concordance between quantitative and qualitative measures of age of onset ranged from 18.2% to 50%. Concordance within one year ranged from 43.2% to 67.7%. Lastly, event ordering was consistent across qualitative and quantitative results in 90.2% (92 out of 102) of the timelines.
Our data suggest that qualitative and quantitative techniques result in data adequately consistent with one another for age of onset of use and injection of a variety of illicit drugs. This is a promising finding in that it supports that these research methods collect consistent data. Age of drug use onset events were reported less consistently than age of drug injection onset events. Onset of injection is likely a remarkable event in the lives of PWIDs and may have occurred simultaneously with other life course altering events, resulting in more consistent recall in comparison to drug use by other means. Furthermore, since our study sample consisted solely of PWIDs, our participants' most memorable drug use experience may have been first injection. If we sampled people who only use drugs by routes other than injection, we may find that they recall with greatest consistency onset of use of the drug that caused the most problems for them, produced the most desired effects, and/or their most commonly used drug. Participants may also be commonly asked to report their age of first injection to medical professionals for health risk assessment, and may have a prepared response to the question as opposed to having to think about their response.
The results varied with the statistic used. ICCs were lower for items where age of onset was reported inconsistently by a large proportion of the sample, regardless of the magnitude of the discrepancy. Percentage concordant within one year was above 50% for almost all items; the majority of participants was able to recall their age of first use consistently with a one year margin for error. However, the large standard deviations and ranges for the differences between qualitative and quantitative measures suggest that some of those participants who inconsistently report age of onset may do so with a large error. Large standard deviations and small magnitudes for differences in report of age of onset have been noted in previous studies of consistency of self-report data (18).
While the ICC quantifies the level of consistency between quantitative and qualitative measurements, there is no formal scale for assessing ICC values. Landis and Koch provide a rubric for determining what an acceptable reliability coefficient is, although, as they note in their paper, it was developed with arbitrary distinctions (30). According to them, moderate agreement ranges from 0.41-0.60, substantial from 0.61-0.80, and almost perfect from 0.81 to 1. This scale is helpful in assessing ICC, but it was originally developed for the kappa statistic and highlights a weakness of the ICC---interpretation may be subjective.
The quantitative methods produced slightly higher values age of onset in comparison to the qualitative methods. This difference was significant for heroin use, heroin injection, and injection of any drug. For some participants, these three events are not independent of one another. For example, a participant may first have used heroin by injecting it, and that injection event may be the first time they have ever injected drugs (31). Low sample size may be affecting the results as well, and this study may have been underpowered to detect differences in measurement for items reported less frequently.
As we cannot know the true age at which these events occurred (validity), we cannot say if quantitative methods overestimate age or if qualitative methods underestimate age. It may also be the case that both methods either underestimate or overestimate age of onset. We consider both explanations plausible. Forward telescoping may have occurred during the quantitative portion of the study. Participants may have overestimated their age when answering questions regarding first use/injection. Qualitative techniques may have helped the participants put drug use/injection events in context of the rest of their lives and prompted them to report more accurate ages (32). Additionally, the quantitative portion of the study occurred first for almost all participants. Only one participant completed the quantitative interview after the qualitative interview. The act of completing the quantitative survey may have caused participants to change how they interpreted their memories or their actual memories of their first drug use, resulting in the discrepancy. Participants also had time in between completing the quantitative and the qualitative interviews, allowing them more time to think about their responses to the questions. The noted differences may not be due to measurement method, but to ordering of assessment.
Question interpretation may have affected the results for those PWIDs who had stopped their drug use and began again. When asked when they first injected in the quantitative interview, it appears a couple participants provided the age of when they began injecting most recently, that is, at what age they began injecting after a period of abstinence as opposed to when they first injected ever in their life. The timelines helped to identify this discrepancy. This finding suggests that care should be taken when designing quantitative questionnaires and pilot testing may be necessary to check that participants are interpreting questions as intended by the researcher.
Researcher effects were additionally of concern in this study. Participants may have provided false information during the qualitative and/or quantitative study due to embarrassment or the illicit nature of drug use. We did conduct t-tests by drug and for average discrepancy to examine if those participants who had a consistent interviewer for the qualitative and quantitative portions of the studies provided more or less consistent results. Data on interviewer was available for 90 participants. Of these 90, 34 participants (38%) had the same interviewer for both interviews. We found that having a consistent interviewer had no effect on discrepancies (ps = .09-.99). All researchers were experienced and trained and no evidence suggests that researcher effects account for the discrepancies found.
Methamphetamine and heroin use and injection exhibited discrepancies with the largest magnitude in terms of the difference in the number of years between quantitative and qualitative responses. Findings for methamphetamine are supported by past longitudinal research which identified amphetamine use as having the highest number of discrepancies in reporting of a variety of substances when ever/never use was assessed over a 10-year period in a repeated-measures study (32). The context in which methamphetamine is used or the effects of the drug itself may result in inconsistent recall of memories surrounding the time of first use.
One important finding was that while the age of use measured by qualitative and quantitative methods did not always match up perfectly, the order of events did not differ for most participants and concordance of age of onset within one year was at least 43% for all items. There appears to be stability in report of the progression of age of onset events regardless of the method of measurement. This suggests that whether events are measured through a qualitative interview or through a quantitative instrument the sequence of events should be consistent.
Study results should be considered in light of potential methodological limitations. First, some of the items had very small sample sizes. In particular, neither powder cocaine injection nor crack cocaine use was commonly reported in the qualitative interviews. The low sample size is potentially why we failed to find statistically significant difference between the quantitative and qualitative measures for these and other smaller sample sizes items. There are recommendations for sample size for reliability studies when calculating the ICC (33). Given that these analyses are part of a larger study, we were not able to address this prior to data collection, and sample sizes for some items may be lower than ideal. The qualitative interviews were not structured in a way to capture all first drug use events. As this information was not necessarily elicited by the interviewer, there is missing data. It is unknown if this missing data has affected the results.
Second, our data may not be representative of all PWIDs, and findings may not generalize to the larger population of PWIDs. The qualitative sample here was chosen specifically due to their status as late initiates (30 or older) or typical initiates (before age 30) of drug injection. Since everyone in the qualitative study had to be at least 30 years of age, these data do not generalize to persons under age 30.
Future research should focus on confirming the current findings in larger samples and different populations and exploring causes for discrepancies. Statistics used to assess consistency can affect interpretation; multiple statistics may be necessary to describe consistency in self-report data. Overall the data exhibited a high rate of concordance for age of onset reported, particularly when considering the majority of the participants was over 40 years old and age of onset for all items occurred on average prior to the age of 35. However, significant differences in age of onset were noted between measures for some items, and lack of significance for the rest of the items does not indicate equality in their measurements. Fuller understanding of factors influencing self-report of age of onset may improve the collection, analysis, and interpretation of data for mixed-methods studies of PWIDs.
We would like to thank the large research team which conducted this study. The San Francisco team consisted of Sonya Arreola, Askia Muhammad, Andrea Lopez, Jahaira Fajardo, and Michele Thorsen. The Los Angeles team consisted of Daniel Chu, Frank Levels, Richard Hamilton, Guillermo Felix, Vahak Bairamian, Luis Maldonaldo, Jacob Curry, and Brett Mendenhall. And we would like to acknowledge the participants, without whom we would be unable to conduct this research.
Funding: We would like to thank the National Institute on Drug Abuse (Grant # 5R01DA027689) and the Tobacco-Related Disease Research Program (Grant # 23DT-0112) for funding this research.
Declaration of Interest: The authors report no declarations of interest.