|Home | About | Journals | Submit | Contact Us | Français|
Missing data in substance use disorder (SUD) research pose a significant threat to internal validity. Participants terminate involvement or become less likely to attend intervention and research visits for many reasons, which should be addressed prior to becoming problematic. During a 9-month study targeting stimulant abuse, early dropouts and participant reported attendance barriers led to implementing a structured, pre-randomization protocol with participants about retention and solution-focused strategies (the “Fireside Chat”). Our aim is to outline this approach and present data on intervention participation and research visit attendance after implementation.
STimulant Reduction using Dosed Exercise (STRIDE) was a two-arm, multisite randomized clinical trial testing treatment-as-usual for stimulant abuse/dependence augmented by Exercise or Health Education. For both groups, study intervention visits at the site were scheduled 3/week for 12 weeks followed by 1/week for 24 weeks. During The Chat, research staff thoroughly reviewed participants’ expectations, and barriers and solutions to retention. Fifteen participants were randomized (to Exercise or Health Education) prior to and fourteen were randomized after Chat implementation. Intervention and monthly follow-up attendance (before and after implementation) were compared at the site (N=29) that developed and rigorously implemented The Chat.
Individuals who participated in The Chat (n=14) attended significantly more intervention visits during weeks 1–12 (p<0.001) and weeks 13–36 (p<0.05) and attended more research visits (p<0.001).
Proactive discussion of expectations and barriers prior to randomization was associated with greater study attendance. SUD researchers should consider tailoring this approach to suit their needs. Further investigation is warranted.
Missing data in longitudinal substance use disorder (SUD) research pose a significant problem for drawing valid inferences, with 20% of participants routinely lost to follow-up after 3 months of participation and nearly one-third of participants missing data beyond 12 months (Hansen, Tobler, & Graham, 1990). This poses a severe threat to internal validity, and researchers are frequently faced with the dilemma of how to handle missing observations, particularly for rigorous, longitudinal SUD research. All commonly employed imputation methods within SUD research (e.g., last-observation-carry-forward, missing equals positive, multiple imputations, and full-information maximum likelihood) are subject to error. Further, these methods assume data are missing at random (MAR) or missing completely at random (MCAR). When the data are not missing at random (NMAR; i.e., the probability of a missing value depends on the variable that is missing) the missing data mechanism must be modeled to obtain valid parameter estimates. For this reason and others, missing information should be minimized through proactive, ongoing study staff efforts to refine and improve the recruitment and retention process and facilitate participant attendance; however, little formal research guides best practices for retaining participants.
Participants join studies for a variety of reasons (Scott, Walker, White, & Lewith, 2011), including personal relevance, altruism, and monetary compensation (Kost, Lee, Yessis, Coller, & Henderson, 2011). Also, participants terminate involvement or become less involved for many reasons, including perceived stigma, scheduling difficulties, SUD symptoms, and waning motivation (Ball, Carroll, Canning-Ball, & Rounsaville, 2006; Claus, Kindleberger, & Dugan, 2002). Most reasons can and should be addressed prior to becoming a retention issue, particularly for longitudinal research designs.
Engaging participants in a comprehensive, collaborative discussion prior to randomization about real-world barriers to research attendance may hold promise to maximize retention. Evidence supports participant comprehension as an important factor of involvement in multisite RCTs (Lipton et al., 2011). Further, team recruitment approaches have had success in a variety of studies (e.g., weight gain prevention; Stockton, McClanahan, Lanctot, Klesges, & Beech, 2012).
In response to dropouts and participant-reported attendance barriers, we initiated a structured protocol with ongoing participant collaboration, called the “Fireside Chat.” This approach augmented the informed consent process with participants and was applied rigorously within the context of a multi-site, longitudinal intervention study with stimulant abusers to maximize available data and participant retention. The aims of this secondary data analysis are to describe the Fireside Chat and present data on intervention and research visit attendance for participants who did and did not participate in a Fireside Chat at one of the study sites.
A multisite RCT (STimulant Reduction using Dosed Exercise [STRIDE]) was conducted by the National Drug Abuse Treatment Clinical Trials Network at nine U.S. residential substance abuse treatment programs (RTPs). STRIDE (N=302) compared Exercise versus Health Education as augmentation to addiction treatment-as-usual in individuals with stimulant use disorders (Trivedi et al., in press). Men and women (ages 18–65) who were admitted to RTPs, used stimulants in the 30 days prior to admission, met DSM-IV criteria for stimulant abuse/dependence, and medically cleared to exercise, were eligible.
Randomization to Exercise or Health Education occurred soon after treatment admission to maximize days of study participation during the RTP stay and increase the likelihood of intervention attendance. Study visits (i.e., Exercise or Health Education intervention and research visits) occurred 3/week for three months (weeks 1–12; acute phase), followed by six months of weekly visits (weeks 13–36; continuation phase). Exercise was prescribed at approximately 50 minutes, 3 days per week. Health Education consisted of online, video, and written educational materials for an equivalent period of time. Complete design and rationale are described elsewhere (Greer et al., 2012; Stoutenberg et al., 2012; Trivedi et al., 2011).
At study initiation, staff proactively discussed and monitored participants’ treatment status (e.g., outpatient treatment plans) to help ensure decisions to randomize were fully informed. Further, due to the large participant-time investment (>50 hours for highly adherent participants, not including travel time), incentives were provided to increase intervention and research visit attendance. Specifically, participants received monetary compensation for weekly research visits ($15–$25) to offset participation costs (e.g., travel) and received additional adherence incentives, such as water bottles, notebooks, and monetary compensation, for completing adherence milestones.
Due to a high frequency of early dropouts at the beginning of the trial and participant reported attendance barriers that became evident only after randomization of participants had begun, the study team determined that a more structured and formal retention-related protocol should be added prior to randomization. The Fireside Chat was therefore developed to use with every participant by study teams at each site, including the site PI, before a decision to randomize was made by the team and participants. The Chat was conceptualized and developed by Houston, Texas-based study team members and study leadership.
Following implementation of the Fireside Chat (approximately midway through STRIDE recruitment at the Houston site), The Chat occurred with each potential participant and included all members of a site study team, to identify barriers to retention, minimize negative effects from these barriers and build a foundation for ongoing retention efforts. It also helped both potential participants and staff determine the “fit” between the participant and study prior to randomization. The Chat was scheduled after informed consent was obtained and after most or all screening measures were completed and all other eligibility criteria were met. This avoided overburdening participants prior to determining their likelihood of meeting eligibility criteria. Parts of the informed consent were revisited during The Chat; however, it was not simply a more in depth informed consent discussion. The Site PI typically led the discussion, using a structured guidance tool (see Table 1), over a 30- to 60-minute period. This provided the opportunity for open, non-judgmental exploration of specific barriers and solutions in a thorough and structured way, and included educating participants about their study responsibilities and evaluating their understanding.
During The Chat, the study team first began by describing the reasons for The Chat and the importance of the research question. Further, study staff explicitly detailed the structure and time requirement for each visit type. Participants were asked to describe how the study might fit into their lives over nine months and were asked to prospectively identify solutions to staff-presented and participant-elicited attendance barriers during The Chat. Staff were mindful about superficial and socially desirable answers, and continued to re-state barriers to elicit a thoughtful, in-depth discussion about concerns regarding study involvement raised by both staff and participant. “What if” questions were frequently used to challenge participants to find alternative solutions to attendance barriers (e.g., What if your work schedule changes? How else could you make it to visits?). Sources of outside social support were investigated as well (e.g., What friends and family could you ask for help if you have competing demands on your time?), which often gave a clearer picture of participants’ abilities to get rides from friends/family or receive childcare assistance, in order to complete study visits. The opportunity to discuss the plausible course of participation helped participants fully envision the experience prior to randomization. Participants were often randomized in the STRIDE study the day following The Chat, with some randomized on the same day.
The team considered participants’ reactions before final STRIDE randomization decisions were made. Consistent with eligibility criteria in this hybrid efficacy-effectiveness trial, candidates could be excluded if deemed to be at high risk for dropping out. For example, some candidates were excluded if at risk for non-attendance due to onerous study-related travel. While this criterion was in place prior to The Chat protocol implementation, The Chat helped evaluate this in a thorough, thoughtful manner. If a participant did choose to be randomized and had difficulty making visits during the trial, follow-up Chats revisited barriers and potential solutions. The Chat was often revisited at the beginning of the continuation phase (week 13). Chat topics, talking points, questions, and the intent of questions are thoroughly described in Table 1.
The Chat was disseminated to all nine sites via national training phone calls and e-mails from study leadership. Participant study attendance before and after Chat implementation, however, are only available from the Houston-based site (N=29), which rigorously implemented The Chat, and will be presented here. Fifteen participants had already been randomized in the STRIDE study when The Chat protocol was implemented. Thus, 14 participants who enrolled in the STRIDE study at the Houston site following implementation of the Chat protocol participated in a STRIDE pre-randomization Chat, and the 15 previously enrolled participants did not.
During the acute phase (weeks 1–12), 36 intervention visits were possible (for Exercise and Health Education participants). During the continuation phase (weeks 13–36), Exercise participants could complete a total of 72 sessions (24 weekly intervention visits at the RTP and 48 home-based exercise sessions), while Health Education participants could complete a total of 24 (RTP-based) visits. Beginning with the continuation phase (week 13) participants completed a longer assessment visit every 4 weeks until study completion (week 37); attendance at these 7 visits were used to compare research-visit retention.
Means of intervention and research visit attendance were compared across the participants who did not receive The Chat (n=15 [n=7 Exercise; n=8 Health Education]) and those who did (n=14 [n=9 Exercise; n=5 Health Education]). Statistical comparisons were made using T tests (PROC TTEST; SAS 9.3 [Cary, NC]; Satterthwaite variance method) and evaluated with 2-tailed tests at the alpha=0.05 level.
The mean age of participants at the Houston site was 34.4 years (SD=10.8) and participants were predominantly male (62.1%, n=18) and White, non-Hispanic (82.8%, n=24). The remaining participants were Black (17.2%, n=5).
Participants in both intervention conditions who received The Chat attended significantly more intervention visits during the acute phase compared to those who did not receive The Chat, t(20)=−4.28, p<0.001 (32.2 visits [SD=5.3] vs. 18.3 visits [SD=11.3]; see Figure 1). Sensitivity analyses found that these results did not differ by condition within Chat status. Specifically, among participants who did not receive The Chat, Health Education participants attended more acute phase intervention visits than Exercise participants (22.8 [SD=12.7] vs. 13.3 [SD=8.5] visits, respectively) but the difference was not significant (t=−1.77, p=0.10). Similarly, among participants who received The Chat, no significant differences were found between Health Education and Exercise participants for intervention visits in the acute phase (34.6 [SD=1.7] vs. 30.9 [SD=6.2] visits, respectively, t=−1.69, p=0.12).
In the continuation phase, Exercise participants who received The Chat attended significantly more intervention visits, t(10)=−2.38, p<0.05 (24.7 visits [SD=22.2] vs. 6.0 visits [SD=6.7], respectively). Similarly, Health Education participants who received The Chat attended more intervention visits during the continuation phase, t(11)=−3.02, p=0.01 (18.0 [SD=3.6] vs. 8.8 [SD=7.4] visits, respectively). Participants in both conditions who received The Chat attended significantly more research visits (out of 7 possible visits) as well, t(24) =−4.46, p<0.001 (6.4 [SD=1.3] vs. 3.5 [SD=2.2] visits). Sensitivity analyses revealed that Health Education and Exercise participants who did not receive The Chat did not differ on research visit attendance (3.9 [SD=2.2] vs. 3.0 [SD=2.2] visits, respectively, t=−0.77, p=0.45). This finding was consistent among participants who received The Chat (6.8 [SD=0.4] vs. 6.2 [SD=1.6] visits, respectively, t=−0.99, p=0.34).
This secondary data analysis demonstrated the potential for a thorough, highly engaging retention protocol with participants (prior to randomization) to potentially mitigate missing data, particularly NMAR data (e.g., missing due to issues with intervention adherence). Indeed, the results at our site were dramatic, with approximately 90% of acute phase intervention and research visits completed after implementation of The Chat—an excellent marker of study attendance and retention in this field (Hansen, et al., 1990).
Post-hoc analyses of factors associated with attendance and attrition are common in this area of research (e.g., Claus, et al., 2002; Northrup, Green, Evans, & Stotts, 2014); however, prospective studies (secondary to the main trial) focused on increasing retention are less so. A weight-loss trial (with 6-, 12-, and 18-month follow-up visits) employed Motivational Interviewing and “active learning” in a group format, prior to randomization, to address participant retention and expectations. Only the first 51 participants (of 162 total) took part in the pre-randomization groups, and the researchers reported ≥94% attendance at all three visits, regardless of whether a participant took part in the pre-randomization groups (Goldberg & Kiernan, 2005). Similar efforts to prevent individuals from dropping off waitlists for publicly funded substance abuse treatment have not met with success (Donovan, Rosengren, Downey, Cox, & Sloan, 2001) but these studies may differ from the present study in important ways. For example, administering a retention/problem-focused protocol close to the time of research participation may capitalize on when participants’ motivation peaks and fully engage their problem-solving skills; whereas, patients’ motivation for change while on a waitlist may wane while their treatment status remains in limbo.
Our preliminary data suggest the Fireside Chat is promising; however, our sample size was modest and our pre-post design precludes drawing a stronger conclusion about the association between The Chat and improved study visit attendance. Other explanations and potential sources of bias cannot be ruled out. For example, there may be time effects in that our staff may have improved in efficiency and knowledge over the life of the trial. Procedural changes in response to identified barriers may also have affected attendance. Prospectively designed studies with random allocation are needed to fully assess the efficacy of this approach. We acknowledge that other research designs may not allow for a multiple-day baseline, with at least a day for participants to consider the full meaning of research involvement. Abbreviated Chat adaptations may prove feasible depending on the research context, and this approach could be modified to address other research-specific problems (e.g., medication adherence); however, The Chat may be inappropriate for some research designs. In spite of these limitations the exemplary attendance/retention markers following Chat implementation at our site suggest promise for a formalized retention protocol to augment routine consenting and screening procedures with participants and the need for further evaluation of this method.
In closing, the myriad barriers to research engagement that participants face require careful consideration by all stakeholders involved in research. The time investment for The Chat by potential participants and research staff is not trivial but it could produce important benefits for the study. Indeed, the increased validity of study results when less data are missing and staff time savings as the study progresses (e.g., less need for efforts to contact participants who have missed visits) are both quite important. Cooperative dialogue, beginning early in the recruitment phase and prior to randomization, is critical to engaging and retaining participants in SUD research as the collaborative partners we intend them to be.
Role of Funding Sources
STRIDE (CTN-0037) was supported by the National Institute on Drug Abuse of the National Institutes of Health under Award Number U10DA020024 and UG1DA020024 (PI: Trivedi). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Dr. Greer has received research funding from NARSAD and consulting fees from H Lundbeck A/S and Takeda Pharmaceuticals International, Inc. Madhukar H. Trivedi is or has been an advisor/consultant to: Alkermes, AstraZeneca, Bristol-Myers Squibb Company, Cephalon, Inc., Cerecor, Concert Pharmaceuticals, Inc., Eli Lilly & Company, Evotec, Forest Pharmaceuticals, GlaxoSmithKline, Janssen Global Services, LLC/Janssen Pharmaceutica Products, LP/Johnson & Johnson PRD, Lundbeck, MedAvante, Medtronic, Merck, Mitsubishi Tanabe Pharma Development America, Inc., Naurex, Neuronetics, Otsuka Pharmaceuticals, Pamlab, Pfizer Inc., Phoenix Marketing Solutions, Rexahn Pharmaceuticals, Ridge Diagnostics, Roche Products Ltd., Sepracor, SHIRE Development, Sunovion, Takeda, Vivus, and Wyeth-Ayerst Laboratories. In addition, he has received research support from: Agency for Healthcare Research and Quality (AHRQ), National Alliance for Research in Schizophrenia and Depression, National Institute of Mental Health, and National Institute on Drug Abuse.
The authors wish to thank the research and clinical staff at the Houston site for their support of this research.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of Interest
Drs. Northrup, Walker, Rethorst, Warden, & Stotts declare that they have no conflicts of interest.
ContributorsAll authors developed and assisted with implementation of the pre-randomization protocol (i.e., The Fireside Chat). Dr. Northrup conducted the statistical analyses for this manuscript and wrote the first draft, in consultation with Drs. Greer, Walker, Rethorst, Warden, Stotts, and Trivedi. All authors contributed toward and provided edits on several versions of the manuscript as well. All authors approved the final manuscript.