Results of the review of training for multi-site trials in behavioral treatment for addictions are displayed in . Each of the elements in the training process, represented as columns in the table, are described below.
3.1. Therapist Selection
Across the 12 studies there was considerable variation in how therapists were identified, screened, and selected. Two different approaches to identifying study therapists were employed based on clinical setting and research design. Three studies (MATCH, Cocaine Collaborative, COMBINE) offered treatments primarily in research-affiliated settings and recruited clinical staff specifically for the multi-site trial. Other studies, including UKATT and all CTN trials, expected clinical staff to be employed (salaried or fee-for-service) within the participating community treatment agencies, most of which had not conducted treatment research previously. Cannabis Youth employed both of these methods differentially for its five therapeutic interventions.
Regarding screening and selection, seven studies required experience in treating addictions, typically 2 years, and five studies required specific professional educational achievement beyond bachelor’s degrees, most typically master’s degrees. Two studies required therapists to be allied professionally with the therapeutic approach under investigation. In contrast, some studies (i.e., CTN MI/MET; CTN Women and Trauma) excluded therapists who had prior training in the experimental treatment. Six studies used specific performance screens, including taped samples of clinical work to evaluate specific therapeutic skills (MATCH, Cocaine Collaborative, Women and Trauma, BSFT, UKATT), or taped role plays or clinical samples to observe more general skills such as empathy (COMBINE). Of the seven CTN studies included in this review, only two (Women and Trauma and BSFT) required that therapists demonstrate specific clinical skills for selection. Other CTN studies generally accepted therapists found in collaborating agencies who were interested in participating with relatively minor requirements (i.e., being a non-smoker to lead smoking cessation treatments, or having some experience in psycho-educational groups to lead Job Seeking group programs).
Variability in therapist selection appears to be a central dimension in external validity. Those studies seeking to demonstrate comparative efficacy of relatively new treatments or interactions between treatment and individuals tended to have the most restrictive procedures. For example, in studies comparing the efficacy of different treatments, such as MATCH and Cannabis Youth, therapists with demonstrated experience and commitment to the therapeutic approach were sought. Both UKATT and COMBINE used clinical skill performance on tapes to determine eligibility prior to training. In contrast, most CTN studies (but not all, see BSFT) where treatments were compared to “treatment as usual” and conducted in community agencies, requirements for therapist selection were quite modest.
3.2. Centralized Training
There was little variability across the 12 studies with respect to the use of centralized trainings. All but one study brought therapists from multiple sites to a single location for group training prior to study initiation. The length of centralized training ranged from 1 to 4 days, with the exception of the Cocaine Collaborative, in which there were 4 separate 2-day trainings (8 full days) and CTN BSFT, which included one 4-day and three 3-day workshops (13 days total). All studies reported that training was based on detailed treatment manuals of the therapeutic method. All studies reported training to include didactic portions for background and rationale of treatment, reviews of specific treatment procedures, and demonstrations of treatment elements by video and/or role play. Variations in the amount of time devoted to centralized training presumably reflected differences in the nature and complexity of the behavioral treatments themselves, as well as the training goal. For example, the CTN Safe Sex Skills, which involved training of a psycho-educational group treatment for safer sex practices adjunctive to opioid treatment, required practice but no training beyond the initial centralized or core training. Other studies of more complex stand-alone treatments (e.g., Cocaine Collaborative and CTN BSFT Protocol) required considerable post-training practice and review after centralized training and prior to a certification process.
The one exception to the centralized training methodology was CTN MI/MET. Using an innovative design, each participating “node” (partnership of an research center and several community treatment programs) in the CTN was asked to identify a local MI Trainer who was trained centrally and subsequently conducted training (as well as supervision) of participating therapists and supervisors at the local performance sites.
All studies provide descriptions of certification procedures to ensure adequacy of therapist performance. The most common procedure involved audio- or videotaping a work sample (pilot care) that was reviewed by an expert in the treatment method. In one study, CTN Safe Sex Skills, certification was based on observed role-plays at a central training. In some studies where local supervision was available, there was an option for certification based on direct observation of actual intervention rather than tapes (CTN Smoking Cessation, CTN Job Seekers). Certification was based on session ratings using measures tailored to assess performance of specific clinical procedures (adherence measures are reviewed below). The specific decision rules for certification (cut scores on scales, specific behaviors observed on tape) were not described consistently.
3.4. Supervision and Therapist Support
In all multi-site trials reviewed, ongoing supervision and support of clinical staff was used to promote standardization and reliability of therapeutic activities. Supervision in this context was conceived as coaching, mentoring, and direction to support consistent and high fidelity treatment delivery. We have focused on the following dimensions as shown in : Supervision format, supervisor selection, supervisor training and certification, and monitoring format, setting and methods.
Supervisor qualifications include demonstrable knowledge and expertise in the content area as well as the process of supervision. A number of surrogate indicators were used in the studies we reviewed to determine qualifications and selection criteria. Key among these were professional degrees and credentials. Current position, years of experience, peer nomination (of an acknowledged expert), and unique skill sets were also used. None of the studies reviewed utilized an explicit process to establish the eligibility of supervisors, viz., core competencies for the role. An interesting distinction in supervisor selection seems to exist, however, between the NIDA CTN and other studies. The five published trials uniformly employed nationally acknowledged experts on specific treatments, whereas CTN trials, with some exceptions, engaged local clinical and research supervisors at study sites.
There was considerable variability in supervisor training across the studies reviewed. In one CTN study (BSFT) and three published studies (MATCH, Cocaine Collaborative, Cannabis Youth), no supervisor training (nor selection) was described, presumably obviated by the selection of national experts or previously trained individuals. Other studies, to varying degrees, described supervisor training that included both the content and process of supervision. For those instances in which supervisors received protocol-specific training, a session or mini-course of 4–16 hours typically was appended to a centralized training for clinical and research staff. In the UKATT study, supervisors were themselves supervised by a national expert. Some of these protocols required certification of supervisor proficiency by national experts using established criteria, both objective and subjective. Partial certification also was observed as in the CTN HIV-Detox study in which supervisors were certified as adherence raters using established criteria, but were not certified for other aspects of supervision. Three studies (CTN Women and Trauma, CTN Safe Sex Skills, and CTN HIV-Detox) evaluated the performance of study supervisors by requiring expert review of a subset of supervisor ratings. Approximately 25% of all sessions rated by the supervisor were co-rated by expert supervisors. Discrepancies in coding were reviewed on supervisor group conference calls run by intervention experts. In this way, the supervisors received feedback on their individual ratings and learned from the issues that arose in ratings by other supervisors.
The actual methods of supervision ranged considerably across studies, from on-site assistance with daily problem solving to formal weekly or bi-weekly communication between supervisors and their clinical or research staff. Supervision of clinical staff was described in all cases as based on performance criteria, but the mode of and tools for monitoring skill performance varied widely. The most rigorous approach called for supervisors to rate performance of the study clinicians with an objective assessment scale or skill check list using direct observation, video- or audiotape recording (i.e., MI/MET, HIV-Detox, Smoking Cessation, CTN Women and Trauma, CTN Safer Sex). In other cases, supervisors conducted regular case reviews with their study clinicians to discuss challenges they faced in implementing the protocol (i.e., CTN Job Seekers, CTN Smoking Cessation). Some clinicians were required to maintain detailed logs of their activity (e.g., Cannabis Youth).
Both centralized and local monitoring and supervision of behavioral treatment delivery were employed. While local supervision presumably employed face-to-face meetings, most studies relied heavily upon teleconferences for centralized supervision. Yet, across studies, conference calls varied with respect to participants (supervisors only, supervisors and counselors, or counselors only) and content (feedback from review of the adherence scales or general procedures and discussion surrounding the delivery of treatment). For example, the Cocaine Collaborative, COMBINE, Cannabis Youth, UKATT, MI/MET, Women/Trauma, Smoking, Job Seekers, Safe Sex Skills, HIV-Detox and BSFT trials made use of conference calling with central authorities. The Cocaine Collaborative, COMBINE, Cannabis Youth, UKATT, MI/MET, Women and Trauma and HIV-Detox studies utilized centralized expert raters for adherence and treatment review, which guided some of the discussions on conference calls. In other trials, such as CTN Smoking, the conference calls were guided by feedback from the on-site supervisors’ review of counselor adherence and competence with experts addressing the issues that were raised by the supervisors.
Many of the trials that included centralized monitoring also included some level of local monitoring. In Project MATCH, Cannabis Youth, and CTN MI/MET protocols, therapists completed self rating checklists in addition to the centralized expert ratings. Other CTN studies such as HIV-Detox, Women and Trauma, MI/MET, Safe Sex Skills, Smoking, and Job Seekers had local supervisors rate therapist delivery of treatment with adherence/competence scales. MI/MET utilized local experts as supervisors, who were coordinated by national experts who did not themselves interact with clinicians except when low fidelity was detected. Within all local monitoring, supervision ranged from weekly to monthly, or in some cases was based on counselor performance, with supervision decreasing as counselors demonstrated stable proficiency (e.g., CTN Women and Trauma and Safe Sex Skills).
We observed no obvious relationship between the content or complexity of the behavioral intervention and the supervisory processes used to support it. Given the range of potential supervisory activities, the presumed importance of supervision for therapeutic quality and consistency, and the many elements that could account for variability of supervision in multi-site trials, it is noteworthy that we were unable to find any systematic evaluations of supervisory processes. Indeed, we concluded that none of these multi-site studies subjected supervision to systematic inquiry.
3.5. Adherence Ratings Scales/Procedures for low fidelity
There was considerable similarity across studies in adherence scale procedures. All of the studies in this review used rating scales to monitor or evaluate therapist adherence to study interventions. With regard to construction, these scales ranged from session- specific content elements to broad domains of intervention methods. Differences along this dimension appeared to be related to the complexity of the intervention being assessed. Rating scales for simpler interventions, such as the CTN-HIV Detox study’s single session therapeutic alliance intervention, itemized very specific intervention content. Ratings for more complex interventions, such as BSFT, itemized general intervention methods (e.g., use of “reframing”). Scales also differed in the measurement of competence or quality of intervention delivery as distinct from adherence, two factors that have been found predictive of treatment outcome (Barber et al., 1996a
). Not all rating scales distinguished content adherence from quality of delivery. Eight of the 12 studies reviewed explicitly rated quality, labeled variously as quality, competence or skillfulness. Seven of those used scales that required separate ratings of content and skillfulness for the same intervention elements. Several studies (e.g., Cannabis Youth, CTN MI/MET, MATCH), employed an additional approach, using a general skillfulness scale to rate quality of treatment delivery across different interventions. This allowed for a direct comparison of therapist skill in treatment delivery across the interventions studied, a benefit not available with the methods used in other studies.
All studies reviewed used rating scales for initial therapist certification, establishing criterion scores for certification and ongoing quality standards. All studies also rated a selection of taped sessions across study implementation to assess and reduce intervention drift. Studies varied regarding who conducted ratings: independent experts, central supervisors or local supervisors. However, all of them used ratings’ feedback as part of the supervision process to correct protocol deviations and improve quality. The majority of these studies used adherence scales scores to indicate low fidelity, although the specific criterion used was not always specified. In studies using this practice, therapists who fell below criterion performance received enhanced supervision/training until performance exceeded threshold standards or they were suspended from accepting randomized participants. Most studies first used “redlining warnings” (Miller et al., 2005
), a procedure in which monitoring, supervision and refresher training of therapists is increased while they continue to treat study participants. This was particularly true for studies employing group interventions, in which clinical continuity and cohesion were important. One study reviewed, the CTN-HIV Detox study, temporarily suspended therapists from accepting new study participants immediately upon falling below performance criteria.
Most rating instruments used did not have established psychometric properties. The Cocaine Collaborative reported initial reliability and validity data for adherence and competence scales developed for individual drug counseling, one of its three interventions (Barber et al., 1996b
). For a second intervention, supportive expressive therapy, it used a scale with previously established reliability/validity data (Barber and Crits-Christoph, 1996
). Other studies used unpublished, but previously developed scales (e.g., Seeking Safety Adherence Scale by Najavits, (2003
) for CTN’s Women’s Treatment for Trauma, the Family Therapy Skills Rating System for BSFT by Santisteban), adapted previously published instruments (CTN MI/MET), or developed new scales to assess adherence (CTN Smoking Cessation study). Six of the 12 studies used some form of reliability check for ratings. For example, the CTN studies Women and Trauma, HIV- Safer Sex, and HIV Detox used a co-rating procedure in which supervisors’ and experts’ ratings were compared for selected sessions throughout the study. In COMBINE and MI/MET, central supervisory co-ratings were used as checks on ratings that fell below threshold performance standards. BSFT conducted two independent ratings followed by consensus agreement for initial therapist certification, and then a Head Training Supervisor held weekly meetings with supervisors and randomly listened to supervision sessions to minimize supervisor differences. Two studies, MATCH and MI/MET, did not check supervisory inter-rater reliability, but did use independent raters (blind to condition and results) to determine that the interventions were indeed reliably discriminable.
As described above, there is typically a high level of rigor and commitment of resources to the initial training of interventionists in multi-site clinical trials. Unfortunately, therapist attrition is a common problem for studies lasting extended periods of time. All trials reviewed described procedures for training new therapists during the study and conducted two or more additional trainings for replacement therapists. In three CTN trials, Smoking Cessation, HIV-Detox, and BSFT, sites were required to have back-up therapists who were certified (in HIV-Detox they were supervisors) and immediately available so as not to disrupt the treatment due to therapist attrition.
Subsequent trainings differed widely in their resemblance to the initial training. For example, MATCH trained as many additional, replacement therapists over the course of the trial as were trained in the initial cohort. These 10–12 subsequent trainings consisted of small numbers of clinicians, and the 3-day training was compressed into one-day. Although the original trainers were used for the face-to-face training, therapists were expected to assume greater personal responsibility for learning than was the case with the initial cohort: these additional trainings relied upon reading the manual, reviewing tapes of the original training event, watching session tapes, sitting in on supervision groups at the local site, and even beginning a pilot case prior to the face-to-face training. Although it is unclear whether the same pre-qualification criteria were always used as in the original cohort, the same post-training certification criteria were used before treating randomized cases.
Similarly, Cannabis Youth lost a significant number of therapists (1/4 of total) after the initial training. New therapists received a compressed training by the therapist coordinator, who had been an original trainer along with each treatment model developer, after reviewing training tapes, treatment manuals, and taped cases from their local site. In contrast, the time and resources involved in the initial training phase (which was a study in itself; Crits-Christoph et al., 1998
) for the Cocaine Collaborative made the repetition of this process prohibitive. New therapists were instructed to read the treatment manuals, watch the videotapes made of the initial four weekend training sessions, were assigned one or more training cases, and began seeing randomized participants after certification.
The CTN MI/MET protocols did not rely on a centralized training model for therapists, but rather centralized the training/calibration of local MI experts who provided both the initial training and subsequent re-trainings at their sites. Thus, re-trainings maintained the same pre-training expectations of therapists and number of training days, although trainings typically involved fewer therapists. Other CTN trials blended aspects of all of these approaches. For example, the initial centralized training for the HIV-Detox and Smoking Cessation protocols trained interventionists and supervisors together. Subsequent trainings for new interventionists were managed by the on-site intervention specialist who presented didactic material and reviewed the intervention manual with the trainee who was also required to watch videotapes of the centralized training and practice the intervention through role plays with the supervisor. Thereafter, the new therapist demonstrated the treatment protocol on standardized certification cases with teleconference input provided by the national expert. A similar remote training process was offered when replacement site supervisors were needed. Other CTN protocols (e.g., Safe Sex Skills; Job Seekers) similarly relied upon review of initial training videotapes and reading training manuals and materials under the direction of a local certified supervisor with minimal involvement from the national expert trainers or participation in an intensive, centralized face-to-face training. In contrast, the Women and Trauma protocol maintained central control over the training procedures by either conducting centralized trainings for new therapists or, when that was not logistically feasible, presenting all of the original training material via slideshow teleconference facilitated by one of the original trainers and videotape review with local role plays managed by the on-site, certified supervisor.