|Home | About | Journals | Submit | Contact Us | Français|
The burgeoning number of empirically validated therapies has not been met with systematic evaluation of practical, inexpensive means of teaching large numbers of clinicians to use these treatments effectively. An interactive, computer-assisted training program that sought to impart skills associated with the Project MATCH (Matching Alcoholism Treatments to Client Heterogeneity) Twelve-Step Facilitation (TSF) manual was developed to address this need.
Twenty-five community-based substance use-treatment clinicians were randomized to one of two training conditions: (1) access to the computer-assisted training program plus the TSF manual or (2) access to the manual only. The primary outcome measure was change from pre- to posttraining in the clinicians' ability to demonstrate key TSF skills.
The data suggested that the clinicians' ability to implement TSF, as assessed by independent ratings of adherence and skill for the key TSF interventions, was significantly higher after training for those who had access to the computerized training condition than those who were assigned to the manual-only condition. Those assigned to the computer-assisted training condition also demonstrated greater gains in a knowledge test assessing familiarity with concepts presented in the TSF manual.
Computer-based training may be a feasible and effective means of training larger numbers of clinicians in empirically supported, manual-guided therapies.
For many years, treatment approaches based on the 12 steps of Alcoholics Anonymous and Cocaine Anonymous were widely used and very popular in the treatment community but had comparatively little empirical support from controlled clinical trials (Miller et al., 1995; Morgenstern et al., 1997; Tonigan et al., 1996). Recently, however, several rigorous randomized clinical trials have supported the efficacy of well-defined, manualized, 12-step-oriented treatments (Humphreys et al., 2004). Twelve-Step Facilitation (TSF) (Nowinski et al., 1992) was developed for Project MATCH (Matching Alcoholism Treatments to Client Heterogeneity; Project MATCH Research Group, 1997) to provide a manual-guided, professionally delivered, structured approach that approximated widely used counseling models that invoked the 12 steps and involvement in the fellowship of Alcoholics Anonymous as essential features. The results of Project MATCH suggested that TSF was comparable with cognitive-behavioral therapy and motivational enhancement therapy on most outcomes of several indicators of measures of abstinence and superior to the others at the 3-year follow-up (Project MATCH Research Group, 1998). TSF has also been associated with higher rates of self-help involvement, which in turn has been associated with less alcohol use (Brown et al., 2002; Connors et al., 2001; Humphreys et al., 2004; Owen et al., 2003). TSF and closely related approaches have also been demonstrated to be as effective as, or more effective than, other approaches among several samples of drug users (Carroll et al., 1998b; Crits-Christoph et al., 1999; Wells et al., 1994).
Despite emerging support for the efficacy of TSF, it has proven to be challenging to disseminate TSF and other empirically validated treatments to the clinical community. Many clinicians have limited access to comprehensive training in TSF or other empirically validated therapies (Lamb et al., 1998). Workshops in some empirically supported therapies are becoming more available, but the training sessions are usually brief (e.g., workshops of several hours' duration) and therefore may be unlikely to produce lasting change in clinician's ability to implement new therapies (Walters et al., 2005).
Moreover, it should not be assumed that counselors, even those espousing a 12-step model, can implement TSF without training. Although based on standard counseling models, TSF differs from them in several ways. These differences include TSF's strong emphasis on therapist support, discouragement of aggressive “confrontation of denial” and therapist self-disclosure, and highly focused and structured format. In Project MATCH, supervisors' ratings indicated wide variability in baseline levels of basic psychotherapeutic skills among the predominantly master's-level TSF therapists (Carroll et al., 1998a), and the TSF therapists in Project MATCH required more training cases and supervision to reach acceptable levels of adherence to manual guidelines (Carroll et al., 1994).
There are also very little data on the most effective means by which “real-world” clinicians should be trained to use empirically supported therapies (Baer et al., 2004). The lack of empirical data on the effectiveness of training was recently addressed by two studies that randomized real-world clinicians to different types and intensities of training in specific empirically validated therapies (Miller et al., 2004; Sholomskas et al., 2005). Both studies demonstrated that that intensive workshop training, followed by individualized coaching, feedback, and supervision, was effective in imparting adequate skill levels to community-based clinicians with respect to exposure to the treatment manual alone. Nevertheless, the training strategy of didactic workshops followed by supervision and coaching is both expensive and time intensive. It is hence unlikely to be feasible for training large numbers of clinicians to use empirically supported therapies. Effective and less costly methods of training large numbers of real-world clinicians are needed.
The recent availability of distance learning methods, particularly computer and Web-based training for clinicians, is a novel model that has been used in several areas of health care (Huang and Alessi, 1996; Issenberg et al., 1999; Jones and Cookson, 2001; Piemme, 1988; Todd et al., 1998). Large general reviews have suggested that computer-assisted training has a moderate effect size (.30) and can reduce training time up to 30% compared with traditional instruction (Stephenson, 1994). Given the time constraints, variable educational background, and high rates of turnover among substance-use clinicians, computer-based training may offer a number of potential advantages, including flexibility in scheduling, allowing more clinicians to access training, increasing opportunities for practice; reduced cost; and increased flexibility and individualization in pace and material covered (Williams et al., 2001).
There are, however, virtually no data from controlled trials on the effectiveness of computer-assisted approaches in training clinicians to effectively implement empirically supported substance-use therapies. No trial to date has used rigorous methodology (e.g., pre-post designs with random assignment to training conditions with blind assessment of outcome) to evaluate the effectiveness of computer-based training relative to alternative training strategies. In this report, we describe a randomized training trial comparing the relative efficacy of two methods of training community-based clinicians to implement TSF: (1) exposure to the TSF manual alone or (2) access to the manual plus an interactive multimedia CD-ROM designed to teach TSF skills and techniques.
Participants were 28 clinicians who volunteered to participate in the trial and who provided written informed consent. The participants were required to be currently employed full time as a clinician treating a predominantly substance-using population and to have access to a personal computer. Clinicians were recruited through newsletters and direct contact with clinics throughout the state of Connecticut. Forty-four clinicians were initially contacted; 28 consented to participate and completed baseline assessment. Three dropped out before completing posttreatment assessments.
A manual-only training condition was selected as a reasonable reference condition against which to compare the CD-ROM program based on previous work in this area (Miller et al., 2004; Sholomskas et al., 2005). After completing baseline assessments, clinicians in the manual-only condition received a copy of the National Institute on Alcohol Abuse and Alcoholism TSF manual (Nowinski et al., 1992). Clinicians were instructed to spend at least 10 hours reading the manual and practicing TSF techniques during the next 3 weeks.
Clinicians in the other condition were given both the TSF manual and a CD-ROM, with instructions for installing and running the program on a personal computer. Most participants used the program in their own homes; others, with permission from their employers, used it at their workplaces. Clinicians were asked to spend a minimum of 10 hours working with the program during the next 3 weeks. Research staff was available to answer questions about the program and use of computers as well as to provide additional assistance as needed.
The content of the CD-ROM was drawn directly from the TSF manual and included multiple short vignettes of a TSF supervisor teaching a clinician TSF techniques, followed by vignettes of that clinician implementing TSF with a patient. It was structured in modules and covered the following content areas: (1) introduction to TSF, which reviewed the underlying theoretical foundations of TSF; (2) goals, structure, and objectives of TSF; (3) strategies for building rapport with clients within the TSF structure; (4) taking a drug and alcohol use history within TSF and providing feedback about loss of control and unmanageability (e.g., initial Step 1 work); (5) developing a treatment contract for abstinence as a goal and describing the roles of the TSF clinician and patient; (6) identifying and dealing effectively with resistance in TSF; and (7) facilitating Alcoholics Anonymous involvement and assigning recovery tasks (e.g., a personal journal). To reinforce learning, each section included several interactive tasks that included multiple choice, matching, and “fill in the blank” questions covering the information taught in each module.
Assessments were completed at baseline and 3 weeks after distribution of the manual or manual and CD-ROM. The primary outcome measure was the clinician's ability to demonstrate key TSF techniques via a videotaped role-play exercise in which the participants were asked to demonstrate five key TSF skills: (1) taking a TSF substance-use history, (2) assessing unmanageability and giving feedback to the patient, (3) contracting for TSF treatment goal of abstinence, (4) clarifying the roles of the patient and therapist and confronting denial/resistance to 12-step recovery, and (5) encouraging the patient to get involved in Alcoholics Anonymous/Narcotics Anonymous/Cocaine Anonymous as a recovery task. Three experienced clinicians, who had been trained to follow a standardized script with minimal prompting, played the part of a substance-dependent patient in the role-plays. The role-plays were videotaped for independent evaluation of adherence/skill and took about 1 hour to complete.
The Yale Adherence Competence Scale (YACS; Carroll et al., 2000), a general system for evaluating therapist adherence and skill across several types of manualized addiction treatments, was used to evaluate the extent to which the clinicians were able to demonstrate TSF skills. The YACS has been demonstrated to have excellent interrater reliability (intraclass correlation coefficients [ICCs] of .85 or greater) in several studies and to sharply discriminate TSF from other treatments (Carroll et al., 1998a,b, 2000, 2001). For each item, raters evaluated the clinician on two dimensions using a 7-point Likert-type scale. First, they rated the extent to which the clinician covered the intervention thoroughly and accurately (i.e., adherence); and second, they rated the skill with which the clinician delivered the intervention (i.e., competence), in which a score of 4 or more is consistent with the criterion level that would be required for certifying the therapist to participate in a clinical trial. The mean ICC estimate (fixed effect model) for the adherence and competence ratings (pooled) in the present study was .93 (Shrout and Fleiss, 1979), using a subset of five tapes that were rated by all three experienced master's-level process raters, blind to the clinicians' training condition. The assessment battery also included a TSF knowledge test, with 34 items drawn directly from the TSF manual. This was intended as a secondary measure to assess whether the training methods had an effect on the clinicians' knowledge of TSF.
Baseline demographic and experience characteristics for the clinicians were assessed through t tests for continuous variables and chi-square tests for categorical variables. Changes in the independent ratings of adherence and skill on the role-plays were assessed via repeated measures analyses of variance (ANOVAs), using mean scores for the five individual items.
Table 1 presents demographic and experience characteristics of the participants at baseline. Most were women, almost half had master's degrees, and they averaged 11 years of experience in working with substance users. Thirty-six percent reported they had been treated for a substance-use problem in the past. There were no significant differences in any of these baseline variables by training condition.
Changes in the clinicians' ability to demonstrate TSF skills as assessed by adherence and skill scores are presented in Table 2. Effects for time were statistically significant for both the adherence and competence dimension mean scores, suggesting that the group as a whole improved their performance from pre- to posttraining across the five skills assessed. Moreover, for both the adherence and the skill dimensions, repeated measures ANOVAs suggested that clinicians assigned to the CD-ROM condition made significantly greater improvements than those assigned to the manual-only condition (Adherence × Condition: F = 6.7, p = .017; Skill × Condition: F = 7.9, p = .01). Effect sizes, expressed as Cohen's d (Cohen, 1988) for the adherence and skill dimensions and averaged across the five assessed components, were 0.86 and 1.1, respectively.
Changes from baseline to posttraining on the TSF knowledge test are also presented in Table 2. Before training, clinicians correctly answered an average of 55% of the questions. Repeated measures ANOVA indicated an overall effect for time (F = 46.9, p < .001), suggesting that scores improved for the group as a whole from pre- to posttraining. There were also significant Group × Time effects, suggesting a larger increase in scores for the group assigned to the CD-ROM condition (F = 4.8, p = .01). Clinicians assigned to the CD-ROM condition correctly answered 70% of the questions at the posttraining assessment. For both the knowledge test and the adherence and skill ratings, there were no significant effects for the clinicians' recovery status or Recovery Status × Training Condition.
The clinicians' report of their use of TSF techniques in their clinical work was assessed via self-reports at follow-up. Overall, there were few statistically significant between-condition differences between the two training conditions. Both groups indicated moderately high levels of satisfaction with the TSF manual and indicated that the TSF approach was highly consistent with their regular clinical practice. The two groups reported spending comparable amounts of time reading the manual before the posttraining assessment (a mean of about 5 hours); those assigned to the CD-ROM condition indicated they spent a mean (SD) of 9.3 (16.2) additional hours working with the computer program.
This randomized training study trial of three strategies for training real-world counselors to use a manualized TSF approach suggested that the clinicians' ability to implement TSF, as assessed by independent ratings of adherence and skill for five key TSF interventions, was significantly higher after training for those who had access to an interactive CD-ROM designed to impart skills in TSF compared with those who were assigned to the manual-only training condition. Those who were assigned to the CD-ROM condition also evidenced greater gains in a knowledge test assessing familiarity with concepts presented in the TSF manual. No significant effects of the clinicians' self-reported recovery status were seen on adherence, competence, or knowledge scores. This group of counselors appeared to be enthusiastic about the computer-assisted training method evaluated here and rated it highly.
Taken together, these findings provide initial support for the feasibility and potential efficacy of distance learning methods for therapist training in the treatment of addictions. It was striking that among this group of clinicians (virtually all of whom reported they were very familiar with the TSF approach before training), independent evaluation of their pretraining tapes suggested that their levels of adherence and competence in these techniques fell short of the benchmark criterion level that would be required for certification (and perhaps competent practice) in this approach. Although reading the TSF manual was associated with measurable gains in these ratings, significantly greater increases were seen in the CD-ROM condition. This is consistent with previous observations that even experienced therapists require some training and feedback to reach competence in manual-guided counseling approaches (Crits-Christoph et al., 1998). Unlike our previous training study evaluating training methods for cognitive-behavioral therapy (Sholomskas et al., 2005), recovery status was not associated with the effectiveness of the training method. It should be noted, though, that clinicians who had a substance-use history themselves may have been more familiar with the TSF model than they were with cognitive-behavioral therapy.
The strengths of this study included random assignment of clinicians to training conditions, independent evaluation of the clinicians' adherence and skill in delivering key TSF skills via blind ratings of pre- and posttraining videotapes, and evaluation of a widely distributed treatment manual. Limitations included a fairly small sample size; nevertheless, significant effects of training condition on skill ratings and TSF knowledge were seen and the effect sizes were moderate to large. Moreover, this group of clinicians was similar in terms of demographic, educational, and experience level to other substance abuse clinicians in the state of Connecticut (DMHAS, 2002) and the National Institute on Drug Abuse Clinical Trials Network (Ball et al., 2002; Roman et al., 2006). Another limitation was the lack of measures of the impact of the training conditions on patient outcomes, as this study was designed to assess only the impact of different training conditions on clinicians' ability to demonstrate key TSF techniques. Moreover, clinicians assigned to the CD-ROM condition spent more time in training, and thus increased training time alone may account for the effects suggested here, although increased time in training may also be a benefit of computer-assisted training programs such as these. The results of this study do suggest the promise of distance learning methods as a strategy to impart key skills to larger groups of clinicians. As one of only a handful of well-controlled randomized trials systematically evaluating different means of training clinicians in empirically supported therapies, this study also represents an important initial step in bridging the dissemination gap (Carroll and Rounsaville, 2003).
*This research was supported by National Institutes of Health Awards R44 AA014456-02, P50 DA09241, and K05-DA00457 to Kathleen M. Carroll.