The 78 clinicians had a mean age of 45.5 (SD = 9.8) years, and 54% were women and 46% were men. Of the clinicians, 27% were African American, 8% were Hispanic, 61% were Caucasian, and 4% identified their ethnicity as other. A total of 49% of the clinicians had a master's degree, 28% had a bachelor's degree, 16% had an associates degree, and 7% had a high school diploma or a general equivalency diploma. The clinicians reported a mean of 9.0 (SD = 5.9) years of experience treating substance users, and 47% indicated they had a substance abuse problem in the past. The clinicians reported that they carried an average weekly caseload of 21 clients (SD = 14.4) and that a mean of 91% of their caseload comprised substance users. When asked how many hours of formal supervision they received per week, the most frequent response was 1 hr (47% of the sample). The clinicians were also asked to rate their level of familiarity with several approaches to substance use treatment using a 5-point, Likert-type scale ranging from 1 (not at all) to 5 (extremely). The clinicians indicated that they were most familiar with 12-step or disease-model approaches (M = 4.1, SD = 1.0), followed by interpersonal approaches (M = 2.9, SD = 1.2), motivational approaches (M = 2.7, SD = 1.0), and CBT (M = 2.4, SD = 1.0). The clinicians indicated that they were less familiar with other approaches (mean scores were 2.0 or less for CBT for depression or anxiety disorders, dialectical behavior therapy, and short-term dynamic therapy). Of the sample, 63% reported that they had previous exposure to treatment manuals, and 27% had used computer-aided self-instruction in the past. No statistically significant differences in these baseline characteristics by training condition were found. Moreover, there were no significant differences between the randomized (n = 54) and nonrandomized clinicians (n = 24) on baseline demographic, training, and experience characteristics. There were also no significant differences in baseline adherence and skill ratings or knowledge test scores.
At the posttraining assessment point, all clinicians reported that they had read the CBT manual, and all those assigned to the Web condition reported accessing the Web site at least once. Although the mean number of hours that the clinicians reported reading the manual was about half of what was requested, it did not differ by condition. For the manual only condition, the mean total time that the clinicians reported reading the manual was 9.2 hr (SD = 6.9), for the Web condition, the mean total time was 10.1 hr (SD = 5.7), and for the seminar plus supervision condition, the mean total time was 10.6 hr (SD = 7.3), F(2, 73) = 0.2, p = .82. Clinicians in the Web condition reported spending a mean of 15.7 hr (SD = 7.6) working with the Web site. In the seminar plus supervision condition, all clinicians completed all 3 days of the didactic seminar, for a mean estimate of 20 additional training hr, with an additional 3 hr of telephone supervision. Thus, the mean total hours of training completed was approximately 10 hr for the manual only condition, 26 for the Web condition, and 33 for the seminar plus supervision condition.
Change in Clinicians' Ability to Demonstrate CBT Techniques
As in previous evaluations of the YACS (Carroll et al., 2000
), ratings of both adherence and skill levels were found to have good levels of interrater reliability, as the mean intraclass correlation coefficient was .87 for the three sets of adherence ratings and .83 for the three sets of skill ratings. Adherence and skill ratings by training condition and time (pretraining, posttraining, and follow-up–postsupervision) are presented in . The first set of statistics presented reflects the Contrast × Time effects for pretraining–posttraining comparisons, and the second set refers to the Contrast × Time effects for the posttraining–follow-up comparisons.
Role Play Adherence and Skill Scores by Time and Training Condition
At the posttreatment assessment, all effects for time were statistically significant, suggesting that the group as a whole improved their performance for all three role plays. No main effects of contrast were statistically significant. For the adherence dimension, evaluation of Contrast × Time effects for the two primary contrasts (seminar plus supervision vs. manual only, Web vs. manual only) suggested that the clinicians assigned to the seminar plus supervision condition made significantly greater gains than those assigned to the manual only condition for two of the three role plays. Clinicians assigned to the Web condition tended to have higher adherence ratings compared with the manual only condition, but differences were not statistically significant. Cohen's d
) for the seminar plus supervision versus manual only comparison across the three role plays was 0.85, 0.73, and 0.50, respectively (mean effect size = 0.69). Effect sizes for the Web versus manual only comparison were 0.50, 0.20 and 0.10, respectively (mean effect size = 0.27).
For the skill dimension, clinicians assigned to the seminar plus supervision condition had significantly higher posttraining skills scores than those assigned to the manual only condition for two of the three role plays. Effect sizes for the skill dimension across the three role plays were 0.71, 0.64, and 0.48, respectively (mean effect size = 0.61). Clinicians assigned to the Web condition tended to have higher mean skill scores than those assigned to the manual only condition, but these were not significantly different at the posttraining assessment point. Effect sizes for the Web versus manual only condition for the skill dimension were 0.37, 0.21, and 0.09, respectively (mean effect size = 0.22).
Posttraining to Follow-Up
At the follow-up evaluation (which reflected postsupervision ratings for those assigned to the seminar plus supervision condition), both contrasts were significant for the first two role plays: introduction to CBT, adherence t(64) = 4.1, p = .001, skill t(64) = 4.2, p = .001; coping with craving, adherence t(64) = 3.2, p = .001, skill t(64) = 3.2, p = .001; seemingly irrelevant decisions, adherence t(64) = 1.8, p = .08, skill t(64) = 1.8, p = .07. This suggests that participants assigned to the seminar plus supervision and the Web conditions had significantly higher ratings than those assigned to the manual only condition. However, the Contrast × Time effects were not statistically significant, suggesting that clinicians in both these conditions retained the significant gains seen at the posttraining assessment point, but there was no further differential change by training condition. As illustrated in and , ratings for those assigned to the seminar plus supervision and Web conditions improved slightly during the follow-up–supervision period and decreased somewhat for those assigned to the manual only condition. Effect sizes for the seminar plus supervision versus manual only comparison for the adherence dimension at follow-up were 1.4, 1.4, and 0.47, respectively (mean effect size = 1.1); for the skill dimension they were 1.3, 1.2, and 0.44, respectively (mean effect size = 0.98). Effect sizes for the Web versus manual only comparison for the adherence dimension at follow-up were 0.81, 1.3, and 0.53, respectively (mean effect size = 0.88); for the skill dimension they were 0.81, 1.2, and 0.56, respectively (mean effect size = 0.86).
Figure 1 Role Play 1 (presenting the cognitive–behavioral therapy rationale and functional analysis): adherence scores by time and training condition. Scores range from 1 to 7, with higher scores indicating better adherence. The solid horizontal line indicates (more ...)
Figure 2 Role Play 1 (presenting the cognitive–behavioral therapy rationale and functional analysis): skill scores by time and training condition. Scores range from 1 to 7, with higher scores indicating better skill. The solid horizontal line indicates (more ...)
Because not all participants assigned to the seminar plus supervision condition submitted practice tapes and participated in the supervision sessions, exploratory analyses were conducted to evaluate whether those clinicians who participated in supervision had higher adherence or skill ratings than those who did not. For all role plays and for both the adherence and skill ratings, there were significant effects for time (indicating increases in adherence and skill ratings) and group (indicating a main effect for group, with higher scores overall for those who participated in supervision), but there were no significant Group × Time interactions.
Finally, we evaluated the three conditions in practical (e.g., clinically significant
) terms, that is, the effectiveness of the three conditions in terms of proportions of number of clinicians successfully trained. We used the criterion of final ratings of 3.5 or higher on two of the three role plays for both adherence and skill scores. This standard was selected because it is the criterion used to certify clinicians in previous clinical efficacy trials that have evaluated CBT (Carroll et al., 1998
) and is similar to the redline level concept from the National Institute of Mental Health Treatment of Depression Collaborative Research Project (Addis, 1997
; Shaw, 1984
). The percentage of clinicians meeting this criterion was 15% for the manual only condition, 48% for the Web condition, and 54% for the seminar plus supervision condition, χ2
= 67) = 7.7, p
Randomized Versus Nonrandomized Clinicians
To evaluate whether the results reported above reflected possible bias associated with the inclusion of nonrandomized clinicians, the primary analyses were repeated, including only the 54 clinicians who were randomized to one of the three training conditions. Although the mean scores did not change appreciably, the posttraining Contrast × Time effects were no longer statistically significant. For the full sample, mean posttraining effect sizes (which compared the seminar plus supervision condition with the manual only condition and averaged across the three role plays, Cohen's d
; Cohen, 1988
) were 0.69 for the adherence ratings and 0.61 for the skill ratings. For the randomized sample, effect sizes were 0.67 and 0.69, respectively. At follow-up, mean effect sizes were 1.1 for adherence and 0.98 for skill ratings for the full sample and 1.2 and 1.2, respectively, for the randomized sample.
Effect of Clinician Characteristics on Acquisition of Skills
Exploratory analyses that evaluated the effect of the clinicians' education level (master's vs. bachelor's degree or less), years of experience, and recovery status (self-identified as having had a substance abuse problem vs. not) on outcomes suggested the following: First, there were no significant main effects or interactions by training condition of level of education or years of experience on adherence–skill ratings. Second, there was some evidence that traditional face-to-face training was particularly helpful to the recovering subgroup. That is, there were several statistically significant Group × Time interactions on adherence and skill scores, suggesting greater improvement in the recovering group when assigned to the seminar plus supervision condition compared with the manual only condition (Time × Contrast effects were statistically significant for both adherence and skill scores for Role Plays 1 and 2). At baseline, the recovering group reported having significantly higher caseloads, F(1, 73) = 4.9, p = .03, being more familiar with disease model and 12-step approaches, F(1, 73) = 19.6, p = .001, and receiving less than 1 hr of supervision each week, F(1, 73) = 8.4, p = .03, compared with the clinicians who did not identify themselves as having had a substance use problem in the past. However, because education level was significantly related to self-reported recovery status, χ2(1, N = 78) = 12.7, p < .001, educational level may have some influence on these findings.
Changes in Clinicians' Knowledge of CBT
Changes from baseline to posttraining on the CBT Knowledge Test are presented in . There was an overall effect for time, t(74) = 5.1, p < .01, suggesting that scores improved for the group as a whole over time. However, there were no significant Contrast × Time effects (effect sizes for the Web vs. manual and seminar plus supervision contrasts were 0.30 and 0.33, respectively). Those assigned to the seminar plus supervision condition had the largest increase in scores (4.3 points). Participants with a master's degree had significantly greater increases across time than participants who had a bachelor's degree or less (at posttraining, scores were 42.1 for those with a master's degree vs. 36.9 for those with a bachelor's degree). However, there was no significant interaction of Training Condition × Educational Level on knowledge scores. By recovery status, there was an interaction of Training Condition × Recovery Status, in which clinicians who self-identified as recovering made somewhat greater gains if they were assigned to the Web or seminar plus supervision conditions, F(1, 68) = 3.2, p = .08. There were no significant main effects or interactions of years of clinical experience on knowledge scores.
CBT Knowledge Test Scores by Time and Training Condition
When only those clinicians who had been randomized to a training condition were included in the analysis, the posttraining mean scores for the randomized subsample were not appreciably different from those of the full sample (manual only M = 38.5, Web site M = 40.2, seminar plus supervision M = 41.2), and effect sizes (comparing the seminar plus supervision condition with the manual only condition) were comparable (0.33 for the full sample and 0.44 for the randomized subsample). The CBT Knowledge Test and the adherence and skill ratings were moderately correlated. Pearson's r correlations for the pretraining CBT Knowledge Test and the pretraining mean adherence and skill ratings were .38 and .41, respectively (both p < .01). At posttraining, correlations were .31 and .32, respectively (both p < .01).
Satisfaction and Use of CBT Techniques
The clinicians' report of their use of CBT techniques in their clinical work was assessed via self-reports at follow-up and are presented in . Overall, there were few statistically significant between-condition differences, but the general pattern suggested somewhat higher reported use of CBT in their clinical work and greater satisfaction with the CBT manual among the seminar plus supervision and Web conditions compared with the manual only condition. Moreover, ratings by participants assigned to the manual only condition suggested that they perceived more barriers to using CBT in their clinical work (e.g., seeing CBT as too long, too structured, or not compatible with their style) compared with clinicians assigned to the other conditions.
Self-Reported Use of CBT and Perceived Barriers to Using CBT by Training Condition