|Home | About | Journals | Submit | Contact Us | Français|
Data from 399 adolescents, who participated in one of four randomly controlled trials of the A-CRA intervention, were used to examine the extent to which exposure to Adolescent Community Reinforcement Approach (A-CRA) procedures mediated the relationship between treatment retention and outcomes. Although zero-order correlations indicated that retention in treatment was a significant predictor of AOD use (r = -.18, p < .001), this relationship was reduced to non-significance (p = .39) when exposure to A-CRA procedures was included in the model. Overall, the final model evidenced a very good fit (RMSEA=.00; NFI = .99; CFI = 1.00), and explained 29% and 43% of the variance in adolescents’ post-treatment AOD use and AOD-related problems, respectively. Additionally, CHAID analysis was used to derive a target level of A-CRA exposure, which was found to be significantly related to being in recovery at follow-up. The current findings are useful, as little research to date has identified significant mediators of the relationship between retention and treatment outcomes or identified target thresholds of treatment exposure.
Results from several national treatment evaluation studies with adults have concluded there is an important relationship between treatment retention and treatment outcomes (Gossop, Marsden, Stewart, & Kidd, 2003; Hubbard, Craddock, & Anderson, 2003; Simpson, 1979, 1981; Simpson, Joe, & Brown, 1997; Simpson & Sells, 1982). For example, using data from approximately 1,500 clients who participated in the Drug Abuse Reporting Program (DARP), Simpson (1981) found positive 12-month posttreatment outcomes (e.g., decreased illicit drug use, decreased criminal involvement) increased linearly with length of time spent in treatment. These findings were replicated with data from the Drug Abuse Outcome Treatment Outcome Study (DATOS), which included over 10,000 clients from 96 programs in 11 cities (Simpson, Joe, Broome et al., 1997; Simpson, Joe, & Brown, 1997). In addition to supporting the relationship between retention and treatment outcome, these studies identified minimum retention thresholds of approximately 90 days for outpatient and residential care, approximately a year for methadone treatment, and provided support for the inclusion of “sufficient retention” as a key component of Simpson’s (2004) widely cited conceptual framework for drug treatment process and outcome.
Williams and Chang’s (2000) comprehensive review of adolescent substance abuse treatment outcome studies also has provided support for the important relationship between retention and treatment outcome with adolescent populations. For example, using a data set of more than 5,600 adolescents from 30 outpatient programs, Friedman, Glickman, and Morrissey (1986) found adolescents who stayed longer in treatment reported significantly less substance use at discharge. The adolescent DATOS study (DATOS-A) was the first of the three national evaluations to include a separate cohort of adolescents who were treated in treatment programs specifically oriented towards adolescents. Similar to DATOS findings with adults, Hser and colleagues (2001) analyzed data from the nearly 1,200 adolescents who participated in the DATOS-A study and found that adolescents who stayed in treatment for at least 90 days in residential or outpatient drug-free, or 21 days in short-term inpatient, had significantly lower rates of alcohol and drug use.
Despite the general support for retention as a mediator of treatment outcomes, researchers have emphasized the need for further study to identify mechanisms of change that can provide more complete information about why particular treatments work (Kazdin & Nock, 2003). This is especially important given some researchers have argued the relationship between retention and treatment outcome may simply be an artifact of other more important variables like motivation to change or treatment compliance (McLellan et al., 1996). One study provides some support for the hypothesis that other variables besides retention are playing an important role; this is a large randomized study by Dennis and colleagues (2004), which evaluated the effectiveness of five different outpatient treatment interventions for adolescents. Although the average length of stay in treatment varied considerably among these treatments, with the shortest being 41 days for Motivational Enhancement Treatment/Cognitive Behavioral Therapy (MET/CBT5; Sampl & Kadden, 2001) and the longest being 90 days for Family Support Network (FSN; Bunch, Hamilton, Tims, Angelovich, & McDougall, 1998), all five interventions demonstrated significant pre-post treatment improvements in terms of days of abstinence and percent of adolescents in recovery. Importantly, despite these differences in lengths of stay, there were no statistically significant differences among treatment condition outcomes. As noted by Dennis and his colleagues, it is possible that these similarities in outcomes may have been driven more by general treatment factors rather than from differences in the specific components of each respective treatment.
One study that illustrates a methodology for examining the relationship between particular components of an intervention and treatment outcomes was conducted by Pantalon, Chawarski, Falcioni, Pakes, and Schottenfeld (2004). These researchers examined the relationship between the use of different Community Reinforcement Approach (CRA) procedures (e.g., functional analysis of behavior, skills training, homework) and client outcomes, such as retention (defined as number of study weeks completed) and cocaine abstinence (defined as weeks of consecutive abstinence). Specifically, using the Mechanisms of Action Rating Scale (MARS) trained raters counted the number of times a particular CRA procedure was observed during one randomly selected session for each patient. Although results did not reveal any significant relationships between the MARS total score and outcome, specific components did appear to matter. For instance, the assignment of homework was found to have a significant positive relationship with retention, and higher rates of discussing goals of counseling and providing rationales for procedures were found to have a significant positive relationship with increased cocaine abstinence. Consistent with previous research, retention and therapeutic alliance were also found to be related to cocaine abstinence. However, a major limitation was that their study was based on a sample of only 16 cocaine-dependent pregnant or postpartum women; thus, there was inadequate power for evaluating more complex relationships.
Building upon the findings of Pantalon and colleagues, the current study examined the relationship between exposure to Adolescent-Community Reinforcement Approach (A-CRA; Godley et al., 2001) treatment procedures and subsequent alcohol and other drug (AOD) use and AOD-related problems, with exposure being defined as the number of procedures delivered to treatment participants (Dane & Schneider, 1998). Although a key underlying theory for both CRA and A-CRA is the rearranging of environmental contingencies so non-using behavior is more rewarding than using behavior, which helps prevent or reduce AOD use and AOD-related problems, A-CRA is unique in that it was specifically designed to also take into account differences in adolescents’ patterns of use, address life areas that are developmentally appropriate for adolescents, and add procedures for working with parents/caregivers (Godley et al., 2001). An individually-based stand-alone treatment intervention, A-CRA is generally delivered with 10 individual sessions for adolescents, two with caregivers alone, and two with adolescents and caregivers combined; it has been shown to be among the most effective and cost-effective approaches for the treatment of adolescent substance abuse (Dennis et al., 2004). Additionally, A-CRA has been tested and found effective in the context of outpatient continuing care following residential treatment (Garner, Godley, Funk, Dennis, & Godley, 2007; Godley, Godley, Dennis, Funk, & Passetti, 2002, 2006), and without the caregiver components, as an intervention for street-living homeless adolescents with substance abuse problems (Slesnick, Prestopnik, Meyers, & Glassman, 2007). However, no studies to date have examined the relationship between A-CRA treatment exposure and subsequent treatment outcomes, or the extent to which treatment exposure mediates the relationship between retention and outcomes.
Thus, in addition to being among the first studies to examine the relationship between exposure to A-CRA procedures and treatment outcomes, another purpose of this study was to examine the extent to which exposure to A-CRA procedures account for or “mediate” (see Baron & Kenny, 1986) the relationship between treatment retention and subsequent treatment outcomes. Importantly, because A-CRA encourages therapists to continue to work with adolescents and their families for the entire duration of the planned 12-13 week intervention, there is little variation in length of stay when measured by number of days in treatment. Thus, for the analyses reported below, retention is defined as the total number of sessions delivered to adolescents or caregivers as part of their planned treatment intervention. We have hypothesized (see Figure 1) that: (a) A-CRA exposure would mediate the relationship between retention and client AOD use; (b) this reduction in AOD use would be sustained six months after introduction of the intervention; (c) A-CRA exposure would be indirectly related to reductions in AOD-related problems (via AOD use); and (d) this reduction in AOD-related problems would be sustained six months after the introduction of the intervention. A final aim of this study was to empirically derive a target dosage of A-CRA in order to provide guidance for evaluating the adequacy of implementation efforts.
Data for these analyses were drawn from sub-samples of adolescents who participated in one of four different randomly controlled trials, each of which had A-CRA based conditions. In chronological order, these studies included: (a) the completed Cannabis Youth Treatment (CYT) study, which was a randomized clinical trial comparing the effectiveness A-CRA and four other adolescent outpatient interventions (Dennis et al., 2004); (b) the Assertive Continuing Care (ACC-1) study, which was a randomized clinical trial comparing an assertive continuing care (ACC) intervention that included A-CRA procedures with continuing care as usual (see Garner et al., 2007; Godley et al., 2002, 2006); (c) a second Assertive Continuing Care (ACC-2) study (Godley, 2003); and (d) the Adolescent Outpatient and Continuing Care Study (AOCCS) study (Godley, 2004). The ACC-2 is an ongoing four-group randomized experiment that will evaluate the relative effectiveness of two types of continuing care with and without contingency management for 324 adolescents discharged from residential treatment with respect to intermediate outcomes, including continuing care participation, recovery environment, and other pro-social activities and reductions in relapse and substance use during the 12-months after residential discharge. The AOCCS randomized study is also ongoing and has four conditions and will evaluate the effects of two types of outpatient treatment, two types of continuing care, and their interaction for 324 adolescents by examining the frequency of substance use and problems and cost-effectiveness during the year following admission. Participation in all four studies was voluntary and conducted under the auspices of Chestnut Health Systems’ Institutional Review Board. See Figure 2 for an illustration of how the sample for the current study was derived.
Overall, adolescent participants (N=399) were primarily male (68%), Caucasian (65%), and between 15 and 16 years of age (55%). Additionally, most adolescents were in school (83%), and involved with the criminal justice system (77%). With regard to AOD use, 72% reported weekly AOD use at intake, with nearly all participants self-reporting DSM IV criteria for abuse (30%) or dependence (64%) in the year prior to intake. As illustrated in Table 1, there were significant baseline differences between the adolescents included in these four studies, which is not surprising considering these samples included adolescents admitted to outpatient and residential treatment settings. For the purposes of these analyses, these group differences are not problematic and should help enhance the robustness and generalizability of the current findings.
Overall, 15 therapists delivered the A-CRA intervention to adolescents during these four studies. Eleven of the therapists were female and four were male. Most (60%) had bachelor degrees, while the remainder (40%) had master’s degrees. All therapists were recruited to work in a specific study and trained, monitored, and certified as described below.
As noted previously in the introduction, the underlying theory of A-CRA is that rearranging environmental contingencies so that non-using behavior is more rewarding than using behavior will prevent or reduce AOD use. Additionally, a therapist manual was developed as part of the CYT study and outlines the intervention as adapted for adolescents (Godley et al., 2001). This manual was used to train and supervise A-CRA therapists in all four studies from which the sample was drawn. Sessions were held weekly in the office or in the home (when the adolescent was in continuing care) for a fixed length of time (12 to 14 weeks). In terms of sessions, ten of the sessions are conducted individually with adolescents, two sessions are conducted with parents or other caregivers only, and two sessions are with the adolescents and their caregivers combined. The intervention can best be described as ‘procedure-based’, meaning the therapist decides which procedures are appropriate to use within a session based upon the material the adolescent discusses during a session. Although the goal is to reduce substance use, the therapist demonstrates his/her interest in all areas of the adolescent’s life, knowing that improvement in different life areas can reduce the potential for substance use or relapse. For example, when the adolescent completes the “Happiness Scale”, it may become apparent that they are having problems with a teacher at school; the therapist may teach the adolescent good communication skills and encourage the adolescent to use these communication skills as homework with the teacher before the next session.
Included among the intervention procedures and case-management activities are: (a) functional analysis of substance using/prosocial behavior; (b) an adolescent self-assessment of happiness; (c) goals of counseling; (d) pro-social recreation planning; (e) relapse prevention skills; (f) communication skills; (g) problem solving skills; (h) family relationship skills (i.e., problem solving, communication, and relationship happiness); (i) homework assignments; (j) review of homework; (k) job seeking skills; (l) regular case-management; (m) other case-management; (n) crisis management; and (o) urine testing (see Godley et al., 2001 for detailed descriptions). A-CRA was the primary therapeutic intervention with all the adolescents in this sample. However, as is common for participants in applied research, adolescents might have been referred for other services as needed. For example, an adolescent might be referred for a psychiatric assessment or to another service provider for continuing care, but receipt of the A-CRA intervention was under the control of random assignment.
Therapists delivering A-CRA were trained in three- to four-day training sessions, conducted by the same training team. These training events included a didactic overview of intervention procedures, demonstrations of procedures, and opportunities for the therapists to practice the procedures during the training with expert feedback. Additionally, therapists were provided weekly clinical supervision, which included specific feedback on their delivery of the intervention based on session tape reviews and consultation regarding clinical issues with adolescents on their caseloads.
Several steps were taken to ensure accurate documentation of treatment. For instance, in order to monitor the degree to which each of the A-CRA procedures implemented with adolescents and their caregivers were accurately recorded, therapists were expected to document procedure codes used during a session on a service contact log immediately after each session. These procedure codes, which are then entered into an electronic database, were used to produce summary reports of procedures provided to each adolescent and by each therapist. Supervisors monitored the reliability of therapists’ procedure reports by listening to a random set of session tapes and independently coding which procedures they heard the therapist provide. Agreement rates were computed between the therapist’s report and the supervisor’s independent assessment of what was delivered during the session and reported and reviewed in monthly clinical management reports.
Retention was operationally defined as the total number of A-CRA sessions that were delivered to adolescents or caregivers as part of their planned treatment intervention. A-CRA sessions were defined as therapeutic interactions that lasted for at least 15 minutes and in which at least one A-CRA procedure was delivered.
Considered a model dependent fidelity measure (Orwin, 2000), the AES (coefficient alpha = .85) is a count of the 15 A-CRA procedures and case-management activities listed in Table 2 and which are outlined in the A-CRA and case-management manuals (Godley, Godley, Karvinen, Slown, & Wright, 2006; Godley et al., 2001). Although some procedures (e.g., happiness scale, homework, urine testing) may be repeated multiple times, they are scored dichotomously as either completed or not completed. Thus, the AES score ranges from 0 - 15. After therapists were trained in A-CRA procedures and began to conduct sessions with adolescents, they received training on the A-CRA Procedure Code Manual, which lists codes and describes operational definitions for each A-CRA procedure.
Client baseline (i.e., intake), during-treatment (i.e., three-month period during which A-CRA procedures were provided), and follow-up data (i.e., the subsequent three-month period following A-CRA treatment) were collected as part of a semi-structured interview using the Global Appraisal of Individual Needs (GAIN; Dennis, Titus, White, Unsicker, & Hodgkins, 2003), which is a comprehensive biopsychosocial assessment designed to integrate research and clinical assessment into one structured interview. Currently, over 700 treatment programs use the GAIN (M.L. Dennis, personal communication, May 3, 2008), making it one of the most widely used measures in adolescent treatment. The GAIN’s main scales have been shown to demonstrate good internal consistency (alpha over .90 on main scales, .70 on subscales), test-retest reliability (Rho over .70 on days/problem counts, Kappa over .60 on categorical measures), and to be highly correlated with measures of use based on timeline follow-back methods, urine tests, collateral reports, treatment records, and blind psychiatric diagnosis (Rho of .70 or more, Kappa of .60 or more; Dennis, Chan, & Funk, 2006; Dennis, Ives, White, & Muck, 2008; Dennis et al., 2003). Copies of the instruments, manual, publications and documentation on the 100 plus scales in the GAIN are available at www.chestnut.org/li/gain. Below are brief descriptions of the GAIN measures used in the current study and which were collected at 90-day intervals.
The measure of AOD use was the actual number of days during the past 90 days that adolescents reported using alcohol or any other drugs (other than ones used per physician’s orders). Days of AOD use was selected as the primary dependent measure because (a) it is very easy to understand and thus has clinical appeal; (b) it is widely used by the field; and (c) empirical research has shown it to be one of the most useful single item indicators of overall substance use (Lennox, Dennis, Ives, & White, 2006).
The Substance Problem Scale (SPS; coefficient alpha = .89 with current sample) was used to assess problems related to AOD use or AOD-using behavior. It is based on recency ratings (i.e., past month, 2-12 months ago, more than 12 months ago, never) on 16 symptoms: seven corresponding to DSM-IV criteria for dependence, four for abuse, two for AOD-induced health and psychological problems, and three on lower severity symptoms of use (hiding use, people complaining about use, weekly use). The past month SPS symptom count has been shown to have good test-retest reliability (r =.70; Dennis, Babor, Roebuck, & Donaldson, 2002). A past-year version of this measure was used to help control for differences in baseline problem severity.
Other measures collected as part of the GAIN and included in the current study were Days of Residential Treatment, which is a sum of the number of days spent in residential treatment immediately prior to receiving A-CRA, and Days of Outpatient Treatment, which is a sum of the number of days spent in outpatient treatment immediately prior to receiving A-CRA. These measures were included primarily to help control for differences associated with treatment other than A-CRA.
Amos structural equation modeling software with the full information maximum likelihood estimation method was used to test each of the study hypotheses outlined above. After examination of our initial hypothesized model, we used an iterative procedure where paths with an alpha greater than or equal to .05 were deleted. At each step, the model fit was evaluated using several fit indices, including: the minimum fit χ2, root mean square error of approximation (RMSEA), and the comparative fit index (CFI). As noted by Browne and Cudeck (1993), RMSEA values of less than 0.05 indicate a close fit in relation to the degrees of freedom, values of 0.08 or less indicate a fit with reasonable errors of approximation in the population, and values greater than 0.10 are a poor fit. The CFI ranges from 0 to 1, with values greater than 0.95 indicating very good fit (Bentler, 1990; Hu & Bentler, 1999).
Adolescents in the current study were nested within therapists; thus, it is likely that adolescents from the same therapist share more similarities with one another than they do with adolescents treated by other therapists. Because ignoring such inter-correlations may lead to violations of the ordinary least squares (OLS) regression assumption of independent error terms and may result in misestimated standard errors, we examined the extent to which differences between therapists existed with regard to retention and AES using Analysis of Variance (ANOVA) tests. Differences in retention and AES attributed to therapists were removed by group-mean centering these measures (i.e., converting each score into deviations from the therapist mean score).
In order to empirically derive an optimally effective threshold AES score associated with significantly improved treatment outcomes, we conducted a Chi Squared Automatic Interaction Detection (CHAID) analysis, which is a non-parametric analysis based on statistical recursive partitioning algorithms (Biggs, De Ville, & Suen, 1991). CHAID performs segmentation modeling by using tests based on χ2 statistics to divide cases into groups and subgroups that are significantly different in terms of a criterion variable. Each time CHAID forms (sub) groups, the categories (i.e., values) of the predictor variable being used for the segmentation are combined so as to minimize the within-group variance and maximize the between-group variance. This maximizes the separation among the groups. In the current CHAID analysis, the predictor was AES score and the outcome was post-treatment “in recovery” status. Consistent with the Cannabis Youth Treatment (CYT) study definition (Dennis et al., 2004), “in recovery” was created using adolescent self-reports data collected during the GAIN follow-up assessments and was defined as no past month AOD use, no substance-related problems, and currently living in the community (i.e., not incarcerated or in residential treatment).
Table 2 summarizes several measures of treatment exposure by study and overall. Displayed in this table are the means and standard deviations for the number of times each A-CRA procedure and case-management activity was reported delivered, as well as the length of stay in A-CRA treatment, number of A-CRA sessions, number of A-CRA procedures delivered per session, and AES scores. As indicated in the table, there was an overall significant difference by study for the frequency with which each of the A-CRA procedures and case-management activities was delivered in each study, with the exception of Homework Review. However, there were no significant differences by study with regard to length of stay in A-CRA treatment, the number of A-CRA procedures delivered per session, or the average AES score.
As these data were based primarily upon therapist self-report, Cohen’s (1960) kappa statistic was used to evaluate agreement between therapists’ and supervisors’ report of A-CRA procedures delivered during session. As shown in Table 2, ratings between 89 different randomly selected sessions (approximately 3% of all sessions) indicated that there was an overall high degree of agreement between therapist and model expert reports (average kappa = .86; ranging from .59 to 1.00).
Results of the ANOVA analyses revealed significant variation between therapists with regard to retention, F(14, 384) = 2.28, p < .01, as well as AES, F(14,384) = 3.77, p < .001. Thus, as described above in the analytic plan section, these two measures were group-mean centered in order to remove the systematic differences attributable to therapists (i.e., therapist effects).
Means and standard deviations for each of the variables, as well as the zero-order correlations among each of the measures are shown in Table 3. Fit indices for the hypothesized model indicated a good model fit (χ2(30) = 60.55; RMSEA = .05 [90% CI = .03 to .07]; CFI = .97), as well as supported the majority of the hypothesized paths. For example, results provided support for the primary hypothesis that A-CRA procedure exposure (i.e., AES) mediates the relationship between retention (i.e., number of sessions) and outcome (i.e., AOD use). That is, consistent with the basic guidelines for establishing mediation (see Baron & Kenny, 1986), zero-order correlations indicated significant relationships between retention and AES (r = .81; i.e., independent variable and mediator), retention and AOD use (r = -.18; i.e., independent variable and dependent variable), and AES and AOD use (r = -.19; i.e., mediator and dependent variable). However, the relationship between retention and AOD use was reduced to non-significance (p = .39) when exposure to A-CRA procedures (i.e., AES) was included as a predictor of AOD use, which is indicative of a mediator (see Baron & Kenny, 1986).
Supporting our second hypothesis, this reduction in substance use was shown via the stability path to be sustained out to subsequent follow-up wave (β = .52). Supporting our third and fourth hypotheses, the model also indicated that AES indirectly reduced AOD-related problems (via AOD use; β = .54) and that this reduction also was sustained out to the subsequent follow-up period (β = .11).
In contrast to our initial predictions, none of the four baseline variables (AOD use, AOD-related problems, days of residential or outpatient treatment) hypothesized to predict retention were found to be statistically significant. As these non-significant paths suggest that a more constrained model would be a more concise and efficient representation of the observed data (Anderson & Gerbing, 1988), exploratory post hoc model fitting was conducted following the suggestions of Byrne (2001).
The final model (see Figure 3) resulted in a significantly better fit (p < .001) than the hypothesized model (χ2(23) = 16.64; p = .83; RMSEA=.00 (90% CI = 0 to .03); CFI = 1.00). Differences between these two models included the removal of the days in outpatient treatment variable (which was not found to predict outcome variables in the model), as well as the inclusion of four new paths that were identified as part of the exploratory post hoc model fitting described above. These new paths included: (a) a path between baseline AOD-related problems and during treatment AOD use (β = 12); (b) a path between days of residential treatment and during treatment AOD use (β = -.21); (c) a path between during treatment AOD-related problems and follow-up AOD use (β = .18); and (d) a path between during treatment AOD use and follow-up AOD-related problems (β = -.14). While AOD use and AOD-related problems during and after treatment have significant positive zero-order correlations (r =.18 to .64), there was evidence of a suppressor interaction effect (Cohen, Cohen, West, & Aiken, 2003). Specifically, people whose AOD use decreased from treatment to follow-up had lower AOD problems at follow-up than might be expected, based solely on their during-treatment AOD problems and AOD use at follow-up. The justification for keeping this path is that omission can lead to underestimates of the effects (see Cohen et al., 2003, pg. 458) and is consistent with the effect found in similar longitudinal path analysis (see Garner et al., 2007).
Given the AES accounted for much of the relationship between retention and AOD use, we next used Chi Squared Automatic Interaction Detection (CHAID) analysis to empirically establish whether there was one or more AES cut off points associated with higher rates of being in recovery (no past month AOD use or problems and currently living in the community) at follow-up. Results of the CHAID analysis indicated adolescents who received an AES score of 12 or more were significantly more likely to be in recovery at follow-up (55% vs. 35%; χ2(1) = 14.13, p < .001). It is noteworthy that only about 32% of study participants received this level of A-CRA exposure and that the percent achieving it varied both by study (24 to 45%) and by therapist (11% to 71%).
The primary aim of this study was to examine the extent to which exposure to A-CRA treatment procedures mediated the relationship between treatment retention and adolescent substance abuse treatment outcomes measured at three- and six-months post-intake. Consistent with conventional methods for establishing mediation (Baron & Kenny, 1986), we first established the presence of a significant relationship between retention and our hypothesized mediator (i.e., AES). Results indicated not only were A-CRA procedures being delivered during adolescent and caregiver sessions, but exposure to A-CRA procedures was greater among adolescents who attended more sessions. This indicates therapists were not simply doing one or two procedures over and over, but rather were delivering a broad range of A-CRA procedures as prescribed in the manual.
Second, we confirmed the expectation that retention would be inversely related to during treatment AOD use, which is consistent with both the literature reviewed previously, as well with the Simpson (2004) treatment process model. It is interesting to note that the magnitude of the relationship between retention and AOD use found in the current study (r = -.18) was identical to the magnitude found between retention and consecutive weeks of cocaine abstinence in the Pantalon and colleagues (2004) CRA process and outcome study. Given the preponderance of evidence supporting the relationship between retention and outcome, this finding is primarily noteworthy to the extent that it fulfills a necessary requirement in establishing mediation.
Third, and most importantly, in meeting Baron and Kenny’s (1986) final criteria for mediation, the results of this study indicated that the relationship between retention and AOD use was significantly reduced when AES was controlled. Although considerable variance remained to be explained in AOD use, current findings do suggest that exposure to treatment procedures may be an important mediator within the “black box” of treatment, and is certainly worthy of future attention by researchers. Indeed, according to the final model presented in this paper, AES explained as much of the variance in during-treatment AOD use (β = -.19) as did adolescents’ baseline measure of AOD use (β = .19).
In contrast to previous research (Battjes, Gordon, O’Grady, & Kinlock, 2004; Stark, 1992), the current models did not support significant relationships between baseline AOD use, AOD-related problems, and prior treatment with A-CRA retention. Although examinations of zero-order correlations confirmed the non-significant relationships found between substance use severity and days of outpatient treatment with retention, a significant negative relationship between days of prior residential treatment and retention in A-CRA was revealed. One explanation for why the latter relationship did not remain significant in the models tested is that the variance in retention explained by days in residential treatment was removed when the variance in retention attributable to therapists was removed. Although not significantly related to retention in A-CRA treatment, days in residential treatment was found to be a significant predictor of during-treatment AOD use. This finding is important as it suggests exposure to A-CRA procedures significantly reduced AOD use, even when taking into account the significant reductions explained by prior residential treatment.
The secondary aim of this study was to identify a threshold level of exposure to A-CRA procedures. Based upon the current data, adolescents who were exposed to 12 or more A-CRA procedures were significantly more likely to be in recovery at follow-up (55% vs. 35%). Consequently, an AES score of 12 or greater may be a clinically meaningful target for therapists to achieve. Notably, other factors such as prior AOD use or related problems may have an impact on the cutoff score and thus it is important that further research examining how the threshold level varies by different subgroups (e.g., high AOD use vs. low AOD use). Nevertheless, as only 32% of the current participants (24 to 45% in the four clinical studies) received this threshold, future research focusing on increasing the proportion of clients receiving this target or threshold treatment exposure is needed. There is hope for improving these rates given the large variation by therapist (11 to 71%). One method recommended by the Institute of Medicine (IOM; Institute of Medicine, 2007) to improve the quality of healthcare is pay-for-performance (P4P). Shepard and colleagues (2006) showed that providing financial incentives to therapists could be an effective and cost-effective approach to improve client retention. Although these practices have become increasingly common in healthcare, few studies have examined the impact of P4P methods experimentally (Dudley, 2005).
It is important to acknowledge the limitations of the current study. First, data were primarily based upon self-report by therapists. However, as previously noted, there was generally high agreement (average kappa = .86) between procedures reported by therapists and those reported by model experts. Secondly, across all studies, outcome data was only available for six months after the initiation of the intervention, which limited the ability to examine relationships with longer-term outcomes. In the future, it would be useful to extend the length of follow-up.
There are a number of clinical implications based on the results of the current study. First, the AES is a relatively simple to use and reliable measure of A-CRA treatment fidelity. Although the importance of using fidelity measures is not new, its importance continues to grow as the substance abuse treatment field increasingly moves towards use of evidence-based practices (EBPs). Indeed, the Center for Substance Abuse Treatment and numerous state and local funding authorities are using a variety of incentives and requirements to increase provider usage of EBPs (Schmidt, Rieckmann, McCarty, Svikis, & Roman, 2007).
Second, based upon the findings of the current study, there appears to be a significant relationship between treatment fidelity, as measured by exposure, and reductions in both AOD use and related problems. These findings are important as a review of the adult individual psychotherapy literature concluded that research to date has been unable to draw strong links between treatment fidelity and client outcomes (Miller & Binder, 2002). Although the current findings are consistent with similar CRA process research (Pantalon et al., 2004), future studies examining the contribution of other dimensions of CRA or A-CRA treatment fidelity, such as adherence and competence, are still needed.
Finally, results of the current study suggest that exposure to treatment procedures may be a mediator in the treatment process, which is important for two reasons. First, such a finding is important as few studies have examined mediators of the relationship between retention and outcome, especially considering the number of studies that have examined factors associated with treatment retention (e.g., Battjes et al., 2004; Condelli & De Leon, 1993; Grella, Joshi, & Hser, 2004; Grella, Wugalter, & Anglin, 1997; Hser et al., 2001; Joe, Simpson, & Broome, 1999; Lang & Belenko, 2000; Simpson, Joe, Rowan-Szal, & Greener, 1997; Williams & Chang, 2000). Second, this finding provides support for “specific ingredients” of treatment impacting treatment outcome, which is important given some researchers (Messer & Wampold, 2002) have argued that compared to the level of support for the importance of “common factors”, there is much less support for effect of specific treatment ingredients.
As noted by McLellan, Chalk, and Bartlett (2007), measures of treatment retention are increasingly becoming accepted as indicators of treatment quality even though they are not direct measures of treatment outcome. For instance, the Washington Circle (WC), a group of multi-disciplinary researchers, providers, managed care representatives, and public policymakers convened by the Center for Substance Abuse Treatment (McCorry, Garnick, Bartlett, Cotter, & Chalk, 2000), defined three key treatment performance measures: (a) identification; (b) initiation; and (c) engagement (see Garnick et al., 2002 for more details). Engagement is essentially an “early retention” indicator and defined as receiving two additional AOD services within 30 days of the initiation of care. The impact of these performance measures on substance use outcomes has not been examined; however, research has indicated that engagement was significantly related to decreased criminal activity (Garnick et al., 2007). Similarly, the Network for the Improvement of Addiction Treatment (NIATx), which is a partnership between the Robert Wood Johnson Foundation, the Center for Substance Abuse Treatment, the National Institute on Drug Abuse, and a number of independent addiction treatment organizations, teaches addiction treatment providers to use process improvement strategies to enhance access and retention towards the goal of improving outcomes. Although early results suggest improvements in retention can be obtained, it has been noted that these improvements in retention can be difficult to achieve (McCarty et al., 2007). Given the large body of evidence supporting the association between retention and outcomes, retention remains not only an important mediator in the treatment process, but also serves as a useful target for improvement by both treatment providers and researchers. Nevertheless, as has been aptly noted by Kraemer, Wilson, Fairburn, and Agras (2002), “not all mediators are mechanisms of change, however, all mechanisms of change are mediators.” Thus, it is important for researchers to continue identifying potential mechanisms of change in order to elucidate why treatment works and to provide needed knowledge to help develop more effective treatments.
This work was supported by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA; TI13356), National Institute on Alcohol and Alcoholism Abuse (NIAAA grant AA10368) and the National Institute on Drug Abuse (NIDA grant DA018183). The authors wish to thank all of the individuals who assisted in these projects. The authors also wish to acknowledge Stephanie Merkle, Christopher Roberts, and Kelli Wright for their assistance in preparing this manuscript. The opinions are those of the authors and do not represent official positions of the government.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Bryan R. Garner, Chestnut Health Systems, 720 W. Chestnut, Bloomington, IL 61701, phone: 309-827-6026, fax: 309-829-4661; email: gro.tuntsehc@renragrb.
Susan H. Godley, Chestnut Health Systems, 720 W. Chestnut, Bloomington, IL 61701.
Rodney R. Funk, Chestnut Health Systems, 720 W. Chestnut, Bloomington, IL 61701.
Michael L. Dennis, Chestnut Health Systems, 720 W. Chestnut, Bloomington, IL 61701.
Jane Ellen Smith, University of New Mexico, Logan Hall, MSC03 2220, Albuquerque, NM 87131.
Mark D. Godley, Chestnut Health Systems, 720 W. Chestnut, Bloomington, IL 61701.