This study found that treatment adherence predicted treatment outcome in manualized behavioral interventions for substance abuse and related behavior problems in urban adolescents. Treatment adherence was linked to improvement in multiple outcomes up to 6 months after discharge. Adherence promoted therapeutic change across two different outpatient approaches: individual cognitive–behavioral therapy and multidimensional family therapy. In CBT, greater levels of adherence predicted greater declines in marijuana use. In both CBT and MDFT, stronger adherence predicted greater reductions in parent reports of externalizing behaviors. Also in both conditions, intermediate levels of adherence predicted the largest declines in parent reports of internalizing behaviors, with high and low adherence predicting smaller improvements—a curvilinear (or quadratic) effect on internalizing. Adherence–outcome effects were small-to-medium in size. Contrary to hypotheses, therapist competence was not related to any outcome in either condition, nor did it moderate the impact of adherence on outcome.
These findings support the contention that treatment adherence plays an important role in the success of empirically based behavioral interventions for adolescent mental health problems. Adherence–outcome studies have generated inconsistent results over the past 2 decades, with some studies reporting favorable adherence effects, others no effects, and still others iatrogenic effects. Several explanations for negative results have been offered, including measurement insensitivity, low mean adherence levels and restricted ranges, lack of differentiation from other process variables such as therapeutic alliance, and the inability of adherence measures to account for flexible deviations from treatment protocols that prove beneficial for some sessions or clients (Miller & Binder, 2002
; Perepletchikova & Kazdin, 2005
). The current study was well positioned to detect adherence effects by (a) examining two treatment models with track records of adherence, differentiation, and treatment technique–outcome links (Hogue et al., 1998
); (b) measuring fidelity at the level of molar treatment modules that are broadly applicable across sessions and clients; and (c) utilizing an instrument with adequate reliability and construct validity. Also, previous research with substance using and delinquent adolescents has documented similar positive adherence effects (Huey et al., 2000
; Schoenwald, Sheidow, Letourneau, & Liao, 2003
), raising the possibility that adherence may be particularly salient to manualized treatments for youths with problem behaviors.
This study is one of the first to replicate the innovative work of Barber et al. (2006)
with regard to curvilinear adherence effects. Barber et al. (2006)
suggested that inconsistent findings for linear adherence effects—does greater adherence predict better out-come?—might be due to the existence of underlying curvilinear adherence–outcome relations. Our results support this conjecture. There were contradictory findings for linear effects on internalizing symptoms: Stronger adherence predicted decreased symptoms in MDFT but increased symptoms in CBT. However, when higher order, curvilinear effects were examined, the linear relations faded to non-significance and a quadratic relation emerged across conditions: Intermediate adherence was associated with the greatest improvement, while weaker and stronger levels produced less improvement. Like Barber et al. (2006)
, we interpret curvilinear effects to be a caution against being too lax or too strict in adhering to treatment protocols.
Unlike Barber et al. (2006)
, who found linear adherence–outcome relations for a single outcome only, we found linear effects for marijuana drug use and externalizing symptoms in addition to curvilinear effects for internalizing symptoms. Based on these results, it appears that strong adherence does not unilaterally signify inflexible model implementation that sabotages treatment strength. This begs the question: Under what conditions does (over)extensive use of prescribed therapy processes diminish treatment gains, such that treatment adherence should be tempered by other considerations? For the current study, at least one mechanism seems plausible. A sizable portion of adolescent drug users has co-occurring anxiety and mood disorders (see Rowe, Liddle, Greenbaum, & Henderson, 2004
). For this subgroup, the featured elements of manualized treatments designed to target drug use and externalizing behaviors specifically may need to be moderated in favor of auxiliary interventions that directly target internalizing problems: changing unrealistic negative thoughts, interpersonal problem-solving skills, relaxation training, and so forth (Compton, Burns, Egger, & Robertson, 2002
; Weisz, McCarty, & Valeri, 2006
). Further research is required to verify the prevalence of curvilinear adherence effects for internalizing problems in teen drug-using populations and, if confirmed, to illuminate the mechanisms of action.
It was surprising that therapist competence bore no relation to treatment outcome, nor did it influence the relation between adherence and outcome. If evidence of weak or null competence effects continues to accumulate in equal measure with evidence of small positive effects in randomized trials (Barber, Sharpless, Klostermann, & McCarthy, 2007
), this will challenge clinical researchers to prove rather than presume that greater competence begets better outcome. The newly intriguing question “Does therapist competence matter?” has at least three affirmative responses: (a) Yes, if it can be correctly measured, which is to say, account for contextual variables such as intervention timing and appropriateness, responsiveness to clients, and adaptability across sessions and cases (Elkin, 1999
; Miller & Binder 2002
; Waltz et al., 1993
). The current study took significant steps toward contextual assessment of therapist competence by utilizing expert judges, sampling multiple consecutive sessions in early and later treatment, and incorporating aspects of therapist skill and responsiveness into the competence coding system (Barber et al., 2007
). The derived competence scores for CBT and MDFT had adequate distributions but only fair-to-weak interrater reliabilities. (b) Yes, but only up to a point. Therapists need to meet an acceptable standard of competent model delivery akin to a “red line” benchmark (Shaw & Dobson, 1988
), but beyond that, scaling to greater heights of observed competence may not translate into greater clinical success. Such hypotheses about how competence relates to outcome—the shape of the competence–outcome curve—can now be readily examined with random regression and growth curve modeling techniques. (c) Yes, but primarily so in routine clinical settings with front-line providers exhibiting a wide range of therapy skills. To date, competence research has been conducted almost exclusively in controlled conditions with research-trained therapists, which narrows the band of fidelity scores and potentially mutes fidelity–outcome relations (Dobson & Singer, 2005
This study has several strengths that instill confidence in the reliability and generalizability of findings. Participants included an ethnically diverse group of adolescents and families from a large urban area. Parallel fidelity measures were used for both CBT and MDFT, which permitted us to combine all participants into a single analysis to increase power and generalizability (Elkin, 1999
). Interrater reliability was robust for the adherence items in both scales even though ratings covered molar-level therapy modules rather than discrete treatment techniques (e.g., Barber, Liese, & Abrams, 2003
; Morgenstern et al., 2001
). Fidelity was measured across multiple sessions for each case, and adherence and competence impacts were analyzed after controlling for therapeutic alliance, which reduces “third variable” confounds in the form of non-specific processes and therapeutic relationship factors (Perepletchikova & Kazdin, 2005
). However, there was not enough power to examine fidelity–alliance interactions, which have been found in several other studies (e.g., Barber et al., 2006
One significant limitation of the study was the low interrater reliability of competence scales. Reliability for individual TBRS–C items was generally weak and well below the magnitude found for competence items on most discrete techniques scales (e.g., Barber et al., 2003
). Reliabilities of the averaged competence ratings (.56 for CBT, .55 for MDFT) were modest but in keeping with the magnitude of competence ratings in some studies (e.g., Barber & Crits-Christoph, 1996
; James, Blackburn, Milne, & Reichfelt, 2001
), though decidedly lower than in others (e.g., Carroll et al., 2000; Moyers, Martin, Manuel, Hendrickson, & Miller, 2005
). Barber et al. (2007)
noted that interrater reliability estimates derived from competence measures used in controlled trials tend to be low and list several possible explanations, including differences in how much attention judges pay to different aspects of treatment delivery, difficulties in operationalizing competence, and the use of uniformly competent therapists in randomized studies (which dampens the total variance in competency scores). For the current study, null findings for competence effects must be considered tentative in light of the subpar ICCs obtained.
Note also that interrater reliabilities in this study were estimated by using the one-way analysis of variance model, ICC(1,2; Shrout & Fleiss, 1979
). This model is a conservative approach when used in sampling designs with missing judge data, that is, when every judge does not rate every target or when a balanced incomplete block design (Fleiss, 1981
) is not achieved. Although newer methods for calculating ICCs are available that account for missing data via maximum likelihood estimation (Konishi & Shimizu, 1994
), they are rarely used in observational coding studies. Such methods yield more precise ICCs under conditions of missing data because they accurately estimate variance components for both target and judge, and they can accommodate nested sampling designs (e.g., sessions nested within therapist and treatment condition; see Barber, Foltz, Crits-Christoph, & Chittams, 2004
Another study limitation is that a full examination of treatment dropout effects on fidelity–outcome relations (see Hedeker & Gibbons, 1997
) was beyond the scope of this study. Also, by utilizing case-level fidelity scores that were averaged across individual sessions, we precluded the possibility of examining change in fidelity during the course of treatment. Improvement in fidelity across sessions and cases is thought to be evidence of a “learning curve” in therapist training studies (Crits-Christoph et al., 1998
), and these trends may also meaningfully impact client outcomes in routine practice. Finally, the use of observational coding methods, while adding the rigor of more objective assessment, presents limitations as well. Judges who do not observe (most) every session are not able to track the clinical progress of the case across treatment, which hampers their ability to provide fully informed, case-specific assessments of competence (Barber et al., 2007
; Waltz et al. 1993
). An alternative strategy is to collect therapist-report and/or supervisor-report fidelity data on most or all sessions, pending further verification that fidelity instruments are reliable and valid when used as self-report tools by front-line supervisors (adherence and competence) as well as therapists and clients (adherence only; Carroll, Nich, & Rounsaville, 1998
; Henggeler et al., 1997
; Schoenwald et al., 2004).
As efforts to move research-based treatments into everyday clinical settings gather steam, treatment fidelity concerns will remain at the forefront of dissemination research. The multifaceted relation between treatment adherence and client outcome found in this study awaits replication and further definition in the lab and in the field with adolescent substance-using samples and with additional age groups, ethnic groups, mental health disorders, and treatment models. Based on current findings, future research on adherence–outcome links should routinely explore both linear and curvilinear effects. This will help mark the path for clinicians treading the fine line between protocol adherence and flexible deviation to serve the needs of individual clients and clinical subgroups.