Two Aspects of the Therapeutic Alliance: Differential Relations with Depressive Symptom Change

doi:10.1037/a0023252

J Consult Clin Psychol. Author manuscript; available in PMC 2012 June 1.

Published in final edited form as:

J Consult Clin Psychol. 2011 June; 79(3): 279–283.

doi: 10.1037/a0023252

PMCID: PMC3124012

NIHMSID: NIHMS274457

Two Aspects of the Therapeutic Alliance: Differential Relations with Depressive Symptom Change

Christian A. Webb, Robert J. DeRubeis, Jay D. Amsterdam, Richard C. Shelton, Steven D. Hollon, and Sona Dimidjian

Christian A. Webb, Department of Psychology, University of Pennsylvania;

Contributor Information.

Correspondence concerning this article should be addressed to Christian A. Webb, Department of Psychology, University of Pennsylvania, 3815 Walnut Street, Philadelphia, Pennsylvania 19104-1696. Email: webb/at/sas.upenn.edu

This article is based in part on Christian A. Webb’s doctoral dissertation, which was supported by a Social Sciences and Humanities Research Council of Canada doctoral fellowship. The CPT-II trial (DeRubeis et al., 2005) was supported by grants MH50129 (R10) (Dr. DeRubeis) and MH55875 (R10) and MH01697 (K02) (Dr. Hollon) from the National Institute of Mental Health (NIMH), Bethesda, Md. The UW trial (Dimidjian et al., 2006) was supported by NIMH grant MH55502 (R01) first to Neil S. Jacobson and, after his death, to David L. Dunner. GlaxoSmithKline provided medications and pill placebos for both trials.

The publisher's final edited version of this article is available at J Consult Clin Psychol

Publisher's Disclaimer

Abstract

Objective:

The therapeutic alliance has been linked to symptom change in numerous investigations. Although the alliance is commonly conceptualized as a multidimensional construct, few studies have examined its components separately. The current study explores which components of the alliance are most highly associated with depressive symptom change in cognitive therapy (CT).

Method:

Data were drawn from two published randomized, controlled clinical trials of CT for major depressive disorder (n = 105, M age (years) = 40, female = 62%, white = 82%). We examined the relations of two factor-analytically derived components of the Working Alliance Inventory (WAI) with symptom change on the Beck Depression Inventory (BDI-II) that occurred either prior to, or subsequent to, the examined sessions. WAI ratings were obtained at an early and a late session for each therapist-patient dyad.

Results:

Variation in symptom change subsequent to the early session was significantly related to the WAI factor that assesses therapist-patient agreement on the goals and tasks of therapy, but not to a factor assessing the affective bond between therapist and patient. In contrast, both factors, when assessed in a late session, were significantly predicted by prior symptom change.

Conclusions:

These findings may reflect the importance, in CT, of therapist-patient agreement on the goals and tasks of therapy. In contrast, the bond between therapist and patient may be more of a consequence, than a cause, of symptom change in CT. The implication of these results and directions for future research are discussed.

Keywords: alliance, cognitive therapy, depression

The therapeutic alliance has been examined in relation to treatment outcome across a variety of treatment modalities and mental health problems. In meta-analytic reviews, stronger alliances have been found to be associated with better outcomes (Horvath & Symonds, 1991; Martin, Garske, & Davis, 2000). Several studies (e.g., Adler, 1988; Hatcher & Gillaspy, 2006; Horvath and Greenberg, 1989; Paivio & Bahr, 1998) have reported on the relation between outcome and the constituent subscales of the Working Alliance Inventory (WAI; Horvath & Greenberg, 1986, 1989), the most frequently used alliance measure. The Task subscale has tended to exhibit a stronger association with outcome than have the Goal or Bond subscales (see Strunk, Brotman, & DeRubeis, 2010 for an exception). Therapist-patient agreement on the tasks of therapy, therefore, may play an especially important role in alliance-outcome associations.

These findings derive from studies that have employed theory-derived subscales of the alliance (i.e., the Bond, Goal and Task subscales). Findings from factor analytic studies suggest that this may not be the most appropriate way to partition the WAI. For example, Andrusyna, Tang, DeRubeis, and Luborsky (2001) conducted an exploratory factor analysis of the WAI and obtained findings similar to those reported in an earlier factor analysis (Hatcher & Barends, 1996). Specifically, in the study by Andrusyna et al., a two-factor structure of the measure emerged, where the first factor (Agreement) consisted of all of the items from the Task and Goal subscales, and one item from the Bond subscale. The second factor (Relationship) consisted of the remaining items from the Bond subscale. This factor structure is not surprising given the relatively high correlations often reported between the Task and Goal subscales of the WAI (e.g., Brusseri, & Tyler, 2003; Hatcher & Gillaspy, 2006; Horvath & Greenberg, 1989).

Studies of alliance components and their relation to outcome have tended to rely on patient- or therapist-rated versions of the WAI. Patient ratings would seem preferable given that theories of the alliance emphasize the patient’s experience. However, patients’ ratings of the alliance can be contaminated by their judgment of the benefits already experienced in therapy, and they are subject to response biases, such as acquiescence bias or central tendency bias (Tashakkori & Teddlie, 1998). The influence of prior symptom change on patients’ ratings can be mitigated to some degree by procedures that control for temporal confounds, but response biases can neither be estimated nor controlled for statistically in investigations of the relation between patient-rated alliance and outcome. These potential problems also exist for therapist-rated alliance measures.

In contrast, observer-rated methods typically employ evaluators who are blind to changes in the patients’ symptoms that occurred prior to the rated session. Moreover, unlike patients’ or therapists’ ratings, those of observers can be evaluated for inter-rater reliability, such that any biases a given rater might display can be addressed with further training, and estimated in a reliability analysis. In addition, studies that employ the same raters (patients or therapists) to assess both the alliance and outcome, as is frequently the case, may yield alliance-outcome correlations that are inflated due to problems related to shared variance (Elvins & Green, 2008; but see Horvath, 2001). These problems may be minimized with the use of independent observers, who do not also assess outcome.

The vast majority of studies of the relation between the therapeutic alliance and outcome have not controlled for temporal confounds (Barber, 2009), leaving open the possibility that significant alliance-outcome associations reflect, at least in part, the influence of prior symptom improvement on the alliance (Stiles, Shapiro & Elliot, 1986). In the context of a design that minimizes temporal confounds, our goals were to examine the relation between the WAI, assessed early in CT, and subsequent depressive symptom change, as well as the association between the WAI, assessed late in treatment, and prior symptom change. In the event that significant alliance-outcome relations were obtained, we planned to conduct exploratory analyses to determine which components of the alliance may be driving these effects. Of specific interest was the relation between the empirically-derived Agreement and Relationship factors of the WAI and both subsequent and prior symptom change. In particular, we were interested in whether the components of the alliance early in therapy that are associated with subsequent symptom change might differ from those late in therapy that account for its relation with prior symptom change.

Method

Participants

All patients from the CT conditions of the Cognitive Pharmacotherapy-II (CPT-II; N=60; DeRubeis et al., 2005) and University of Washington (UW; N=45; Dimidjian et al., 2006) studies were included. Both studies targeted adults with major depression, and CT was provided for a total of 16 weeks. (For more detailed information on each of the studies, see DeRubeis et al., 2005, CPT-II; Dimidjian et al., 2006, UW). Local institutional review board (IRB) approval was obtained for all sites and all patients provided written informed consent.

Measures

Working Alliance Inventory - short observer-rated version (WAI-O-S; Tracey & Kokotovic, 1989; Tichenor & Hill, 1989).

The WAI-O-S is a 12-item observer-rated measure of the quality of the therapeutic alliance. Items are rated on a 7-point scale (0 = never to 6 = always). In the current study, we employed the two factors derived from Andrusyna et al.’s (2001) factor analysis. The first factor (Agreement) consists of all four of the items from the Task (1, 2, 8 & 12) and Goal (4, 6, 10, 11) subscales, respectively, as well as one item (5) from the Bond subscale. The second factor (Relationship) consisted of the remaining three items from the Bond subscale (3, 7, & 9; see Table 3 in Andrusyna et al. for a list of all items). The WAI-O-S, henceforth the WAI, has been rated reliably in prior research (Strunk et al., 2010).

Beck Depression Inventory-2^nd Edition (BDI-II; Beck et al., 1996).

The BDI-II, a 21-item self-report measure of depressive symptoms, was administered to all patients in this study prior to each session. The BDI-II exhibits excellent psychometric properties (Beck et al., 1996).

Procedures

Therapy sessions were video- and audio-taped. For each therapist-patient dyad, session 3 (“early session”) and the 3^rd from last session (“late session”) were rated.¹ Raters, who took notes while viewing each session, provided ratings after the end of the session. If a videotape was unavailable, we substituted either a videotape from an adjacent session or an audiotape of the session. Two patients were not included because they dropped out of treatment prior to session 2. All other dropouts (9 CPT-II patients and 5 UW patients) left treatment between Session 2 and the 3^rd from last session; these patients were included in analyses of early, but not late, sessions. In addition, recordings of early sessions for 3 patients and late sessions for 2 patients were unavailable from the UW study. In total, 100 early sessions, and 87 late sessions, were rated.

Five students from the University of Pennsylvania served as raters. Each tape was coded by two raters independently according to a balanced incomplete block design (Fleiss, 1981). Raters were assigned no more than one session per therapist-patient dyad. Raters, who were blind to treatment outcome, site, and the study aims, as well as to participant and session numbers, received 30 hours of training, which included didactic instruction on CT for depression and, in particular, on the construct of the working alliance and the WAI observer-rated measure. During training, raters practiced coding 12 CT sessions, initially along with the study supervisor (CAW), and later, independently. Subsequent to rating each session, raters reviewed all of their ratings with the study supervisor to discuss discrepancies. In addition to considering their global impression of the quality of the working alliance in the session, raters were instructed to pay attention to any instances in the session during which there appeared to be shifts in the quality of the working alliance (e.g., disagreement about treatment goals or about the usefulness of an insession or homework activity). In order to prevent rater drift, raters met weekly with the study supervisor to rate a tape independently and to discuss any discrepancies in ratings.

Results

Means, standard deviations, and correlations for all rated variables are listed in Table 1. Intraclass correlation coefficients (ICCs), estimated for the WAI using a random effects model (Shrout & Fleiss, 1979), for the mean ratings of two raters are also reported in Table 1. The estimates are in the general range of those obtained in previous studies using observer-rated alliance scales (e.g., Strunk et al., 2010; Hanson, Curry, & Bandalos, 2002).

Table 1

Means, Standard Deviations, Intraclass Correlation Coefficients and Correlations for Alliance Variables

Prediction of Subsequent Symptom Change

Multiple regression analyses were performed to examine the relation between alliance variables (i.e., WAI, WAI-Agreement, WAI-Relationship), assessed at the early session, and subsequent change in the BDI.² End of treatment (i.e., week 16) BDI scores were used as the dependent variable (DV), adjusting for BDI scores obtained just prior to the rated session (i.e., session 3). Following DeRubeis et al. (2005) and Dimidjian et al. (2006), we used the last observation carried forward as the DV for patients who dropped out or failed to complete the final assessment. In all cases, positive standardized betas (βs) indicate that higher ratings on the given alliance variable were associated with greater symptom reduction. We performed all analyses using SAS Version 9.2 PROC REG/GLM (SAS Institute, Cary, NC).

The WAI, assessed early in treatment, was a significant predictor of subsequent symptom change (β = 0.23, t (97) = 2.84, p < .01). Agreement, on its own, predicted subsequent symptom change (β = 0.25, t (97) = 3.02, p < .01), whereas a nonsignificant trend was obtained for Relationship (β = 0.15, t (97) = 1.80, p = .07). When both WAI factors were included as predictors, and subsequent symptom change was the criterion, after accounting for Relationship, Agreement was a significant predictor of subsequent symptom change (β = .27, t (96) = 2.39, p < .02). However, after controlling for Agreement, the association between Relationship and subsequent symptom change was no longer significant (β = −.03, t (96) = −0.24, p = .81).³

Prediction of the Alliance from Prior Symptom Change

Following Feeley et al. (1999), a residualized “prior change” score was computed to examine the relation between alliance variables, assessed at the late session, and prior change in BDI. For each patient, a prior change score was calculated as the difference between pre-treatment BDI and late session BDI, adjusting for pre-treatment BDI. This residualized change score served as the IV in the regression analyses, and the late session alliance variables (WAI, WAI-Agreement, WAI-Relationship) served as the DVs. The late session WAI was predicted by prior symptom change (β = .40, t (85) = 4.00, p < .001), as were each of the factors (Agreement: β = .41, t (85) = 4.11, p < .0001; Relationship: β = .29, t (85) = 2.82, p < .01). ⁴

Discussion

Consistent with findings from numerous studies of the therapeutic alliance (Martin et al., 2000) we found, in a sample of depressed patients in CT, that stronger alliances were associated with greater symptom reduction. Of the few studies that have included a control for temporal confounds (see Barber, 2009), only two (Barber et al., 2000; Klein et al., 2003) have reported a significant association between alliance and subsequent outcome. To our knowledge, this is the first study of CT to find that subsequent symptom change is predicted by the alliance.

When we examined the relation between subsequent symptom change and the two factor-analytically derived components of the early WAI, Agreement and Relationship, Agreement appeared to account for most of the predictive variance. In contrast, both components of the late WAI were associated with prior symptom change. This pattern of findings may reflect the importance of therapist-patient agreement on the tasks and goals of CT, and that variation in the bond between therapist and patient plays a less prominent role in contributing to symptom improvement in CT. The bond did vary as a function of prior symptom change, consistent with the view that it may strengthen as a consequence of symptom improvement. It is important to note, however, that at the early session the Relationship factor was associated with subsequent symptom change at the level of a nonsignificant trend, when the Agreement factor was not included as a covariate. And although the standardized regression coefficient (.15) from the latter analysis may be considered “small,” it is by no means negligible, and is not much smaller than the mean alliance-outcome correlation of .22 reported in Martin et al.’s (2000) meta-analysis.

In the current study we only examined the relation between the alliance assessed at the third session and subsequent outcome. It is possible that a significant relation between the therapist-patient bond and subsequent symptom change would have emerged if the alliance had been assessed earlier or later in treatment. On average, approximately one-third of the symptom change occurred prior to the early session, and thus before our alliance assessment. It may be that a different pattern of results would have emerged if the alliance had been assessed even earlier in treatment (see also Gelso & Carter, 1985; Horvath & Greenberg, 1989).

We examined a relatively highly structured form of therapy. It is unclear to what extent these findings would replicate in studies of other treatment modalities, such as dynamic therapy (Summers & Barber, 2010). It may be that the symptom improvement observed in the current study is at least in part a result of the particular goals and tasks that are emphasized and pursued in CT (e.g., those related to identifying, challenging and modifying negative cognitions). Motivational Interviewing (MI; Miller, & Rollnick, 2002), with its empathic, client-centered approach may offer helpful tools for therapists to enhance their alliances (including agreement on the goals and tasks of treatment) with their patients, particularly those that are especially ambivalent about changing.

Several limitations of the present study should be noted. First, although it is fundamental to the concept of causal modeling, the “no omitted variables” assumption is impossible to verify with observational data. Thus, unmeasured third variable confounds could have influenced the results. Second, scores on the Agreement and Relationship factors were highly correlated, especially at the late session, posing problems for the multivariate approach we took to these data. We attempted to minimize problems that result from multicollinearity by using two factor-analytically-derived components of the WAI, rather than the three theory-derived subscales.

Findings from the present study raise several issues that should be addressed in future research. First, it will be important to examine whether the current findings can be replicated in other samples of depressed patients treated with CT, as well as samples comprising patients with conditions other than depression, and studies of treatment modalities other than CT. Tests of the robustness of the present findings should also examine alliance-outcome relations at different assessment timepoints, as well as with alternative methods of assessing the alliance. The results of such future investigations may lead to the specification of variables that play especially important roles in therapeutic improvement.

Acknowledgments

We are grateful to Dianne L. Chambless and Jacques P. Barber for their helpful comments on this manuscript. In addition, we thank Robert J. Kohlenberg, Keith S. Dobson; Karen B. Schmaling and David L. Dunner for their leadership of the UW trial and making data available for this project.

Footnotes

¹Therapist adherence ratings were completed at the third session for both the UW and CPT-II studies using the observer-rated cognitive behavioral subscale of the Collaborative Study Psychotherapy Rating Scale (CSPRS-CB). We examined the mean of the subset of CSPRS-CB items assessing therapist adherence to “concrete,” symptom-focused CT techniques (CT-Concrete factor; see DeRubeis & Feeley, 1990), such as helping patients identify and challenge specific maladaptive cognitions. The CT-Concrete factor mean (2.83) is similar to that reported in previous studies (DeRubeis & Feeley, 1990; Feeley et al., 1999), and suggests that even as early as session 3, the therapists in the current study were engaging in concrete, symptom-focused CT techniques. We also examined the CSPRS items assessing more “abstract” discussions between the therapist and patient (CT-Abstract), including psychoeducation about the cognitive model. The session 3 CT-Abstract mean of 2.32 suggests that therapists were also engaging patients in abstract discussions of CT and their treatment.

²There was no evidence of any Study (CPT-II and UW) X Measure interaction in any of the analyses reported. Thus, results were reported using the pooled sample of the two studies.

Several outliers were identified. With respect to the early session, one outlier for both the WAI (z = −3.92) and WAI-Agreement factor (z = −3.92) was detected. Similarly, for the late session, an outlier for the WAI (z = −4.22), WAI-Agreement factor (z = −3.88) and WAI-Relationship factor (z = −4.63) was identified. Each of these scores was replaced with the next most extreme non-outlier value in the dataset for that variable.

³It is in principle possible that the difference in the strength of the relation of the Agreement and Relationship factors to subsequent symptom change could be accounted for by differences in their inter-rater reliabilities. When we tested these relations when each was corrected for attenuation due to unreliability the same pattern of findings resulted. In addition, we reran the analyses after degrading the Agreement factor (to reduce its reliability), such that its ICC matched that of the Relationship factor (i.e., ICC = .56). The same pattern was observed.

The analyses were also rerun using residualized change scores (see Feeley et al., 1999); they yielded the same pattern of findings we reported in the Results section.

⁴In an effort to replicate the findings from previous studies, we also performed analyses using the three WAI theory-derived subscales. First, three multiple regressions were performed to examine the relation between the three subscales of the WAI and subsequent symptom change (adjusting for session 3 BDI). The Task (β = .26, p < .01), Goal (β = .21, p < .02), and Bond (β = .17, p < .04) subscales each significantly predicted subsequent depressive symptom change, with the Task subscale displaying the strongest relation with symptom change.

After accounting for the other two subscales, the Task subscale (β = .30, p = .08) was associated with subsequent symptom change at the level of a nonsignificant trend. In contrast, in parallel analyses, and when the other two subscales were statistically controlled, the Bond (β = −.01, p = .92) and Goal (β = −.04, p = .80) subscales bore virtually no relation to subsequent symptom change. Finally, with respect to the WAI assessed at the late session, each of the three subscales exhibited similar relations to prior symptom change (Task: β = .40, p < .001; Goal: β = .39, p < .001; Bond: β = .33, p < .01).

The correlation between the Agreement and Relationship factors in the late session was particularly high (r = .76). It was therefore not surprising that, when we examined the relation between each factor and prior symptom change while controlling for the other factor, the standardized betas were reduced substantially in each case (Agreement factor: β = .41 → .20; Relationship factor: β = .29→−.02); the effect remained significant only for the Agreement factor (p < .01).

The following manuscript is the final accepted manuscript. It has not been subjected to the final copyediting, fact-checking, and proofreading required for formal publication. It is not the definitive, publisher-authenticated version. The American Psychological Association and its Council of Editors disclaim any responsibility or liabilities for errors or omissions of this manuscript version, any version derived from this manuscript by NIH, or other third parties. The published version is available at www.apa.org/pubs/journals/ccp

Contributor Information

Christian A. Webb, Department of Psychology, University of Pennsylvania.

Robert J. DeRubeis, Department of Psychology, University of Pennsylvania.

Jay D. Amsterdam, Department of Psychiatry, University of Pennsylvania.

Richard C. Shelton, Department of Psychiatry, Vanderbilt University.

Steven D. Hollon, Department of Psychology, Vanderbilt University.

Sona Dimidjian, Department of Psychology, University of Colorado at Boulder.

References

Adler JV. A study of the working alliance in psychotherapy. University of British Columbia; Vancouver, Canada: 1988. Unpublished doctoral dissertation.
Andrusyna BA, Tang TZ, DeRubeis RJ, Luborsky L. The factor structure of the Working Alliance Inventory in cognitive-behavioral therapy. Journal of Psychotherapy Practice and Research. 2001;10:173–178. [PMC free article] [PubMed]
Barber JP. Toward a working through of some core conflicts in psychotherapy research. Psychotherapy Research. 2009;19:1–12. [PubMed]
Barber JP, Connolly MB, Crits-Christoph P, Gladis M, Siqueland L. Alliance predicts patients’ outcome beyond in-treatment change in symptoms. Journal of Consulting and Clinical Psychology. 2000;68:1027–1032. [PubMed]
Beck AT, Steer RA, Brown GK. Manual for the BDI-II. Psychological Corporation; San Antonio, TX: 1996.
Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice. 1979;16:252–260.
Brusseri MA, Tyler JD. Interchangeability of the Working Alliance Inventory and Working Alliance Inventory, Short Form. Psychological Assessment. 2003;15:193–197. [PubMed]
DeRubeis RJ, Feeley M. Determinants of change in cognitive therapy for depression. Cognitive Therapy and Research. 1990;14:469–482.
DeRubeis RJ, Hollon SD, Amsterdam JD, Shelton RC, Young PR, Salomon RM, et al. Cognitive therapy vs medications in the treatment of moderate to severe depression. Archives of General Psychiatry. 2005;62:409–416. [PubMed]
Dimidjian S, Hollon SD, Dobson KS, Schmaling KB, Kohlenberg RJ, Addis M, et al. Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology. 2006;74:658–670. [PubMed]
Elvins R, Green J. The conceptualization and measurement of therapeutic alliance: An empirical review. Clinical Psychology Review. 2008;28:1167–1187. [PubMed]
Feeley M, DeRubeis RJ, Gelfand L. The temporal relation of adherence and alliance to symptom change in cognitive therapy for depression. Journal of Consulting and Clinical Psychology. 1999;67:578–582. [PubMed]
Fleiss JL. Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement. 1981;5:105–112.
Gelso CJ, Carter JA. The relationship in counseling and psychotherapy: Components, consequences, and theoretical antecedents. The Counseling Psychologist. 1985;13:155–243.
Hanson WE, Curry KT, Bandalos DL. Reliability generalization of Working Alliance Inventory scale scores. Educational and Psychological Measurement. 2002;62:659–673.
Hatcher RL, Barends AW. Patients’ view of the alliance in psychotherapy: Exploratory factor analysis of three alliance measures. Journal of Consulting and Clinical Psychology. 1996;64:1326–1336. [PubMed]
Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the Working Alliance Inventory. Psychotherapy Research. 2006;16:12–25.
Horvath AO. The alliance. Psychotherapy: Theory, Research, Practice, Training. 2001;38:365–372.
Horvath AO, Greenberg LS. The development of the Working Alliance Inventory. In: Greenberg LS, Pinsoff WM, editors. The psychotherapeutic process: A research handbook. Guilford Press; New York: 1986. pp. 529–556.
Horvath AO, Greenberg LS. Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology. 1989;36:223–233.
Horvath AO, Symonds BD. Relation between working alliance and outcome in psychotherapy: A meta-analysis. Journal of Counseling Psychology. 1991;38:139–149.
Klein DN, Schwartz JE, Santiago NJ, Vivian D, Voicing C, Castonguay LC, et al. Therapeutic alliance in depression treatment: Controlling for prior change and patient characteristics. Journal of Consulting and Clinical Psychology. 2003;71:997–1006. [PubMed]
Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. Journal of Consulting and Clinical Psychology. 2000;68:438–450. [PubMed]
Miller WR, Rollnick S. Motivational Interviewing: Preparing People for Change. Vol. 2. Guilford; New York: 2002.
Paivio SC, Bahr LB. Interpersonal problems, working alliance, and outcome in short-term experiential therapy. Psychotherapy Research. 1998;8:392–406.
Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 1979;2:420–428. [PubMed]
Stiles WB, Shapiro DA, Elliott R. Are all psychotherapies equivalent? American Psychologist. 1986;41:65–80. [PubMed]
Strunk DR, Brotman MA, DeRubeis RJ. The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behavior Research and Therapy. 2010;48:599–606. [PMC free article] [PubMed]
Summers RJ, Barber JP. Psychodynamic therapy: A guide to evidence based practice. Guilford Press; New York, NY: 2010.
Tashakkori A, Teddlie C. Mixed Methodology: Combining Qualitative and Quantitative Approaches. Sage; Thousand Oaks, CA: 1998.
Tichenor V, Hill CE. A comparison of six measures of working alliance. Psychotherapy. 1989;26:195–199.
Tracey TJ, Kokotovic AM. Factor structure of the Working Alliance Inventory. Psychological Assessment. 1989;1:207–210.