Psychotherapy is a complex enterprise, involving many factors: patient, therapist, treatment factors, and complex interactions between all these. The randomized controlled trial design is ideally suited to isolating the treatment factor: when patients are randomized to treatments, confounding factors should be equally distributed between groups. However, because it is usually not feasible (and probably not desirable) to randomize therapists to treatments, the problem of confounding therapist and treatment effects remains.
Some researchers have advocated solving this problem by using the same psychotherapists to conduct more than one study treatment. Despite the salience of this design issue, no research has previously reviewed trials with the “crossed therapist” design. Our main finding was that the great preponderance of the 39 such studies we collected failed to report the key issue of therapist treatment allegiance. Despite the glaring need in a crossed design to ascertain therapist allegiance to the respective treatments, less than half (36%, n
=14) the trials reviewed even mentioned therapist or researcher allegiance. Still fewer (13%, n
=5) explicitly reported attempting to control for, and only one (3%) actually measured, therapist allegiance. About two thirds of the studies surveyed did not report therapists’ previous training or experience in the treatments they provided. Most studies included involved CBT as a treatment (87%, n
=34), and researchers with CBT allegiance more often used this design without controlling for therapist allegiance. Thoma et al. (62
) found that the average CBT trial of depression scored at the lower range of adequate methodological quality and that CBT trials did not on average show better quality than psychodynamic trials
Although most researchers did not even mention therapist allegiance as a possible bias, some acknowledged that their study therapists believed the comparison treatment to be ineffective and that this may have biased results:
“The choice of the same therapists for both treatments created a twofold problem. On the positive side, the effect of the same therapist was constant across the two therapeutic conditions. On the negative side, all were CBT-oriented therapists and this may have biased the treatment in favor of CBT as the therapists were requested to use methods they judged as noneffective.” (11, p. 108)
Our analyses, corroborating earlier research (14
) while surveying largely different outcome trials, showed that researcher allegiance clearly influenced treatment outcome. Our main research question was whether this effect diminished in studies that controlled for therapist allegiance. Interestingly, we found no differential researcher allegiance in any of the studies that had reportedly attempted some control for therapist allegiance. This sharply contrasted with the studies not attempting such control, for which mean researcher allegiance score was 2.1 on a scale ranging from 0 to 3. The stronger the researchers’ allegiances to one psychotherapy, the more likely they appeared to ignore therapist allegiance. If this indeed indicates how research using the crossed therapist design is being conducted, it is remarkable and worrisome that researchers strongly allied to one treatment consciously or unconsciously overlook the potential bias of therapist allegiance.
Some researchers may argue that treatment integrity checks provide an objective assessment of whether therapists deliver treatment as it is meant to be delivered, hence integrity checks should reveal therapist allegiance effects as problems in the delivery of treatment. Although our review did not focus on integrity, we did note that almost seventy percent (n
=27) of the included studies reported formal adherence analyses, and only one of these reported an adherence problem. Treatment integrity should ideally include ratings of therapist competence. Assessing competence is difficult, however, and a recent meta-analysis showed no significant relationship of competence ratings to outcome (63
). We doubt that integrity analyses based on fairly coarse measures will detect more subtle effects of therapists who technically adhere to the treatment protocol but do not deliver treatment with the same enthusiasm as therapists who really believe in their model.
Our analyses suggested that controlling for therapist allegiance has fallen out of fashion: five of the six studies that controlled for therapist allegiance were published between 1987 and 1996, and only one since. Our impression is that the psychotherapy research community has seldom discussed the issue. Therapist allegiance remains a crucial unstudied factor in psychotherapy research. Strength of belief in a therapy may affect the therapist’s comfort and authenticity in conducting treatment, the therapy’s plausibility for the patient, and thus the strength of the therapeutic alliance. This matters when a therapist conducts a single therapy, and still more when the therapist delivers more than one modality.
If therapist allegiance influences therapeutic outcome, then the “crossed therapist” design may prove a double-cross: while ostensibly controlling for therapist factors between therapists, it may merely obscure them within the same therapist. Although grant reviewers frequently suggest controlling for therapist effects by “crossing” therapists, the research supporting this injunction is essentially non-existent. Does choosing a “crossed therapist” design solve the problem of confounding therapist effects with treatment effects? The answer depends on what kinds of therapist characteristics affect outcome. If general personality characteristics (warmth, empathy, interpersonal skills, etc.) that could reasonably be considered independent of treatment type matter most for patient outcome, the crossed design might control for those effects. If therapist effects rather reflect competence in or enthusiasm for a particular treatment model, however, the alternative design of “amicably competitive” balanced therapist teams (64
) is a better choice. In this design, equally experienced, competent, and adherent teams of psychotherapists compete in friendly rivalry, demonstrating in effect the optimal effects of a given intervention. Rather than these therapists attempting multiple competencies, their goal is expertise in their assigned modality, with variables such as experience and competence counterbalanced across groups.
One could argue that researcher allegiance will out, regardless of design: the alternative design structure of nested “amicably competitive” expert therapist teams (64
) could be equally compromised if the principal investigator chooses less competent or experienced therapists for comparator conditions. We recommend that all clinical trials (“crossed” or “nested”) measure and report therapist allegiance, at least until more is known about the effect of therapist allegiance on the process and outcome of psychotherapy. To our knowledge, no instrument for measuring therapist allegiance has yet been published. In our own ongoing studies we have developed simple self-report instruments for this purpose.
This is a new field of research. We sought but may not have located all recent studies using the “crossed therapist” design. How common this approach is relative to “nested” competitive therapy teams delivering single modalities is unclear; the latter seems more prevalent. The relatively small number of studies included limits the conclusions that can be drawn, especially about the effect of controlling for therapist allegiance as this occurred in only six of the studies. A key limitation (and key study finding) is the absence of therapist allegiance ratings in the psychotherapy trials under study.
Recommendations for future research
Researchers should rigorously report therapists’ backgrounds when using the crossed therapist design, clarifying the training and experience study therapists have in conducting each of the different treatments they use in the trial, and not just trial-related training but also previous training and experience. Researchers should report whether they have tried to statistically control for therapist effects (including allegiance) in their main outcome analyses; and if not, why not (i.e., what significance criterion they used in preliminary analyses to rule out therapist effects). A “crossed therapist” design, if employed, should control for therapist x treatment interaction effects, not just therapist main effects. (Some researchers infer an absence of therapist effects in their study when they find no differences in outcome between individual therapists (e.g., 20
). Yet finding no therapist differences in outcome does not rule out the possibility that all therapists deliver one treatment better than the other either due to either greater competency or allegiance to that treatment.)
All clinical trials should routinely measure therapist allegiance and relate it to outcome. The FACT could be added to existing RCT quality measures (17
) in reviews and meta-analyses of RCTs.
Given the potential problems of the “crossed therapists” design, why use it? We recommend that in standard efficacy trials, where the predominant question is differential efficacy between two treatments, the “crossed design” be considered a weaker alternative to the “amicably competitive” therapist teams approach. The “crossed therapists” design could have utility for studying interactions between therapist characteristics and treatment method effects on outcome. Another indication for this design might be practical, where geography or other factors constrain patient access to multiple therapists practicing different treatments. We recommend that researchers avoid “crossed therapists” designs unless the research questions clearly focus on issues that this design facilitates, or unless practicalities such as geographic isolation preclude alternative designs. If used, “crossed therapist designs” must measure therapist allegiance.
Therapist allegiance warrants consideration equal to researcher allegiance in designing and interpreting the results of psychotherapy trials. Therapists in every clinical trial could rate on a simple, anchored Likert-type scale their belief in as well as their self-perceived skill in all the study’s therapies; how well each therapy fits their clinical perspectives; and their prediction of the prospective study outcome. Such evaluations would help to control for potential study biases and might yield interesting findings about psychotherapy outcome trials. Combining such data with psychotherapy process and outcome variables might elucidate the suitability of particular therapists for particular therapies.
- Therapist belief in treatment is likely to be a strong “non-specific” effect of psychotherapy, yet this factor has almost never been studied
- Researcher allegiance may influence study findings in part through selecting biased therapists in “crossed therapist” study designs