Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Clin Psychol Psychother. Author manuscript; available in PMC 2017 May 1.
Published in final edited form as:
PMCID: PMC4861691

Interpersonal Factors Are Associated with Lower Therapist Adherence in Cognitive–Behavioural Therapy for Panic Disorder



The contributions of disorder severity, comorbidity and interpersonal variables to therapists’ adherence to a cognitive–behavioural treatment (CBT) manual were tested.


Thirty-eight patients received panic control therapy (PCT) for panic disorder. Trained observers watching videotapes of the sixth session of a 24-session protocol rated therapists’ adherence to PCT and their use of interventions from outside the CBT model. Different observers rated patients’ behavioural resistance to therapy in the same session using the client resistance code. Interview measures obtained before treatment included the Panic Disorder Severity Scale, the anxiety disorders interview schedule for Diagnostic and Statistical Manual of Mental Disorders (DSM)-IV and the structured clinical interview for DSM-IV, Axis II. Questionnaire measures were the anxiety sensitivity index at intake, and, at session 2, the therapist and client versions of the working alliance inventory—short form.


The higher the patients’ resistance and the more Axis II traits a patient had, the less adherent the therapist. Moreover, the more resistant the client, the more therapists resorted to interventions from outside the CBT model. Stronger therapist and patient alliance was also generally related to better adherence, but these results were somewhat inconsistent across therapists. Pretreatment disorder severity and comorbidity were not related to adherence.


Interpersonal variables, particularly behavioural resistance to therapy, are related to therapists’ ability to adhere to a treatment manual and to their use of interventions from outside of the CBT model.

Keywords: Cognitive–Behavioural Therapy, Adherence, Resistance, Panic Disorder, Personality Disorders

A major goal of current psychotherapy research is the development and dissemination of empirically supported, manualized treatments (Addis & Waltz, 2002; Carroll & Nuro, 2002). Treatment manuals are intended to ensure consistent delivery of prescribed interventions, prevent the use of interventions from outside the treatment of interest (Waltz, Addis, Koerner, & Jacobson, 1993) and foster dissemination of evidence-based treatments (Task Force on Promotion and Dissemination of Psychological Procedures, 1995). When the efficacy of manualized treatments is evaluated, assessing and ensuring protocol therapists’ adherence to the manual is necessary to protect the integrity of the independent variable and to justify the conclusion that the patients received the therapy in question (Perepletchikova, 2011).

In published randomized controlled treatment efficacy trials, therapists have typically been extensively trained and supervised during the course of the trial to maintain their adherence at a high level. Treatment adherence in effectiveness studies wherein treatments are disseminated to therapists in the community has proved more challenging. For example, in a study on dissemination of brief strategic family therapy to community agencies, Shoham (2011) reported that the therapists trained for the trial provided the treatment with adequate fidelity (including both adherence and competence) in only a third of sessions sampled for objective fidelity ratings. If therapists fail to deliver the treatment with adequate levels of adherence, the investigators have not tested the treatment, and conclusions from the trial, other than the difficulty of training, are dubious.

In studies of treatment efficacy, therapists’ adherence to the treatment manual is monitored, and therapists falling below a specified threshold are flagged. Investigators’ responses to poor adherence generally focus on retraining the therapist and if this fails, removing the therapist from the protocol. Frequent supervision sessions are recommended to prevent therapist drift (Perepletchikova, 2011). This approach seems to assume that poor adherence is solely or primarily due to inability or unwillingness on the part of the therapist to follow the treatment as written and does not take into account patient or process variables that may contribute to poor adherence. However, the available evidence does not suggest that therapist adherence is a stable characteristic: In two studies, authors have reported not only between therapist variance in adherence but also within therapist variance, that is, variance in therapists’ adherence across their patients (Boswell et al., 2013; Imel, Baer, Martino, Ball, & Carroll, 2011). In both cases, there was as much variability in therapists’ adherence across their caseloads as there was between therapists. This strongly suggests that factors other than the therapist are at play in adherence. Accordingly, it is unfortunate that little research has been conducted to explore variables (therapist, client or dyadic) that contribute to variation in adherence to treatment protocols. For example, it seems reasonable to conjecture that therapists may be more adherent when cases are uncomplicated and less when clients have multiple problems or are interpersonally difficult (Boswell et al., 2013; Perepletchikova & Kazdin, 2005).

In the present study, we examine several factors that may interfere with the therapists’ ability to deliver a manualized, cognitive–behavioural treatment (CBT) for panic disorder. Panic control therapy (PCT; Craske, Barlow, & Meadows, 2000) is an efficacious treatment for panic disorder and agoraphobia that generalizes well to community settings (e.g., Addis et al., 2006). The PCT protocol is highly structured: for each session, the therapist is expected to cover several different interventions, exercises and psychoeducational modules. Thus, good adherence requires organization and careful planning on the part of the therapist and cooperation on the part of the client if all tasks are to be completed for a given session. The structure of PCT lends itself to clear assessment of adherence and thus makes it an ideal candidate for research on factors affecting therapist adherence. Our data come from a randomized controlled trial in which PCT was compared with panic-focused psychodynamic therapy and applied relaxation training (Milrod et al., 2014). In this trial, we accepted patients with a broad range of Axes I and II comorbid conditions, so long as panic disorder was the primary diagnosis. The resulting complex clinical picture of many of the cases may pose a challenge for therapists in adhering to a highly structured protocol. Following typical procedures for adherence coding, we randomly sampled cases and sessions within cases to assess therapists’ adherence for a total of 37 sessions. On average, independent coders rated adherence to the PCT protocol as adequate and observed no use of panic-focused psychodynamic or applied relaxation interventions. However, the apparent success viewed from the point of average adherence belies the fact that 19% of sessions were rated below the cut-off for acceptable PCT adherence. In the present study, we seek to identify factors that account for this variation in therapist behaviour.

In this investigation, we examine factors that are related not only to poor adherence to the PCT manual but also to the use of off-protocol interventions derived from treatment models other than CBT. Given the inherently dyadic nature of individual psychotherapy with adults, our primary interest is in interpersonal factors including patients’ resistance to therapy, early working alliance and Axis II psychopathology. In a recent survey of cognitive–behavioural practitioners, Wolf and Goldfried (2014) found that over a third reported resistance and comorbid conditions were impediments to their implementation of CBT for panic disorder, and over half cited personality disorders as a barrier to their use of the treatment. Of secondary interest are Axis I comorbidity and disorder severity. Concerning comorbidity, it seems reasonable to assume that when patients have multiple problems, therapists might have more difficulty adhering to this highly focused treatment protocol. However, to our knowledge, this conjecture has not been tested. Concerning disorder severity, prior data are mixed: Huppert, Barlow, Gorman, Shear and Woods (2006) reported that when patients started treatment higher on anxiety sensitivity, therapists were more adherent in PCT, perhaps because PCT is strongly focused on treating anxiety sensitivity and is thus readily applicable to treating high anxiety sensitivity patients. However, these authors as well as Boswell et al. (2013) failed to find that patients’ overall severity of panic disorder predicted adherence.

There is little in the way of published findings on the relationship of patients’ resistance to therapists’ adherence. In a study of interpersonal therapy for depression and therapists’ competence (rather than adherence), Foley, O’Malley, Rounsaville, Prusoff and Weissman (1987) found that supervisors’ and therapists’ own reports of therapists’ skill during therapy sessions decreased as a function of clients’ resistance, defensiveness and hostility towards the therapist. In our view, these findings would likely hold for adherence as well. Indeed, Boswell et al. (2013) found that a self-report measure of patients’ trait interpersonal aggression predicted poorer adherence on the therapists’ part to the techniques of PCT and accounted for the within therapist variability in adherence. It is possible that patients’ aggression was directed towards the therapist as well as towards others in their environment.

Given that the working alliance includes therapists’ and clients’ agreement on the goals and tasks of therapy (Horvath & Greenberg, 1986), it is reasonable to hypothesize that higher alliance would be related to better adherence. Surprisingly, the limited empirical evidence does not consistently support this hypothesis. DeRubeis and Feeley (1990), although not reporting the correlation coefficients themselves, noted that the correlation between adherence and alliance in cognitive therapy for depression was not high. Also examining cognitive therapy for depression, Castonguay, Goldfried, Wiser, Raue and Hayes (1996) found that therapists’ focus on intrapersonal consequences (including the relationship of distorted cognitions to depressive symptoms) was moderately but not significantly related to lower alliance ratings. After a qualitative examination of their sample, the authors suggested that some therapists might have difficulty maintaining a strong alliance while being adherent when patients want to deviate from the prescribed treatment to address other issues, whereas other therapists are able to do so. This suggests that there may be significant variability across therapists in the alliance–adherence relationship.

There is also limited empirical evidence of the relation of Axis II personality traits to adherence. In a study of interpersonal therapy during the maintenance phase of remitted depression, Frank, Kupfer, Wagner, McEachran and Comes (1991) found no relationship between Axis II diagnosis and therapists’ adherence to interpersonal therapy principles. However, in a naturalistic study of eating disorder treatment, Thompson-Brenner and Westen (2005) found that self-identified cognitive–behavioural spectrum therapists were more likely to report borrowing techniques from dynamic therapy when treating interpersonally difficult clients, although not to use fewer CBT techniques. The authors did not formally assess Axis II pathology, but the personality types they assessed roughly conformed to clusters B (dramatic/dysregulated) and C (anxious/constricted) descriptions. Whether these discrepant findings are due to differences in treatment type (interpersonal versus CBT) or to other factors warrant further research.

In summary, our goals in the present study are to examine predictors of adherence to a CBT protocol for panic disorder. We hypothesize that interpersonal variables such as therapeutic alliance early in treatment and patients’ resistance and Axis II symptoms will be related to lower adherence to the protocol and more use of off-protocol interventions. We also test the contribution of Axis I symptom severity and comorbidity. Given the conflicting findings in the literature as to the relationship of symptom severity to adherence (Boswell et al., 2013; Huppert et al., 2006) and the paucity of evidence on comorbidity, we make no prediction concerning these relationships.


Materials used in this study, including video-recorded therapy sessions and questionnaire and interview measures, were collected as part of a randomized controlled trial of the efficacy of 24 sessions of three different psychotherapies for panic disorder (Milrod et al., 2014). One of these, PCT (Craske et al., 2000), is a form of CBT and is the focus of the present investigation. Patients who failed to respond to the first treatment were offered their choice of the other two therapies. Three patients in the present sample had PCT as their second treatment and were assigned to a new therapist for this phase of treatment.


Patients included in parent study were between ages 18 and 70 years, had a primary diagnosis of panic disorder with or without agoraphobia and were treated at the University of Pennsylvania from 2006 to 2011. Of the 45 patients receiving PCT, 38 remained in the study at least until the fifth therapy session of whom 37 are included in the present sample.1 Of these, 16 (43%) were women, and 26 (70%) were diagnosed with panic disorder with agoraphobia. The mean age of the participants was 41.7 (standard deviation [SD] = 11.4). Twenty-nine (78.4%) of the participants were White, 6 (16.2%) were African American, 1 (2.7%) was Native American and 1 (2.7%) was Asian. Two (5.4%) of the White participants were identified as Hispanic. This research was conducted with the approval of the Institutional Review Board of the University of Pennsylvania, and informed consent was obtained from all participants in the parent study, a randomized, controlled comparison of three psychotherapies for panic disorder (Milrod et al., 2014).


Cognitive–behavioural treatment therapists were four doctoral-level clinicians (two women, two men and all White) with an average 11.75 years of post-graduate clinical experience (SD = 8.5). All had extensive experience in conducting CBT but received specific training in PCT by studying treatment manuals and attending a 2-day workshop on the specific PCT protocol. Those without extensive experience in treating panic disorder with CBT treated a pilot case under supervision before beginning with study patients. A psychologist specializing in CBT for panic disorder (Dianne L. Chambless) served as the supervisor. She initially met with therapists weekly, then monthly, and provided therapists with weekly feedback as to their adherence to the treatment manual.


Interview Measures

Anxiety disorders interview schedule for Diagnostic and Statistical Manual of Mental Disorders-IV: adult version

The anxiety disorders interview schedule for Diagnostic and Statistical Manual of Mental Disorders (DSM)-IV (ADIS-IV) (Brown, DiNardo, & Barlow, 2004) was used to diagnose panic disorder and agoraphobia for study entry as well as to assess the number of Axis I comorbid disorders for each patient. Interviewers were master’s-level psychologists trained according to recommendations of Brown and colleagues (Brown, Di Nardo, Lehman, & Campbell, 2001). Inter-rater reliability was calculated on a sample of 28 patients from the parent study and proved excellent for panic disorder (κ = 1.00) and agoraphobia (κ = .94) and acceptable to excellent for comorbid disorders that occurred frequently enough to assess, including social phobia (.70), generalized anxiety disorder (.78), obsessive–compulsive disorder (.64), specific phobia (.65), post-traumatic stress disorder (.67), major depressive disorder (.79), dysthymic disorder (1.00) and hypochondriasis (.77). Other diagnoses assessed by the ADIS include alcohol and drug abuse and dependence, somatization disorder, bipolar disorder and acute stress disorder; there were too few patients with these diagnoses in the reliability sample to calculate kappa. The ADIS includes a section of screening questions for psychotic disorders as well. Given the small number of patients with any specific comorbid diagnosis, the variable of interest for analysis was the total number of comorbid Axis I disorders other than agoraphobia. ADIS clinical severity ratings ranging from 0 (absent) to 8 (severe) were used as measures of the pretreatment severity of panic disorder and agoraphobia. Inter-rater reliability for these ratings, calculated with intraclass correlation coefficients, was excellent: ρI (2,1) = 0.86 for panic disorder and 0.95 for agoraphobia.

Structured clinical interview for DSM-IV, Axis II

The structured clinical interview for DSM-IV, Axis II includes a self-report questionnaire followed by a structured clinical interview to assess Axis II traits and permit diagnosis of Axis II disorders based on DSM-IV criteria (First, Gibbon, Spitzer, Williams, & Benjamin, 1997). In the present study, analyses examining the relationship of Axis II traits to other variables of interest were conducted using continuous variables reflecting the sum of the number of individual trait criteria met by each patient for each of the DSM-IV personality disorders. Inter-rater reliability for the total number of Axis II traits was excellent: ρI (2,1) = 0.92.

Questionnaire Measures

Anxiety sensitivity index

The anxiety sensitivity index (ASI) (Reiss, Peterson, Gursky, & McNally, 1986) is a self-report measure of belief that anxiety symptoms will have dangerous consequences. The ASI has been shown to be internally consistent and to have good convergent, discriminant and criterion-related validity (Reiss et al., 1986). The ASI was assessed at baseline and used as a measure of symptom severity. In the present sample, internal consistency was excellent: α = .91.

Working alliance inventory—short form

The working alliance inventory—short form (WAI-S) (Tracey & Kokotovic, 1989), a 12-item scale derived from the original WAI (Horvath & Greenberg, 1986), is highly correlated with the full scale (Busseri & Tyler, 2003). The client and therapist versions of the WAI-S each have excellent internal consistency (α = .83–.96; Busseri & Tyler, 2003). Because the three subscales of the WAI-S (task agreement, goal agreement and bond) were moderately to highly intercorrelated in the present sample both within and between reporter (rs = .41–.81), a single WAI-S score was computed by taking the mean of all items from the therapist-report and client-report WAI-S. Scores have a possible range from 1 (never) to 7 (always). In the present sample, internal consistency was excellent: for clients, α = .85, and for therapists, α = .86.

Coding Measures

Client resistance code

Developed by Chamberlain et al. (1986) and modified by Westra, Aviram, Kertes, Ahmed and Connors (2009) for use in studies of individual therapy for anxiety disorders, the client resistance code (CRC) is an observer measure of resistant behaviour in therapy. Coders watch an entire therapy session from beginning to end, making individual ratings at 30-s intervals. Each 30-s bin is rated for peak resistance on a scale of 0 (absence of resistance), 1 (possible/conditional resistance), 2 (clear resistance) or 3 (openly hostile behaviour). Resistance is broken down into five behaviour categories: disagree/ confront/challenge, helpless/blame/complain, own agenda/ sidetrack, not answering and questioning. For the present analyses, a single resistance variable was created by summing 1s and weighted 2s and 3s (the number of 2s was multiplied by 2 and the number of 3s by 3) and dividing it by the total number of bins to control for session length. This resulted in a score that reflects the proportion of session time characterized by resistance, weighted by the severity of the resistance.

Two advanced undergraduates were trained to use the CRC with sessions from patients not included in the present study. Although one coder is a study author (Hana F. Zickgraf), the other was uninformed as to the study hypotheses. The coders independently rated nine videos provided by the developer of the modified CRC (Henny Westra), and their scores were averaged. Inter-rater reliability of these coders with Westra’s master coders was excellent, ρI (1,2) =0.78. Subsequently, the two coders independently coded all 37 videos for the present study, and their scores were averaged for analysis. Their inter-rater reliability was excellent: ρI (3,2) = 0.86.

Multitheoretical list of therapeutic interventions

The multitheoretical list of therapeutic interventions (MULTI) (McCarthy & Barber, 2009) is an observational measure of interventions judged by experts to represent a variety of therapeutic orientations including behavioural, cognitive, dialectical behaviour, interpersonal, person-centred, process-experiential and psychodynamic psychotherapies.2 Coders watch an entire therapy session and then rate each intervention on 1 (not at all) to 5 (very typical) scales. Although the MULTI can be used by raters with psychotherapy training to discriminate among various therapeutic orientations, it is explicitly designed to be used by untrained observers as well, including undergraduate raters and psychotherapy patients themselves. The items are descriptive of therapists’ behaviours (rather than goals or intentions) so as to avoid the requirement that respondents have expertise in the theoretical systems in question to be able to perform the ratings. The subscales have acceptable to excellent internal consistency, and the measure has been shown to have high criterion-related validity as evinced by the MULTI’s accuracy in discriminating among psychotherapy sessions from different theoretical orientations. For the present study, the MULTI was used to assess therapists’ use of non-CBT or off-protocol interventions by construction of a summary measure including items from the five non-CBT orientations.3 Where CBT items appeared on both cognitive or behavioural sub-scales and a non-CBT scale, we removed these from the off-protocol measure, leaving 29 items with an internal consistency of α = .80. These items were then averaged for the summary score. Five undergraduates independently rated all sessions without being informed of study hypotheses, and their ratings were averaged for analysis. MULTI training was conducted by one of the creators of the measure, Kevin S. McCarthy. Inter-rater reliability for the off-protocol scale was acceptable, ρI (3,5) = 0.60.

Panic control therapy session 6 adherence

Adherence was rated on session-specific PCT scales developed for the treatment trial (Chambless & Sharpless, 2011). Therapists were rated on 1 (not at all) to 7 (completely) scales for their adherence to five specific prescribed interventions and on assignment of all required homework for session 6. Four undergraduate adherence coders were trained by Dianne L. Chambless using PCT sessions from patients not involved in the present study. Each rater then independently coded all 37 sessions for the present investigation. Ratings were averaged across items and across raters. Reliability was excellent, as indicated by an intraclass correlation coefficient (ρI 3,4) of 0.82.


Interview and self-report measures were administered at intake. Clients and therapists completed the WAI-S after the second therapy session to assess early alliance. Observer coding for resistance, adherence and therapists’ off-protocol behaviour was conducted on video recordings of the sixth therapy session in the PCT protocol. When adherence is assessed for a report of treatment fidelity in a treatment study, the investigator randomly samples sessions across the course of treatment. Here, to achieve greater homogeneity of session material, we elected to study adherence during the same session for all clients and therapists. PCT sessions vary markedly in activities across the protocol, and the content of some sessions is more likely to pull for resistance than others. Session 6 was chosen to assess adherence from a relatively early session4 that involved interventions other than psychoeducation. In session 6, patients reported their success with completing the first behavioural homework exercise (practice in breathing retraining), and therapists taught slow breathing techniques and then introduced cognitive restructuring, providing opportunities for patients to resist homework assignments and to object to the therapists’ challenges to their anxious thoughts. By selecting an early session, we sought to avoid cases in which adherence might be poor because treatment was failing. Indeed, Boswell et al. (2013) demonstrated that therapists’ adherence falls across the course of PCT, and the success of treatment to date might be a factor in that decline.

Data Analysis

All variables were normally distributed with the exception of the weighted resistance variable, to which an arcsine transformation was applied. Outlying data points were winsorized for resistance (n = 2), Axis II traits (n = 3), early alliance (n = 3) and off protocol (n = 5).

Because PCT adherence and off-protocol behaviour proved to be only modestly correlated with one another (r = −.33, p = 0.05), they were treated as separate dependent variables. In light of our small sample size (n = 31–37 depending on the analysis) and the complexity of the analyses, we analysed each independent variable of interest separately. To address the clustering of the patients nested within therapist, we used mixed effects modelling (MEM; Verbeke & Molenberghs, 2000). By including a fixed effect for the substantive independent variable of interest and modelling a random effect due to therapist, we accounted for the within-therapist correlation in the data. From the MEM model, we derived not only the overall slope estimates but also the slope estimates per therapist. Differences in the effect of the substantive variable across therapists were modelled by including a therapist X independent variable interaction. When the interaction was statistically significant, we conducted contrasts to test differences between pairs of therapists on slopes. When superscripts are not presented in the tables, the differences between therapists are not statistically significant.

Sensitivity analyses were then conducted by eliminating the three patients who had CBT as their second treatment after failing to respond to either applied relaxation therapy or panic-focused psychodynamic psychotherapy and repeating the analyses. In all cases, the results of the sensitivity analyses were consistent with those of the full sample, indicating that including these three patients did not affect the nature of the obtained relationships.

The effect size partial eta squared is reported and provides an indication of the percentage of variance in adherence or off-protocol interventions accounted for by the predictor. Following the Cohen (1988) conventions, we interpret partial eta squared values of .01, .06 and .14 as small, medium and large, respectively. Type I error was set at 0.05 for all analyses.


Descriptive statistics are presented in Table 1. As is evident from Table 1, adherence was generally high in the coded sessions, ranging from 5.04 to 6.80 on a scale with a maximum possible value of 7, on which 4 was considered adequate adherence. Off-protocol items were scored on a 1–5 scale, with 1 indicating that the intervention was not used during the session and 5 indicating that the session was dominated by a given intervention. Use of off-protocol interventions was relatively low, ranging from 1.78 to 2.40, with a mean of 2, indicating that use of off-protocol interventions was ‘slightly typical’ of the average session. A table of zero-order correlations among predictors and dependent variables is available from the first author. The results of hypothesis testing using MEM are reported in Tables 2 and and33.

Table 1
Descriptive statistics for ratings of therapists’ interventions and predictors
Table 2
Multilevel modelling results for panic control therapy adherence
Table 3
Multilevel modelling results for off-protocol interventions

Initial Severity

Axis I comorbidity (other than agoraphobia) was high: 27 participants (72.9%) had at least one comorbid disorder other than agoraphobia, and 15 (55.5%) had two to five other comorbid disorders. The most common comorbidities were generalized anxiety disorder (n = 16, 43.2%) and social anxiety disorder (n = 13, 35.1%).

Measures of severity included initial clinical severity ratings for panic disorder and agoraphobia on the ADIS-IV, anxiety sensitivity index and number of comorbid Axis I disorders other than agoraphobia. None of these proved to be a significant predictor of adherence: Effect sizes were all small except in the case of severity of agoraphobia, for which the effect size was medium but not statistically significant. Interactions of predictor with therapist were not significant, indicating that the findings were consistent across the four therapists. See Table 2.

The results were less consistent where off-protocol interventions were concerned. In no case was the main effect for the predictor variable significant; however, there were significant therapist X predictor interactions for the number of Axis I disorders, anxiety sensitivity index and severity of agoraphobia with therapist D differing from other therapists in different ways depending on the variable.

Interpersonal Variables


Early alliance ratings were very positive with the mean for alliance approaching the top of the scale (Table 1). The main effect for the prediction of adherence from alliance was small and not statistically significant. However, moderation by therapist was significant. As shown in Table 2, for therapists A, B and D, the more positive the early alliance, the higher their adherence. In contrast, therapist C was less adherent when early alliance was higher. Early alliance did not predict use of off-protocol interventions (Table 3).


Resistance in this sample was relatively low. Only one client received a score of 3 (most overt/hostile resistance) for even one 30-s bin. Over the course of a session (mean length was approximately 46 min, or 92 bins, SD = 8.71), patients received a score of 1 for an average 5.39 bins (SD = 3.11) and a score of 2 for an average 6.10 bins (SD = 7.63). On average, clients spent 6.13 out of 46 min, or 13.3% of the session, expressing resistance. The mean of the weighted variable before transformation was 18.82 (SD = 18.46), with a range of 1.0–84.

Consistent with our hypothesis, there were large and significant main effects for resistance such that when resistance was higher, adherence was lower and use of off-protocol interventions was greater. See Tables 2 and and3.3. The absence of significant moderation effects indicates that these results were consistent across therapists. All moderation effects were small and not statistically significant.

Axis II comorbidity

Fourteen patients (37.8%) met criteria for a personality disorder diagnosis. Cluster C (anxious) personality disorders were most common, with 9 (24.3%) meeting criteria. Five patients (13.5%) met criteria for cluster B (emotional/erratic) personality disorders, and three (8.1%) met criteria for cluster A (odd) personality disorder traits.

As hypothesized, there were large main effects demonstrating that more Axis II traits predicted lower adherence and greater use of off-protocol interventions. See Tables 2 and and3.3. However, in the case of off-protocol interventions, this effect was moderated by therapist. Therapist D differed from all other therapists in using fewer rather than more off-protocol interventions with patients who had more Axis II traits.

Unique Effects of Resistance and Axis II Psychopathology

The two most consistent predictors of therapists’ behaviours, resistance and number of Axis II traits, were themselves substantially correlated (r = .45, p = 0.01). A plausible hypothesis is that Axis II traits lead to higher resistance, which leads to lower adherence. Given the reduced power associated with this collinearity and our small sample size, our study is severely underpowered for formal tests of mediation. However, for exploratory purposes, we conducted analyses in which we included these variables in one equation, along with their interactions with therapist.

In the analysis of PCT adherence, effect sizes for both resistance and Axis II traits dropped from large when only one of them was in the equation to medium in size when both were in the equation (η2 = 0.09 for resistance, p = 0.11; η2 = 0.12, p = 0.06 for Axis II). Although medium, the effects were no longer statistically significant with our sample size. Interactions with therapist were small and not statistically significant, ps >0.47. These data suggest that resistance and Axis II traits may have both shared and unique effects on adherence.

In the analysis of off-protocol behaviour, the effect size for resistance was reduced from the very large effect seen when it was the sole predictor but remained large and statistically significant (η2 = 0.19, p = 0.02). Where Axis II traits were concerned, the main effect dropped from large in the single predictor analysis to small (η2 = 0.04, p = 0.29) when both variables were in the equation. The significant interaction with therapist was again observed (η2 = 0.22, p <0.01). Thus, although causal interpretation is tenuous with these correlational data, the findings of this analysis are at least consistent with the notion that patients with higher Axis II traits pull therapists towards off-protocol behaviours through their higher levels of resistance.

Unique Effects of Axis II Psychopathology and Early Alliance

Axis II traits and early alliance both predicted lower PCT adherence, although in the case of alliance, this effect was inconsistent across therapists. Because these two variables were themselves moderately and negatively correlated (r = −.38, p = 0.02), it is possible that Axis II traits may have had their impact on adherence through lowering early alliance. In the absence of sufficient power for a formal test of mediation, we conducted a final exploratory analysis of adherence in which we entered both early alliance and Axis II traits as predictors, along with their interactions with therapist. In this analysis, the main effect for alliance was no longer significant and fell to a small effect size, whereas the effect of Axis II remained large (η2 = 0.03 for early alliance, p = 0.75; η2 = 0.25, p = 0.005 for Axis II). The interactions with therapist were small and not statistically significant (ps >0.28). These data are inconsistent with a mediation hypothesis.


Consistent with prediction, interpersonal factors predicted poorer adherence to the PCT protocol. In contrast, symptom severity and Axis I comorbidity did not. We examined the contributions of significant predictors to PCT adherence and conducted tests for moderation of each relationship by therapist. Both resistance and Axis II trait levels made significant contributions that were consistent across therapists. These contributions proved to be partially overlapping when these variables were tested together in one prediction equation. Early alliance was also predictive of better adherence on the whole, although this effect was not entirely consistent across therapists. Moreover, early alliance effects were small and not statistically significant once the overlap between Axis II and early alliance was controlled. Thus, of those interpersonal factors tested, resistance and Axis II traits appear to be the key variables related to lower adherence

It is striking that significant relationships emerged in spite of the restricted range on several key variables; PCT adherence was generally quite high, and high levels of resistance were rarely observed. It should be noted that resistance was the only predictor gleaned from the same session as adherence and off-protocol ratings, and thus, the strength of its relationships to indices of therapists’ behaviour may have been boosted relative to the other independent variables. Our results are consistent with survey data reported by Wolf and Goldfried (2014) who found that practitioners reported difficulty in implementing CBT for panic disorder with patients with Axis II characteristics and with resistant patients.

Results were less consistent for the prediction of therapists’ use of off-protocol interventions, permitting fewer conclusions to be drawn regarding such behaviour. Therapists were clearly enjoined from using interventions from the other treatment conditions in the study (panic-focused psychodynamic psychotherapy and applied relaxation training), and data from the main study’s adherence checks (Milrod et al., 2014) indicate that they complied with this requirement. However, their training was otherwise silent as to use of off-protocol interventions. Given the lack of strict guidelines in this regard, perhaps it is unsurprising that the relationship between interpersonal and symptom severity/comorbidity variables and use of off-protocol interventions tended to differ among therapists. The exception was resistance: for all four therapists in our sample, increased resistance appeared to lead to more off-protocol borrowing.

Our findings confirm those of prior research on PCT adherence (Boswell et al., 2013; Huppert et al., 2006) in demonstrating no significant relationship between pre-treatment severity of panic disorder (nor of agoraphobia in our case) and adherence. Unlike Huppert et al. (2006), we did not find higher anxiety sensitivity to predict greater adherence. Perhaps, if we had sampled adherence during a session involving interoceptive exposure (which specifically targets anxiety sensitivity), our results would have been different. To our knowledge, our study is the first to test the relationship of Axis I comorbidity to adherence. The effect size for this predictor was close to zero. It is possible that had we had the sample size to test the relationship of specific Axis I disorders, such as major depression, to adherence, our findings might have been different. For example, we observed one session in which a patient’s worsening depression required the therapist to go off track to counter increased suicidality and hopelessness. So far, our results do not justify the common pessimism that protocol therapies cannot be used with comorbid cases.

We examined the prediction of adherence from early working alliance. Based on qualitative analysis of their session recordings, Castonguay et al. (1996) suggested that therapists differ in their ability to maintain an alliance while adhering to a treatment manual and that for some therapists, higher adherence leads to poorer alliance. Our findings concur with the notion of therapist variability in this regard, but in the opposite direction. For most of our therapists, better early alliance predicted better rather than worse adherence at a later session, suggesting that a good working relationship established early in therapy allowed therapists to proceed smoothly with the protocol. For one of our therapists, however, higher alliance tended to be related to lower adherence. Observation of this therapist’s sessions suggested that when rapport was particularly good, the therapist sometimes chatted with patients about mutual interests and was not as closely attentive to the session agenda as the protocol required. We found no evidence that in the face of a poor early alliance, therapists attempted repairs by greater use of off-protocol interventions. However, recall that early alliance was generally very high.

Finally, patients’ resistance had a significant relationship to therapists’ behaviour. Unlike other predictors, there was no evidence of moderation by therapist where use of off-protocol interventions was concerned: For all four therapists in our sample, increased resistance appeared to lead to more off-protocol borrowing. More significantly for the present study, all four therapists appeared to decrease their adherence to the PCT manual when faced with resistant clients. To our knowledge, we are the first researchers to examine the relationship of resistance to adherence. However, Foley et al. (1987) reported that higher resistance was related to therapists demonstrating less skill in delivering interpersonal therapy. To some degree, resistance is part and parcel of CBT with clients with anxiety disorders because they are often reluctant to confront their anxiety and may avoid both within sessions and in the context of homework assignments. Every CBT therapist is likely to be familiar with handling that form of resistance, and it can be readily addressed within the PCT protocol. However, resistance as defined here also included such behaviours as rejection of the therapy model, belittling the therapist, reluctance to stay on track during sessions, evasion and complaining about the therapist or therapy. The PCT manual provides little to no guidance in how to cope with such behaviours while remaining within model. Rather, the manual’s authors probably assumed that skilled therapists can manage such behaviours. That may well be true for the highly supervised, experienced therapists who take part in protocols such as ours, but we believe that this is less true for less intensively trained therapists in the community to whom protocol therapies are to be disseminated. Even among the experienced therapists in our trial, there was clearly variability in their success in managing resistant clients while staying on protocol.

The results of our investigation indicate that if investigators wish to maintain treatment integrity, they need to consider how to incorporate procedures for addressing therapy-interfering interpersonal behaviour into treatment manuals. For example, Aviram and Westra (2011) found that adding motivational interviewing to CBT for generalized anxiety disorder led to lower resistance to CBT. Manuals might include a motivational interviewing module for therapists to bring into the treatment when they encounter resistance. Moreover, highly structured manuals such as PCT might be modified to permit a certain number of off-protocol sessions to address crises that arise in patients’ lives. This is particularly important when patients with difficult comorbid conditions, particularly severe personality disorders, are included in trials, as almost inevitably will be the case as treatments are disseminated into the community for effectiveness studies. For example, our therapists could not have reasonably or ethically limited themselves to the PCT interventions when faced with a patient who was drinking at alarming levels or was suicidal. The Pediatric OCD Group (2004), for example, included a provision in their treatment manual for two off-protocol sessions to cope with urgent problems when they arise.

Developers and disseminators of paediatric psychotherapy have already begun to address this need for greater flexibility within treatment manuals. For example, modular interventions, such as the modular approach to therapy for children, have proven effective in the community, outperforming both usual care and single disorder-specific interventions for depression, anxiety and conduct problems (Weisz et al., 2012). The authors speculated that the flexible nature of such approaches allows clinicians to tailor a treatment programme to the diverse needs of clients and to address secondary comorbidities, without sacrificing fidelity to evidence-based principles.

Optimizing patient outcomes is the goal of any clinical intervention. In some cases, this goal may conflict with the need to maintain treatment adherence during a research trial. The findings of the present study offer further evidence for the need to build into treatment manuals model-consistent and evidence-based interventions to address clients’ changing needs. Resistant behaviour during a session and the presence of Axis II comorbidity are just two of an array of client behaviours and needs that might derail a session and cause a clinician to deviate from a highly structured manual. Such deviations are appropriate and clinically warranted when necessary to maintain the therapeutic relationship or keep the client safe. Nevertheless, to the degree that necessary modifications for common problems are not built into the treatment manual, the treatment actually tested in a trial is not the treatment as described in the manual. This is unfortunate because the treatment as described is what will be disseminated to the community, and the manual will be misleading to the extent that does not include the procedures that were necessary to obtain the effects found in efficacy trials. In the context of treatment efficacy and effectiveness research, manuals that explicitly provide guidance for flexibility within the treatment model will help to address emergent patient needs without sacrificing the validity of the treatment. By anticipating common causes for deviation and building in specific modules to address them (e.g., motivational interviewing modules for resistant clients (Aviram & Westra, 2009) and modular interventions for common comorbidities (Weisz et al., 2012)), efficacy researchers can (a) remove a systematic source of variance from their studies, improving the ability of efficacy research to reveal significant treatment effects (Crits-Christoph et al., 1991), (b) provide a more faithful description of the treatment as actually delivered for dissemination to practitioners and (c) hopefully deliver a more efficacious treatment for study clients.

Limitations of the present study include the small sample size, which precluded tests of the contribution of specific comorbid conditions. Also, the restricted range on both early alliance and PCT adherence may have hampered our ability to detect significant relationships between these variables. It is possible that in a less carefully selected or supervised group of therapists, alliance might not always be high, and adherence might be more variable. Thus, the degree to which the present results will generalize to effectiveness studies remains to be seen. The convergence of our results with practitioners’ unsystematic observations of problems in their practices with delivery of CBT for panic disorder (Wolf & Goldfried, 2014) strongly suggests that our results will hold. Indeed, we would expect that the observed relationships would be even stronger with greater variance in the data.

The results of this study are correlational, and thus, it is possible that unmeasured third variables account for the relationships between interpersonal variables and adherence that we observed. In most cases, we were able to establish the temporal sequence of the independent and dependent variables, the first step in establishing a causal sequence, but in the case of resistance, this was not so: both resistance and adherence were necessarily coded from the same therapy session to test the study hypothesis. It remains possible, although in our view implausible, that patients became more resistant in sessions in which therapists were less adherent or used more off-protocol interventions rather than the converse. Similarly, because adherence was not measured before early alliance, it is possible, although not plausible, that early adherence failures on the therapists’ part led to poor alliance formation. Finally, our resources did not extend to replicating the results of our investigation across multiple sessions of the protocol. We think it unlikely that our results represent only early problems with adherence, in part because adherence problems seem to grow across treatment (Boswell et al., 2013). Nonetheless, additional research is needed to verify that this is correct.

We encourage replication of our findings but believe that they are suggestive enough to warrant investigators’ consideration of ways to increase adherence by guiding therapists in their treatment of patients with therapy-interfering interpersonal behaviour. Moreover, like Foley et al. (1987), we believe that our results indicate that patients’ therapy-interfering interpersonal behaviours need to be taken into account when judgments of therapists’ performance are to be made, for example, in training and selection for a clinical trial.

Key Practitioner message

  • Patients’ behavioural resistance to therapy may make it more difficult for cognitive–behavioural clinicians to adhere to a structured treatment protocol and more likely for them to borrow interventions from outside the CBT model.
  • Patients’ Axis II traits may make adherence to treatment CBT protocol more difficult, although whether this is true varies across therapists.
  • Therapists’ adherence to a structured protocol and borrowing from outside of the CBT model do not appear to be affected by disorder severity or Axis I comorbidity.


This work was supported in part by NIMH R01 MH 070664 and R01 MH070918. The funding agency exercised no control over the design, conduct and report of this study. The authors are grateful to Henny Westra and her graduate students for training in resistance coding and to resistance and adherence coders Lina Deghalyi, Marissa Schwartz, Cameron Kiani, Emma Byrne and Rachel Russell.


1For three patients whose data were used in the present study, the session 6 recording was not available, and session 5 was substituted. Sessions 5 and 6 covered comparable material, and adherence ratings were made on the same 1–5 scale (see measures and panic control therapy adherence). Because session 5 was used when session 6 recordings were unavailable, resistance was necessarily coded from session 5 in these cases as well. One patient who remained in the study until at least the fifth session was excluded from the present study sample because she was treated by a therapist who did not treat any other patients in the PCT condition of the study.

2Dialectical behaviour therapy is a specific form of CBT rather than a separate school of therapy. However, for the purposes of this study, we judged dialectical behaviour therapy to be different enough from PCT to include as a deviation from the PCT protocol.

3The MULTI also has a common factors subscale. Because this sub-scale is designed, by definition, to be pertinent to all theoretical systems, items from this subscale were included in neither the CBT nor the off-protocol measures.

4In the standard PCT protocol, these interventions would come sooner than the sixth session. For the purposes of the parent study, PCT, typically 11 sessions in length (e.g., Barlow, Gorman, Shear, & Woods, 2000), was stretched to 24 sessions, and thus, the pace of treatment was slower. This was performed to equate PCT in length and amount of therapist contact with psychodynamic psychotherapy for panic disorder (Milrod et al., 2007), a treatment to which it was being compared.


The authors declare that they have no conflict of interests.


  • Addis ME, Hatgis C, Cardemil E, Jacob K, Krasnow AD, Mansfield A. Effectiveness of cognitive-behavioral treatment for panic disorder versus treatment as usual in a managed care setting: 2-year follow-up. Journal of Consulting and Clinical Psychology. 2006;74:377–385. doi: 10.1037/0022-006X.74.2.377. [PubMed] [Cross Ref]
  • Addis ME, Waltz J. Implicit and untested assumptions about the role of psychotherapy treatment manuals in evidence-based mental health practice. Clinical Psychology: Science and Practice. 2002;9:421–424. doi: 10.1093/clipsy/9.4.421. [Cross Ref]
  • Aviram A, Westra HA. The impact of motivational interviewing on resistance in cognitive behavioural therapy for generalized anxiety disorder. Psychotherapy Research. 2011;21:698–708. doi: 10.1080/10503307.2011.610832. [PubMed] [Cross Ref]
  • Barlow DH, Gorman JM, Shear MK, Woods SW. Cognitive-behavioral therapy, imipramine, or their combination for panic disorder: A randomized controlled trial. Journal of the American Medical Association. 2000;283:2529–2536. doi: 10.1001/jama.283.19.2529. [PubMed] [Cross Ref]
  • Boswell JF, Gallagher MW, Sauer-Zavala SE, Bullis J, Gorman JM, Shear MK, … Barlow DH. Patient characteristics and variability in adherence and competence in cognitive-behavioral therapy for panic disorder. Journal of Consulting and Clinical Psychology. 2013 doi: 10.1037/a0031437. Advance online publication. [PMC free article] [PubMed] [Cross Ref]
  • Brown TA, DiNardo PA, Barlow DH. Anxiety disorders interview schedule adult version (ADIS-IV): Client interview schedule. New York: Graywind Publications; 2004.
  • Brown TA, Di Nardo PA, Lehman CL, Campbell LA. Reliability of DSM-IV anxiety and mood disorders: Implications for the classification of emotional disorders. Journal of Abnormal Psychology. 2001;110(1):49–58. doi: 10.1037//0021-843X.110.1.49. [PubMed] [Cross Ref]
  • Busseri MA, Tyler JD. Interchangeability of the working alliance inventory and working alliance inventory, short form. Psychological Assessment. 2003;15:193–197. doi: 10.1037/1040-3590.15.2.193. [PubMed] [Cross Ref]
  • Carroll KM, Nuro KF. One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science and Practice. 2002;9:396–406. doi: 10.1093/clipsy/9.4.396. [Cross Ref]
  • Castonguay LG, Goldfried MR, Wiser S, Raue PJ, Hayes AM. Predicting the effect of cognitive therapy for depression: A study of unique and common factors. Journal of Consulting and Clinical Psychology. 1996;64:497–504. doi: 10.1037/0022-006X.64.3.497. [PubMed] [Cross Ref]
  • Chamberlain P, Davis JP, Forgatch M, Frey J, Patterson GR, Ray J, … Trombley J. Unpublished manual. Oregon Social Learning Center; Eugene, OR: 1986. The therapy process code: A multidimensional system for observing therapist and client interactions.
  • Chambless DL, Sharpless BA. Unpublished measure. Department of Psychology, University of Pennsylvania; 2011. Adherence ratings for the panic control therapy protocol for psychotherapies for panic disorder study.
  • Cohen J. Statistical power analysis for the behavioral sciences. 2. Hillsdale, NJ: Lawrence Erlbaum; 1988.
  • Craske MG, Barlow DH, Meadows EA. Mastery of your anxiety and panic: Therapist guide for anxiety, panic, and agoraphobia. 3. New York: Oxford University Press; 2000.
  • Crits-Christoph P, Baranackie K, Kurcias J, Beck A, Carroll K, Perry K, … Zitrin C. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research. 1991;1:81–91. doi: 10.1080/10503309112331335511. [Cross Ref]
  • DeRubeis RJ, Feeley M. Determinants of change in cognitive therapy for depression. Cognitive Therapy and Research. 1990;14:469–482. doi: 10.1007/BF01172968. [Cross Ref]
  • First MB, Gibbon M, Spitzer RL, Williams JBW, Benjamin LS. Structured clinical interview for DSM-IV Axis II personality disorders (SCID-II) Washington, DC: American Psychiatric Press; 1997.
  • Foley SH, O’Malley S, Rounsaville B, Prusoff BA, Weissman MM. The relationship of patient difficulty to therapist performance in interpersonal psychotherapy of depression. Journal of Affective Disorders. 1987;12:207–217. doi: 10.1016/0165-0327(87)90029-2. [PubMed] [Cross Ref]
  • Frank E, Kupfer DJ, Wagner EF, McEachran AB, Comes C. Efficacy of interpersonal psychotherapy as a maintenance treatment of recurrent depression: Contributing factors. Archives of General Psychiatry. 1991;48:1053–1059. doi: 10.1001/archpsyc.1991.01810360017002. [PubMed] [Cross Ref]
  • Horvath AO, Greenberg LS. Development of the working alliance inventory. In: Greenberg LS, Pinsoff WM, editors. The psychotherapeutic process: A research handbook. New York: Guilford; 1986. pp. 529–556.
  • Huppert JD, Barlow DH, Gorman JM, Shear MK, Woods SW. The interaction of motivation and therapist adherence predicts outcome in cognitive behavioral therapy for panic disorder: Preliminary findings. Cognitive and Behavioral Practice. 2006;13:198–204. doi: 10.1016/j.cbpra.2005.10.001. [Cross Ref]
  • Imel ZE, Baer JS, Martino S, Ball SA, Carroll KM. Mutual influence in therapist competence and adherence to motivational enhancement therapy. Drug and Alcohol Dependence. 2011;115:229–236. doi: 10.1016/j.drugalcdep.2010.11.010. [PMC free article] [PubMed] [Cross Ref]
  • McCarthy KS, Barber JP. The multitheoretical list of therapeutic interventions (MULTI): Initial report. Psychotherapy Research. 2009;19:96–113. doi: 10.1080/10503300802524343. [PubMed] [Cross Ref]
  • Milrod B, Chambless DL, Gallop R, McCarthy KS, Sharpless BA, Barber JP. Dynamic therapy vs. CBT vs. ART for panic disorder: A randomized controlled trial. 2014 Manuscript in preparation.
  • Milrod B, Leon A, Busch F, Rudden M, Schwalberg M, Clarkin J, … Shear M. A randomized controlled clinical trial of psychoanalytic psychotherapy for panic disorder. American Journal of Psychiatry. 2007;164:265–272. [PubMed]
  • Pediatric OCD Group. Cognitive-behavior therapy, sertraline, and their combination for children and adolescents with obsessive-compulsive disorder: The pediatric OCD treatment study (POTS) randomized controlled trial. JAMA, the Journal of the American Medical Association. 2004;292:1969–1976. doi: 10.1001/jama.292.16.1969. [PubMed] [Cross Ref]
  • Perepletchikova F. On the topic of treatment integrity. Clinical Psychology: Science and Practice. 2011;18:148–153. doi: 10.1111/j.1468-2850.2011.01246.x. [PMC free article] [PubMed] [Cross Ref]
  • Perepletchikova F, Kazdin AE. Treatment integrity and therapeutic change: Issues and research. Clinical Psychology: Science and Practice. 2005;12:365–383. doi: 10.1093/clipsy/bpi045. [Cross Ref]
  • Reiss S, Peterson RA, Gursky DM, McNally RJ. Anxiety sensitivity, anxiety frequency, and the prediction of fearfulness. Behaviour Research and Therapy. 1986;24:1–8. doi: 10.1016/0005-7967(86)90143-9. [PubMed] [Cross Ref]
  • Shoham V. Treatment fldelity: The elusive independent variable in psychotherapy research. Paper presented at the meeting of the North American Chapter of the Society for Psychotherapy Research; Banff, Alberta, Canada. 2011. Sep,
  • Task Force on Promotion and Dissemination of Psychological Procedures. Training in and dissemination of empirically validated treatments: Report and recommendations. The Clinical Psychologist. 1995;48:3–23.
  • Thompson-Brenner H, Westen D. Personality subtypes in eating disorders: Validation of a classification in a naturalistic sample. British Journal of Psychiatry. 2005;186:516–524. doi: 10.1192/bjp.186.6.516. [PubMed] [Cross Ref]
  • Tracey TJ, Kokotovic AM. Factor structure of the working alliance inventory. Psychological Assessment. 1989;1:207–210. doi: 10.1037/1040-3590.1.3.207. [Cross Ref]
  • Verbeke G, Molenberghs G. Linear mixed models for longitudinal data. New York: Springer-Verlag; 2000.
  • Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology. 1993;61:620–630. doi: 10.1037/0022-006X.61.4.620. [PubMed] [Cross Ref]
  • Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Miranda J, Bearman SK. … Research Network on Youth Mental Health. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry. 2012;69(3):274–282. [PubMed]
  • Westra HA, Aviram A, Kertes A, Ahmed M, Connors L. Unpublished manual. Department of Psychology, York University; Toronto, ON, Canada: 2009. Manual for coding resistance in CBT for anxiety.
  • Wolf AW, Goldfried MR. Clinical experiences in using cognitive-behavior therapy to treat panic disorder. Behavior Therapy. 2014;45(1):36–46. doi: 10.1016/j.beth.2013.10.002. [PubMed] [Cross Ref]