In our exploration of the source of site differences in treatment response in TORDIA, we found several participant characteristics that differed across sites and were related to outcome, including suicidality, duration of the index MDD episode, hopelessness, and family conflict. A logistic regression that reported main effects of family conflict, treatment, and a treatment by hopelessness interaction fit the data well, and the addition of a treatment by site interaction did not add to the model. This can be interpreted to mean that the treatment by site interaction was in part explained by baseline differences in these clinical variables that predicted outcome.
ROC analyses added confidence to the logistic regression analyses and found two clinical variables that were different across sites and related to outcome. These variables were greater intensity of adolescent-reported parent-child conflict and severity of self-reported hopelessness. ROC analyses indicated that low levels of hopelessness and family conflict resulted in response rates approximately 20% higher than the remaining participants. Indeed, the site with the highest CBT plus medication response rate had a significantly higher percentage of patients with low scores on these variables than the three sites with the lowest CBT plus medication response rate. When analyses were conducted stratifying these variables, overall site differences were no longer statistically significant. We also looked at within site, across conditions differences and found statistically significant differences at four sites. At three of these four sites (Sites 1, 5 and 6), the direction of the difference was directly comparable to the response rate by condition. These findings help to explain the very discrepant response rates at these sites in which the medication only condition was superior to the CBT plus medication condition at Site #1 but the opposite finding was noted in Sites #5 and 6.
We are unaware of any studies that have examined clinical predictors of site differences. Nonetheless, TORDIA site difference predictors are very similar to those found in other studies of general predictors of treatment response in adolescent depression. The hopelessness findings in TORDIA are consistent with the predictors found in the Brent et al. (1998)
treatment study of adolescent depression as well as TADS (Curry et al., 2006
). Chronic depression was also a predictor in both TADS and TORDIA. Rohde et al. (2006)
found predictors comparable to TORDIA at one year follow-up of group therapy for depression: earlier MDD onset, hopelessness, and low family cohesion. However, parent-adolescent conflict, which was found to differ across sites and be related to outcome in TORDIA, was not a predictor in TADS. TADS and the Brent et al. (1998)
study also found that comorbid anxiety disorder was a predictor of response, but this was not true of TORDIA.
Our examination of other potential reasons for site differences in response rates in TORDIA did not find site differences to be related to variations in fidelity to the assessment protocol, protocol deviations, the number of participants enrolled per site, recruitment sources, or fidelity to the CBT or pharmacotherapy protocol. Although there was no relationship between CBT fidelity and outcome, these findings might also have been affected by the restricted range of CTRS scores at 5 of the 6 sites (Trepka et al., 2004
The use of particular CBT modules also did not differ across sites, with one exception. More frequent use of the Motivational Interviewing module was related to a lower response rate. Use of MI did not necessarily reflect a site difference in therapist effectiveness, but rather was more likely related to differences in participant characteristics across sites. That is, MI was only used if a participant reported significant substance use during the trial that a therapist believed might interfere with treatment of the patient's depression.
Attrition in the CBT plus medication condition differed by site and played a role in outcome differences. In the CBT plus medication condition, completers of the 12 week acute treatment protocol had response rates 23.4% higher than drop-outs. The response rate was 31.4% higher for patients who completed the medication only condition compared to dropouts. These findings should be interpreted cautiously because some participants were preemptively withdrawn by the research team because of a worsening clinical course. Although statistical analyses did not find a relationship between baseline clinical characteristics and number of CBT sessions attended, family conflict was related to drop-out. Thus, drop-out was related, in part, to a clinical characteristic (family conflict) which differed across sites and was related to overall response rates. Although one might speculate that CBT would be better able to address family conflict than a medication-only condition, the CBT protocol in TORDIA was primarily an individual protocol. Although there were family modules available, there was limited time to address family conflict in the first 12 weeks of treatment.
There are several limitations to this study of site differences. First, we chose to examine site differences by a dichotomous variable, clinical response to treatment, rather than a continuous variable, such as change in depressed mood as measured by the CDRS. Site differences, and the reason for site differences, might differ based on the outcome variable selected. Second, there were variables not assessed in this study which may have affected site differences. For example, site differences in attrition may have been related to the skill with which some sites were able to maintain participants in a demanding protocol. Alternatively, differences in therapist allegiance to CBT across sites may have been related to attrition. TORDIA did not systematically assess therapist beliefs about the effectiveness of the treatment they were delivering. Nonetheless, intervention staff were hired on the basis of prior CBT training and experience, which presumably is a reasonable proxy for allegiance to a given treatment approach. Congruence between patient desire to receive a particular treatment, i.e. CBT vs. no CBT, and their treatment assignment may have also affected drop-out, but this belief was not assessed in TORDIA.
In conclusion, as MRCTs are conducted with more clinically complex patient populations, the inclusion of multiple sites will typically lead to a more diverse clinical sample. Broadening the range of clinical characteristics contributes to the richness of studies but may also result in site differences in outcome. Consequently, the design step of MRCTs is important because lack of attention to study procedures across sites may affect the ultimate interpretability and significance of the findings. In MRCTs it is important to ensure that study procedures, i.e. factors under the control of investigators, do not result in site differences so that any site differences can be investigated in relation to participant characteristics. In TORDIA, clinical characteristics contributed to site differences much more substantially than any inconsistencies in treatment protocol delivery across sites. This finding helps allay concerns regarding the adequacy of the implementation of the clinical trial. Our findings regarding site differences also have implications for the dissemination of evidence-based treatments. If each site in an MRCT is considered its own small study, then the variability across sites in response outcomes may reflect the range of treatment response that might be expected if this treatment were to be disseminated to the community.1
MRCTs are rarely, if ever, powered a priori to account for site differences in outcomes. Noda et al. (2006)
note that researchers should instead assume that, “power of test and precision of estimates depends not on the absence of site differences, but on the degree” (p. 932). Indeed, even though most studies do not find any, or at most only a few, site-by-treatment interactions in outcomes, the lack of statistical significance is typically due to the fact that the studies are powered to detect a main effect of treatment, not site-by-treatment interactions (Kraemer, 2000
). Thus, the lack of statistically significant findings with respect to site differences does not necessarily prove the null hypothesis (Kraemer, 2000
). In the future, MRCTs should ideally be powered to detect site differences given how frequently they occur.