This study used multi-level modeling to identify child, family, and therapist characteristics associated with observed use of psychotherapeutic strategies common in evidence-based practices for children with disruptive behavior problems. The results indicate that the overall intensity of observed delivery of elements of EBP was relatively low for strategies directed towards both children and caregivers. Despite the significant proportions of variation in EBP delivery globally accounted for by child/caregiver and therapist characteristics, few of the specific characteristics measured in this study were significantly associated with EBP delivery. Great Child EBP was associated with older child age, higher caregiver educational level, greater caregiver alcohol use, and having a therapist with a self-reported cognitive-behavioral or behavioral primary theoretical orientation (compared to “eclectic/other”). Although no child, family, or therapist characteristics were significantly associated with Caregiver EBP, certain child symptom, family sociodemographics, and therapist experience characteristics were marginally associated with EBP delivered to caregivers.
Although EBP elements were observed in almost all sessions for almost all children/caregivers and by almost all therapists, overall intensity of EBP elements was relatively low. On a scale with a possible range from 0 to 6, the average EBP composite score was approximately one for use with children and caregivers. This finding is consistent with previous analyses of these data (Garland et al. under review) using different metrics to measure EBP delivery. Specifically, in our previous analyses, we examined the frequency of occurrence and intensity (if observed) separately for individual EBP elements, whereas in the current study we examined, overall, to what degree EBP strategies were observed within a session, child/caregiver, and therapist. In our previous analyses, average intensity when the individual strategies were used was relatively low. Regardless of the method we used to characterize EBP strategies, observed delivery of EBP strategies could be characterized as lacking depth.
The only significant child characteristic associated with EBP delivery was child age. The finding that older children had higher Child EBP scores is consistent with the targeted age groups of evidence-based youth skills training interventions (Eyberg et al. 2008
). It is also consistent with our own research indicating that therapists value EBP elements more with older than younger children (Brookman-Frazee et al. 2009
). Child symptom severity was also a marginally significant predictor of Caregiver EBP. Specifically, caregivers of children with higher ECBI intensity scores had higher Caregiver EBP composite scores. It may be that therapists perceive more severe problem behavior as requiring more intensive parent training (i.e., Caregiver EBP).
Socioeconomic status indicators did predict Child EBP composite scores. Specifically, caregiver level of education was positively associated with Child EBP and annual household income was marginally positively associated with Child EBP. This may be explained by the demand characteristics of the family. Although speculative, it may be that more highly educated families expect more active (consistent with elements of EBP) treatment for their children.
The only caregiver psychosocial characteristic associated with EBP delivery was alcohol use. The finding that caregiver alcohol use was marginally associated with greater Child EBP may be explained by therapists focusing more intensively on child skills than parent skills in therapy if the caregiver is seen as less engaged in treatment due to their own psychosocial needs.
Certain therapist characteristics were associated with greater EBP delivery. Therapists who self-identified as cognitive-behavioral or behavioral had higher Child EBP composite scores than therapists who identified as eclectic (or “other”) orientations. This finding is consistent with the fact that most of the EBPs for youths with disruptive behavior problems are behavioral or cognitive behavioral (Eyberg et al. 2008
). Less experienced therapists (defined by fewer months practiced) had marginally higher Caregiver EBP than more experienced therapists. This is consistent with research indicating that less experienced therapists may hold more positive attitudes towards EBPs (Aarons 2004
Although there were a few important factors identified as associated with observed delivery of EBP, many factors that previous research suggests may be important, or might be assumed to be related to treatment delivery (e.g., client diagnosis, therapist discipline) were not significantly associated with observed use of EBP. Given that significant proportions in variability of EBP composite scores were associated with child/family and therapist differences, the few significant characteristics (particularly with Caregiver EBP) is striking and indicates that much of the variability in treatment process is explained by characteristics not examined. Key factors that we did not measure include therapist training in EBP models, type and amount of supervision (many of the therapists were unlicensed and were therefore receiving supervision), and previous experience treating children with disruptive behavior problems. Further, we did not include the child’s auxiliary treatment. It may be that receiving other services (e.g., in home behavior support) or being placed in a higher level of care during the course of outpatient treatment impacts the treatment process. Further, we did not include a measure of caregiver-therapist or child-therapist therapeutic alliance, which may be associated with the strategies that therapists employ. Alternatively, the lack of findings may be related to the nature of the data, given the EBP composite scores were positive skewed. Further, the lack of significant associations with EBP strategies directed to caregivers may be due to less power given the reduced sample size in those models. Unfortunately, we cannot determine whether power was an issue given the lack of current technology to conduct three-level power analyses, based on the sample sizes in the Caregiver EBP analyses. Based on our sample sizes in caregiver analyses (81 therapists, 180 caregivers, 762 sessions), however, we would expect to have sufficient power to detect moderate effects in analyses of between-group differences, suggesting that this may not be a significant issue.
One of the primary strengths of this observational study is the large, representative sample of therapists, patients, and psychotherapy sessions. The distributions of clinician education level, gender, race/ethnicity, and trainee status is similar to other studies of community-based mental health providers (Glisson et al. 2008
; Hawley and Weisz 2005
). Selection bias was minimized by initially recruiting therapists by random selection and then sequentially as they entered the clinic. The patient sample is also comparable to other clinical samples of children in publicly-funded mental health care. Specifically, the male to female ratio, over-representation of race/ethnic minority children and diagnostic distribution are consistent with other studies (Bickman et al. 1995
; Eyberg et al. 2008
; Foster et al. 2001
; Rosenblatt and Rosenblatt 2000
; Zima et al. 2005
Another important strength of this study was our method of characterizing UC psychotherapy. That is, observational data on psychotherapy treatment processes provide the richest information to date on what actually happens in usual care. Despite this strength, the resulting measure only assesses observable therapist behavior. We did not capture therapists’ intentions, goals, or decision-making processes, patients’ responses to different intervention strategies, or communication outside the office or via telephone. Further, although we attempted to minimize the effect of observation by establishing video-taping as a routine practice in the clinics and using small, unobtrusive cameras mounted in the upper corners of therapists’ offices, we do not know how videotaping psychotherapy sessions may have influenced participants’ behavior (see Garland et al. 2009
). It is important to note that the PRAC TPOCS-S was not intended to be used by usual care providers to assess practice. Rather, it was developed for use in research studies aimed to rigorously characterize usual care practice. See Garland et al. (2009
) for a discussion of methodological challenges to characterizing usual care practice.
In the current study, we differentiated strategies based on whether they had been identified as a common treatment element in EBPs for children with disruptive behavior problems, and conducted analyses to identify characteristics associated with a composite of EBP elements. We designated treatment strategies as “common in EBPs” based on one method of identifying these elements (Garland et al. 2008a
). Other methods might yield additional or different common elements of EBPs.
Despite these limitations, the results of this study have important implications for the implementation of evidence-based interventions in UC. In particular, these findings highlight the complexity of UC psychotherapy, such that there are no clear patterns of characteristics associated with delivery of EBP elements. Given the generally low intensity of observed elements of EBP, there is a need for training interventions to strengthen therapists’ use of these treatment elements, particularly the individual strategies that are infrequently observed to occur or are observed with low intensity (Garland et al. 2009
). These observational data can provide a “road map” for individual EBP treatment elements that require particular attention in therapist training intervention. In addition, the findings suggest that we can not make assumptions about who is delivering what type of service to whom. However, the results can provide some initial direction for efforts aimed at implementing EBP treatment models in usual care settings. For example, extra attention may be paid to encouraging EBP delivery with children in lower socioeconomic families compared to higher socioeconomic families based on the results. Next steps in this line of research include examination of outcome trajectories and analyses to determine how specific therapist strategies may be associated with different child or family outcomes. We will also conduct detailed analyses of the associations between child, family and therapists characteristics and treatment outcomes.