|Home | About | Journals | Submit | Contact Us | Français|
Many youth receiving community mental health treatment do not receive evidence-based interventions. Research suggests that community mental health therapists use a broad range of therapeutic techniques at low intensities. The present study examined the relationship between therapist- and client-level predictors on community-based therapists’ report of cognitive, behavioral, psychodynamic, and family techniques within the context of implementation efforts.
One hundred thirty therapists from 23 organizations in an urban publicly funded behavioral health system implementing evidence-based practices participated. Therapist-level predictors included age, gender, clinical experience, licensure status, and participation in evidence-based practice initiatives. Child-level predictors included therapist-reported child primary disorder (i.e., externalizing, internalizing, or other) and child age. Therapists completed the Therapist Procedures Checklist- Family Revised, a self-report measure of therapeutic techniques used.
Unlicensed therapists were more likely to report use of both psychodynamic and behavioral techniques. Therapists who did not participate in an evidence-based practice initiative were less likely to report use of cognitive techniques. Those with externalizing clients were more likely to report use of behavioral and family techniques. Therapists with the youngest clients (aged 3-7) were most likely to report use of behavioral techniques and less likely to report use of cognitive and psychodynamic techniques.
Results suggest that both therapist and client factors predict self-reported use of therapy techniques. Participating in an evidence-based practice initiative increased report of cognitive techniques. Therapists reported using more behavioral and family techniques for youth with externalizing disorders and fewer cognitive and psychodynamic techniques with young clients.
The past fifty years have resulted in a proliferation of evidence-based practices (EBPs) for psychiatric disorders for a range of client populations and presenting problems (1, 2). Despite the development and validation of these treatments, most youth do not have access to EBPs in their communities (3, 4). While the need to translate EBPs from research to practice settings has been identified (5), progress has been slow (6, 7) and hampered by a lack of understanding of community-based usual care (8). The limited research to date examining usual care has focused on characterizing the types of treatment techniques used by community providers (9), that is the procedures employed during the intervention process (e.g., role play, relaxation training; 10). While this has helped elucidate the landscape of usual care, little is known about variables that predict the use of specific therapeutic techniques delivered by community therapists.
Studies focused on usual care of disruptive behavior disorders have demonstrated that community therapists self-report using a wide-range of techniques, both evidence-based and not (11). Observational studies have shown that community therapists generally deliver a broad range of techniques at relatively low intensities (12, 13) in the treatment of externalizing youth. Studies that have examined therapist self-report of usual care in youth receiving treatment for internalizing disorders have corroborated the usual care literature on externalizing disorders (e.g., 14, 15). Observational studies by both McLeod & Weisz (16) and Southam-Gerow and colleagues (17) have demonstrated that community therapists also utilize a variety of techniques in their treatment of youth with internalizing disorders, generally favoring client-centered approaches. Weisz and colleagues (18) found that therapists delivering usual care to youth with depression were observed to utilize more family and psychodynamic techniques when compared to cognitive-behavioral therapy (CBT) techniques.
Increasing the use of EBPs in usual care is an important objective for mental health services research. The literature suggests that children receiving community-based usual care do not clinically improve at the same magnitude as EBPs (19-21). In direct comparisons EBPs generally outperform usual care. For example, in a series of meta-analyses by Weisz and colleagues, differences in effect sizes between usual care and EBPs were approximately .29 (Cohen’s d; 21, 22). Youth receiving usual care in publicly-funded systems have worse treatment trajectories than youth in managed-care systems (20). Taken together, these data suggest considerable room for improvement in usual community-care practices.
To date, few studies have examined contextual predictors of use of evidence-based therapy techniques in usual care. Therapist (e.g., knowledge, attitudes) and client (e.g., presenting disorder) variables may be important predictors of whether a particular therapist uses evidence-based techniques in usual care settings (23, 24). In one recent study, therapist factors (i.e., knowledge, attitudes) were found to be more predictive of use of psychodynamic techniques but organizational factors (i.e., culture and climate) were found to be more predictive of use of cognitive-behavioral and family techniques (25). A limitation of the literature to date is that client factors have been largely ignored as predictors of use of therapy techniques. Understanding both therapist and client factors that predict use of therapy techniques is important because it allows for the identification of factors which can be targeted to change therapist behavior.
The present study allows for exploration of predictors of therapist self-report of techniques that fall into cognitive, behavioral, psychodynamic, and family domains within the context of a system-wide effort to improve usual care through implementing EBPs (see 26). The community mental health therapists in the present sample provide care to youth in a large, urban, publicly-funded system, which supports generalizability. Additionally, we add to the literature by utilizing providers in a system implementing multiple EBPs which provides a unique context to examine usual care within. Specifically, in the publicly-funded mental health system in Philadelphia, therapists are receiving training and consultation in a number of EBPs including cognitive therapy (27), trauma-focused CBT (28), prolonged exposure (29), and dialectical behavior-therapy (30). This study examines whether community-based therapists in this system report using cognitive, behavioral, psychodynamic, and family techniques and if therapist- or client-level factors predict use of these techniques. Given the limited data on predictors of usual care practices, this study was exploratory in nature and no a prior hypotheses were made.
The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) in the City of Philadelphia (26) has engaged in pilot CBT implementation initiatives in the public mental health system since 2007. Implementation efforts are supported by a full-time city employee who coordinates the initiative. The training and ongoing consultation provided to therapists closely follows the recommendations of the respective treatment developers and includes post-training consultation. For example, outpatient providers enrolled in the cognitive therapy training participated in 22 hours of didactic workshops followed by six months of weekly group consultation (31). Other initiatives have similarly intensive procedures.
Purposive sampling was used to recruit the 29 largest agencies within the 100+ community mental health agencies in Philadelphia that provide outpatient services to youth (personal communication, Community Behavioral Health, 2012). These agencies together serve approximately 80% of Philadelphia youth receiving publicly-funded mental health care. Of these 29 agencies, 18 (62%) agreed to participate. An additional agency involved in EBP efforts approached us to participate, resulting in a final sample of 19 agencies representing 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators. There were no exclusion criteria for therapist participation and approximately 60% of therapists from the 23 sites participated. Of participating therapists, 123 (95%) completed all measures on one occasion. Of the agencies enrolled, 16 had participated in DBHIDS sponsored EBP initiatives (mean years of participation = 2.90 ± 2.70); of the therapists enrolled, 54 (41%) had participated in DBHIDS sponsored EBP initiatives. Table 1 provides demographic information about therapists and clients.
All procedures were approved by and conducted in compliance with the appropriate Institutional Review Boards. See Beidas et al. (25) for a complete description of the procedures of the larger study within which these data were collected. Participants provided informed consent and were compensated $50 for participation. The principles outlined in the Declaration of Helsinki were followed. Data were collected between March 1st and July 25th, 2013. Therapists were informed that their data would be kept confidential, that only aggregated and de-identified information would be shared with agency leadership and DBHIDS.
Therapist demographics were assessed using the Therapist Background Questionnaire [TBQ; 32], a 21-item questionnaire that obtains information on personal characteristics such as age, gender and licensure status. Additionally, therapists indicated whether they had participated in any of the four DBHIDS EBP initiatives (cognitive therapy, trauma-focused CBT, prolonged exposure, and dialectical behavior therapy) on the Evidence-Based Practices Training Survey. We asked specifically whether they had received training and a year of consultation to ensure that they formally participated in the city-sponsored initiative.
Therapists identified a representative client, defined as a client of their choosing who was most like a typical client on their caseload, and reported on that client’s age, ethnicity, and primary diagnosis. Child primary diagnoses, when necessary, were coded into DSM-IV disorders (e.g., ‘Depression’ was coded as Major Depressive Disorder). To maximize power for statistical analyses, we subsequently coded diagnoses as internalizing, externalizing, and other as there is considerable overlap in the techniques that are evidence-based across the internalizing disorders, and similarly for externalizing disorders (e.g., 33). See Table 1. Therapists then reported on the therapeutic techniques utilized with the representative client they identified.
The Therapist Procedures Checklist-Family Revised [TPC-FR; 15, 34] is a 62-item self-report therapist checklist that assesses therapeutic techniques from the following modalities: cognitive, behavioral, family, and psychodynamic. The TPC-FR informed the development of the Therapy Process Observational Coding System (TPOCS; 16), a gold-standard observer-rated measure of therapy fidelity. The TPC-FR is a revised version of the TPC (34) developed to also query about family therapy techniques (15). Therapists selected a recent representative client about whom they reported, on one occasion, on which techniques they had used with that client over the course of therapy to date. Each subscale is a mean of the items that fit within that factor, measured on a continuum from 1 to 5 with 1 indicating rarely; 2, seldom; 3, sometimes; 4, often; and 5, most of the time. Higher scores indicate greater self-reported use of the set of techniques. Good internal consistency has been reported for the TPC-FR (15) and the factor structure has been confirmed (35). In our sample, subscale α’s ranged from .84-.94.
Four mixed-effects linear regression models examined the relationships of the set of variables with self-reported use of specific therapy techniques (i.e., cognitive, behavioral, psychodynamic, family), as measured by the TPC-FR. Mixed-effects models included random intercepts for organization to account for therapists being nested within organizations. Predictor variables included therapist predictors (age; gender = male or female; clinical experience; licensure status = yes or no; EBP initiative participation = yes or no) and child predictors (primary diagnosis = externalizing, internalizing, or other; age = early childhood, middle childhood or adolescence). These predictor variables were included because treatment recommendations vary somewhat for older vs. younger youth and for those with internalizing vs. externalizing disorders and therapists may have differentially participated in training opportunities in EBPs. Missing data were minimal (<10%). Series means were imputed for missing predictor variables. Analyses were conducted using PROC MIXED in SAS 9.0. Effect sizes (Cohen’s f2) were computed using the procedures outlined by Selya and colleagues (36): .02 represents a small effect, .15 a medium effect and .35 a large effect (37).
Specific therapeutic techniques reported were assessed via the TPC-FR (see Table 2 for model parameters and ICCs). Regarding therapist-level predictors, therapists who had participated in EBP initiatives were more likely to report use of cognitive techniques (f2 = .05) while therapists who were not licensed were more likely to report use of behavioral (f2 = .02) and psychodynamic techniques (f2 = .07). Regarding client-level predictors, therapists working with clients with a primary externalizing disorder were more likely to report use of behavioral (f2 = .13) and family (f2 = .02) techniques than therapists working with youth with primary internalizing diagnoses over the course of therapy. Therapists working with early childhood aged youth were more likely to report use of behavioral techniques (f2 = .08) than those working with adolescents; likewise, therapists working with adolescents were more likely to report use of cognitive (f2 = .05) and psychodynamic (f2 = .05) techniques than those working with youth in early childhood, and more likely to report use of psychodynamic techniques (f2 = .05) than those working with youth in middle childhood.
The structure of the analyses required the use of a reference group for dummy coded variables. Thus, report of techniques used could not be compared between externalizing disorders and other disorders in the above model. When the reference group was changed from primary internalizing disorder to primary other disorder, the results were largely unchanged. Therapists working with clients with a primary externalizing disorder were more likely to report use of behavioral techniques (B = .39, p = .04, f2 = .13) than therapists working with youth with primary other diagnoses over the course of therapy. A significant difference in reported use of family therapy (B = .08, p = .63) was not observed.
This study is the first to examine the relationship between therapist- and client-level predictors on community-based therapists’ self-report of techniques that fall into cognitive, behavioral, psychodynamic, and family domains within the context of implementation efforts. Results indicate that both therapist- and client-level variables predicted self-reported use of therapy techniques. Overall, these findings are largely encouraging and suggest that therapists in this system are self-reporting use of evidence-based usual care practices, such as cognitive and behavioral techniques, in certain circumstances. This may be attributed to the system-wide emphasis on implementation of EBPs; indeed the finding that therapists participating in initiatives were more likely to report use of cognitive techniques strengthens this interpretation. This study provides preliminary support for the idea that community therapists can self-report ascribing evidence-based techniques to client needs.
Therapist factors, namely therapist licensure status and participation in EBP initiatives, predicted self-reported use of therapy techniques. Specifically, when compared to licensed providers, therapists who were unlicensed were more likely to report using behavioral and psychodynamic techniques. Given that we controlled for therapist age and years of experience, it is unlikely that these findings are driven by these factors. Unlicensed providers are supervised by licensed providers, so perhaps supervisors in this system are emphasizing behavioral and psychodynamic techniques with supervisees. Understanding the impact of supervisor support on use of therapy techniques is an important area for future inquiry (38). Therapists who had participated in EBP initiatives were more likely to report use of cognitive techniques. Given that there is a longstanding cognitive therapy training initiative in the system (31), this result is not surprising and suggests a positive impact of this EBP initiative on the system.
Importantly, client-level variables also predicted use of therapy techniques. Therapists working with youth with primary externalizing disorders were more likely to report use of behavioral and family techniques when compared to those working with youth with primary internalizing diagnoses. This is consistent with the evidence-base that suggests that at least some behavioral and family based techniques are more effective for youth with externalizing disorders (39, 40). Also of note, therapists were more likely to report use of cognitive and psychodynamic techniques with adolescents than with younger youth (i.e., in early childhood) and more likely to describe the use of behavioral techniques with younger youth compared to adolescents. Cognitive and psychodynamic techniques both involve insight (41) and therapists may be more comfortable using these techniques with older youth.
Although therapists reported using a range of evidence-based cognitive and behavioral therapy techniques, they also reported using non-evidence based techniques, particularly psychodynamic techniques. This is consistent with previous work showing that therapists tend to be eclectic in their approach to therapy and use a variety of therapy techniques (8, 42). Deimplementaion, or exnovation, refers to an organization divesting itself of a previously adopted innovation (43). This represents an area ripe for future research. As unlicensed providers report using psychodynamic techniques more frequently than licensed providers, unlicensed providers may be a particular group worth targeting for exnovation.
These findings are preliminary because we relied on therapist report for two important variables: use of therapy techniques and client presenting diagnosis. First, studies on the concordance of therapist self-report with observation are equivocal. For example, a study conducted by Hurlburt and colleagues (44) casts doubt upon concordance, but deeper examination of this paper suggests that the demand characteristics of the task differed for observers and therapists, calling into question the applicability of these findings to our work. Specifically, therapists reported on their behavior several days after the session, whereas observers scored every minute of treatment sessions. Furthermore, therapists were not trained in how to make ratings of the intensity of their behavior whereas coders had intensive training and detailed instructions on how to score intensity of behavior. On the other hand, a recent study where therapists, youth, caregivers, trained raters, and treatment experts rated therapist adherence to a substance abuse treatment protocol found high concordance between self-reported therapist ratings and trained raters (which showed high concordance with treatment expert ratings). In this study, the methodology was more similar to ours in that therapists were asked to reflect on the overall techniques used in treatment over a larger period of time (1 month; 45). Other studies corroborate this finding (46). Second, therapists reported on their client diagnoses and we did not independently verify these diagnoses using semi-structured interviews; concordance between therapist diagnosis and semi-structured interviews is low (47). However, for the particular question at hand, it is more important to capture the diagnosis which therapists report they are targeting in treatment, whether or not the diagnosis is consistent with research criteria. Therefore, if therapists are differentially implementing therapy techniques to client clinical presentation we would expect to detect that using the methodology employed herein.
The present study is strengthened by its large and diverse sample of community mental health therapists drawn from a major metropolitan area. Several additional limitations exist. Therapists did not provide information about client severity or comorbidity, though a recent study of children receiving services though the publicly-funded mental health system in Philadelphia suggests low levels of comorbidity as reported in Medicaid claims (i.e., < 5%; 48). Information was obtained about a single client on a therapist’s caseload and treatment duration was not reported. It is not known whether treatment duration is associated with amount of intervention techniques utilized by therapists, and future studies would benefit from examining this empirically. Therapists self-selected which clients to report on. Therapists could have been motivated to report on a client who they deemed to be a good candidate for EBP techniques; however we observed a wide range of presenting disorders and ages, suggesting this may not be a significant limitation. Finally, data regarding the TPC’s ability to predict therapist behavior or distinguish between therapists who use varying levels of specific techniques is lacking at present.
Results suggest that both therapist and client factors predict self-reported use of therapy techniques. Most importantly, participating in an EBP initiative increased self-reported use of cognitive techniques, suggesting that a system-wide effort to implement EBPs can result in individual therapist report of evidence-based technique use. Therapists self-reported differentially using techniques, reporting increased use of behavioral and family techniques with youth with externalizing disorders and less use of cognitive and psychodynamic techniques with young clients. Future studies would benefit from utilizing observational methods to ascertain the veracity of these findings. Additionally, studies examining therapists’ use of EBPs with clients with a range of presenting problems (14) and comorbidity are needed, as are those that examine the role of treatment length and sequencing of EBP techniques in community care.
Dr. X receives royalties from Oxford University Press and has served as a consultant for Kinark Child and Family Services. Dr. Y has received grant support from Ortho-McNeil Janssen and Forest Research Institute and has served as a consultant to AstraZeneca and Alkermes.
This manuscript was supported by National Institute of Mental Health grants to Dr. Z (MH103955) and Dr. X (MH099179). Additionally, the preparation of this article was supported in part by the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25 MH080916) and Quality Enhancement Research Initiative (QUERI), Department of Veterans Affairs Contract, Veterans Health Administration, Office of Research & Development, Health Services Research & Development Service. Dr. X was an IRI fellow from 2012-2014. Drs. X, Y, & Z are fellows of the Leonard Davis Institute of Health Economics, University of [OMITTED FOR BLIND REVIEW].
None of the reported disclosures are related to implementation of evidence-based practices for youth in the City of [OMITTED FOR BLIND REVIEW]. The following authors have no disclosures to report: (Dr. Z; Dr. A, Dr. B; Dr. C; Dr. D).