Mental Health Care for Children with Disruptive Behavior Problems: A view inside therapists’ offices

doi:10.1176/appi.ps.61.8.788

Psychiatr Serv. Author manuscript; available in PMC 2011 January 12.

Published in final edited form as:

Psychiatr Serv. 2010 August; 61(8): 788–795.

doi: 10.1176/appi.ps.61.8.788

PMCID: PMC3019612

NIHMSID: NIHMS181473

Mental Health Care for Children with Disruptive Behavior Problems: A view inside therapists’ offices

Ann F. Garland, Ph.D.,^1,² Lauren Brookman-Frazee, Ph.D.,^1,² Michael Hurlburt, Ph.D.,^1,³ Erin C. Accurso, M.S.,^1,^2,⁴ Rachel Zoffness, M.A.,^1,^2,⁴ Rachel A. Haine, Ph.D.,¹ and William Ganger, M.A.¹

¹Child and Adolescent Services Research Center at Rady Children’s Hospital San Diego

²Department of Psychiatry, University of California, San Diego

³School of Social Work, University of Southern California

⁴Department of Psychology, San Diego State University

Lauren Brookman-Frazee: Lbrookman/at/ucsd.edu; Michael Hurlburt: hurlburt/at/usc.edu; Erin C. Accurso: eaccurso/at/ucsd.edu; Rachel Zoffness: racheljentry/at/gmail.com; Rachel A. Haine: rhaine/at/casrc.org; William Ganger: bganger/at/casrc.org

Corresponding Author: Ann F. Garland, Ph.D., Child and Adolescent Services Research Center, 3020 Children’s Way MC 5033, San Diego, CA 92123, Email: agarland/at/ucsd.edu, Phone: (858) 966-7703 ext. 3756, Fax: (858) 966-7704

The publisher's final edited version of this article is available at Psychiatr Serv

See other articles in PMC that cite the published article.

Abstract

Objectives

In the United States, more money is spent on treatment for children’s mental health problems than for any other childhood medical condition, yet little is known about usual care (UC) treatment for children. Objectives of this study were to a) characterize UC out-patient psychotherapy for children with disruptive behavior problems, and b) identify consistencies and inconsistencies between UC and common elements of evidence-based practices to inform efforts to implement evidence-based practices in UC.

Methods

Participants included 96 psychotherapists and 191 children ages 4–13 presenting for treatment for disruptive behavior to one of six UC clinics. An adapted version of the Therapy Process Observational Coding System for Child Psychotherapy – Strategies scale (TPOCS-S) was used to assess psychotherapy processes in 1215 randomly selected (out of 3241 collected) videotaped treatment sessions for up to 16 months.

Results

Most children received a lot of treatment (mean number of sessions=22, plus other auxiliary services), and there was great variability in amount and type of care received. Therapists employed a wide array of treatment strategies directed to children and parents within and across sessions, but all strategies were delivered at low average intensity. Several strategies conceptually consistent with evidence-based practices were observed frequently (e.g., affect education, positive reinforcement); however, others were observed rarely (e.g., assigning/reviewing homework, role-playing).

Conclusion

UC treatment for these youths reflected great breadth but not depth. The results highlight specific discrepancies between evidence-based care and UC, thus identifying potentially potent targets for improving the effectiveness of UC.

Introduction

More money is spent on treatment for mental illness in children in the United States than for any other childhood medical condition (1). Unfortunately, outcome data on the effectiveness of community-based psychotherapeutic care are discouraging (2–4), and virtually nothing is known about what types of psychotherapeutic care are delivered in usual care (UC) settings, making it difficult to know how to target care improvement (5–7). National research and policy initiatives call for dissemination and implementation of evidence-based (EB) practices in UC (8). These efforts could be more efficient and sustainable if informed by a better understanding of the current care context (9,10), i.e., “it is difficult and perhaps foolhardy to try to improve what you do not understand” (6).

Studies of children’s psychotherapeutic UC have focused primarily on examining outcomes as opposed to treatment processes. On average, findings regarding the effectiveness of UC youth psychotherapy reflect minimal impact on children’s symptom severity or functional status (2–4). However, none of these studies included detailed descriptive data about the nature of the UC treatment; therefore, attributions regarding the links between UC treatment processes and outcomes are largely speculative.

In one of the only studies of the nature of UC for children receiving publicly-funded mental health services, Zima and colleagues used a chart review method to assess quality of care (12). Approximately half of the 813 cases studied met broad quality of care indicators for psychosocial treatment. However, the authors acknowledged that chart review was not a good data source for details about psychotherapy practice. Characterizing psychotherapeutic treatment processes is methodologically complex (11). Treatment can be characterized across a continuum ranging from broad molar classifications (e.g., theoretical orientations) to molecular level detailed verbal and non-verbal behaviors (13). “Clinical strategies” represent an intermediate level of analysis and include therapeutic techniques (e.g., modeling of skills, interpreting meaning of behavior) and content (e.g., problem solving skills, family member roles) (14). The most objective method to assess clinical strategies in psychotherapy is direct observation, but it is labor-intensive (11).

This study utilizes observational assessment to assess clinical strategies delivered in publicly-funded UC to children with disruptive behavior problems (DBPs) and their parents. This patient population was selected because the vast majority of youths in publicly-funded care are referred for disruptive problems, including oppositional, defiant, aggressive, and/or delinquent behavior (15). Effective treatment is essential because children with DBPs are at significantly elevated risk for multiple maladaptive outcomes in adolescence and adulthood, including criminal behavior and psychopathology (16,17). Further, families served in the public sector are at particularly high risk due to multiple risk factors and life stressors (12).

Many psychotherapy treatment models have demonstrated efficacy for children with DBPs (for reviews see 18–21) and several common core elements across individual EB treatment models have been identified (22,23). Common elements of EB treatment for this patient population include clinical strategies directed toward children (e.g., affect/anger management) and parents/caregivers (hereafter referred to as parents) (e.g., principles of positive reinforcement and limit-setting). It is assumed that delivery of complete EB treatment models is rare in UC (21); however, no studies have assessed the extent to which UC includes treatment strategies generally consistent with common elements of EB practice. Empirical data that identify specific convergence and divergence between elements of EB practice and UC are needed to provide a “road map” for targeted efforts to improve care.

The goals of this study are to a) provide a detailed description of UC psychotherapy for a representative sample of children with DBPs treated in publicly-funded out-patient clinics, and b) examine the extent to which therapists employ strategies conceptually consistent with common elements of EB practices to inform efforts to implement evidence-based practices in UC.

Methods

Participants

Participating Clinics

The six participating clinics were selected because they represented the largest contractors for publicly-funded, clinic-based out-patient care for children in one of the largest counties in the U.S. serving ethnically and diagnostically diverse children and their families. None of the clinics specialized nor provided therapist training in delivery or incorporation of EB treatments before or during the study period.

Therapist Participants

In late 2003, all therapists from participating clinics were assigned random numbers and recruited sequentially to fill cells reflecting the county’s therapist distribution by discipline and staff/trainee status. From 2004 to 2006, all new therapists working at least halftime were recruited. The sample includes a large representation of trainees and early career therapists given greater turnover rates for these therapists. Of the 163 therapists recruited, 131 (80%) agreed to participate; only 96 had a patient in the study who met criteria, agreed to participate, and attended sessions for video data collection. Therapists who declined to participate did not differ significantly from participant therapists on age, gender, or race/ethnic distribution; licensed staff had a slightly lower participation rate (72%) compared to unlicensed staff/trainees (86%). Characteristics of the 96 therapists are presented in Table 1.

Table 1

Characteristics of Participating Therapists (n = 96).

Child and Parent Participants

Inclusion criteria for child participants were a) presenting problems included a disruptive behavior problem (aggression, defiance, delinquency, oppositional behavior by parent report), b) age 4–13 years, c) primary language for child and parent was English or Spanish, and d) child was entering a new episode of psychotherapy (defined as no therapy for previous three months) with a participating therapist. Clinic administrative staff screened parents who contacted the clinic seeking an appointment for permission to share contact information with the researchers for recruitment; 10% declined to be contacted. Of the 550 who agreed to be contacted and met inclusion criteria, 292 (55%) did not attend a clinic appointment, leaving 258 potential participants; 218 (85%) agreed to participate. Videotape data were collected for 191 children (27 participants did not continue treatment after the baseline research interview). HIPAA restrictions prohibit examination of how patient non-participants may have differed from participants. Characteristics of the child and parent participants are presented in Table 2.

Table 2

Child and Family Participant Characteristics.

Informed written consent was provided by therapists and parents and assent was provided by children ages 8–13. Participants were compensated for research participation and all protocols were approved by affiliated human subjects review committees.

Procedures

Descriptive data on children, parents, and therapists were collected during baseline interviews. Number of treatment sessions attended and clinician-assigned child psychiatric diagnoses (grouped into 6 categories listed in Table 2) were collected from billing and administrative records. Data on other treatment services received were obtained from telephone interviews with parents conducted every four months for 16 months.

All psychotherapy sessions occurring between the baseline research interview and 16-month follow-up were videotaped; 3241 videotaped sessions were collected. A random sample of up to 10 sessions per child was selected for coding (four tapes from within the 0–4 month interval, three tapes at 5–8 months, two tapes at 9–12 months, and one tape at 13–16 months, to provide the most data during the intervals in which most patients were attending), resulting in 1215 total coded sessions.

Measures

Adaptation of Therapeutic Process Observational Coding System for Child Psychotherapy – Strategies scale (TPOCS-S)

The TPOCS-S (24, University of California, Los Angeles, 2001; 25) assesses for a wide array of intervention strategies that are theoretically and non-theoretically, or cross-theoretically, derived. The content was adapted from the Therapy Procedures Checklist (TPC) (27) and the format is based on the Therapist Behavior Rating Scale (TBRS) (28). The TPOCS-S was adapted for this study in collaboration with the project’s Therapist Advisory Group (11,29) to maximize relevance to community practice. The final revised PRAC TPOCS-S includes 27 clinical strategies (listed in Table 3), divided into 15 therapist techniques (e.g., modeling, addressing client-therapist relationship) and 12 therapeutic content areas (e.g., affect management, principles of positive reinforcement) (26, Child and Adolescent Services Research Center, 2008). Occurrence and intensity of each strategy are coded as directed to children or parents (or both). Occurrence indicates whether the strategy was observed at all. Intensity reflects both the time spent on the strategy and the thoroughness with which it was pursued (28). Intensity was rated at the end of the session for each observed strategy on a Likert scale of 1 to 6 (1–2=low, 3–4=medium, 5–6=high). For example, a low intensity rating on the content strategy “problem-solving skills” would reflect addressing one aspect of problem-solving skills, such as generating alternative solutions, but only for one particular experience the child or parent faced, and in a somewhat fleeting or cursory manner. High intensity would reflect a thorough approach addressing multiple steps in problem solving and generalization to other problems.

Table 3

Occurrence and Intensity of Therapeutic Strategies Observed (total number of sessions=1215).

Among the PRAC-TPOCS-S 27 strategies, a subset has been previously identified as common elements of EB treatment for children with DBPs and these are indicated in bold text in Table 3 (23). This subset was identified through an iterative process of culling out elements of established empirically-supported youth and/or parent-training treatment models, identifying elements common across treatments, and validating through expert consensus.

Coders and Coder Training

Seventeen research assistants served as coders (three were fluent in Spanish and coded sessions conducted in Spanish). Coder training was conducted by three of the authors and included didactics, manual review, practice sessions, and training to criterion (11).

Inter-rater Reliability of PRAC-TPOCS-S

Of the 1215 total coded sessions, 379 (31%) were randomly selected for double-coding to test inter-rater reliability. The ICC assessing reliability on the full occurrence/intensity scale (0–6, incorporating “0’s” for “not observed”) across all codes at the session level was .78, representing strong reliability. ICCs were also calculated at the individual code level and ranged from .21 to .91, with a mean of .61, which is within an acceptable range (30). The eight codes with very low occurrence (observed < 10% of sessions) had the lowest reliability.

Data Analyses

Descriptive data on frequencies of observed clinical strategies across all 1215 coded sessions are presented in Table 3. Therapists varied in the number of cases (mean=2.5±1.8, range=1–7) and sessions (mean=12.7±11.6, range=1–44) coded; however, the observed frequencies do not vary significantly (all are within three points and the majority are within two points) when just one case per therapist is randomly selected. Thus, analyses presented rely on the full sample of 1215 sessions.

Results

Treatment Received

Participants attended an average of 22.4±15.6 sessions (range=0–70) during the 16-month study period, with 54 participants (28%) still attending at 16 months. Across the 1215 coded sessions, children participated in 1184 sessions (97%) and parents participated in at least part of 851 sessions (70%). By parent report, children received a variety of services in addition to out-patient psychotherapy. Specifically, during the 16-month study period, 134 (63%) of the children received some type of psychoactive medication, 30 (15%) presented to emergency rooms for mental health/behavioral reasons, 19 (9%) were placed in a psychiatric hospital or residential treatment center, and 177 (88%) received school-based psycho-educational services.

Psychotherapeutic Treatment Strategies Observed Within Treatment Sessions

The average number of treatment strategies observed per session directed to children was 10.9±3.7 (range=1–20 of 27) and to parents was 8.1±3.9 (range=1–22 of 27). Table 3 presents data on the observed occurrence and intensity for each of the assessed strategies in rank order of occurrence frequency. “Any Occurrence” represents the percentage of sessions in which the strategy was observed relative to the total number of sessions in which each target (child or parent) participated (e.g., affect education was observed in 964 (81%) of the 1184 sessions in which a child participated). “High Intensity Occurrence” represents the percentage of sessions in which the strategy was observed at an intensity level of 5 or 6 on the 6-point scale. “Mean Intensity” reports the average intensity of the strategy when it was observed (i.e., range of 1–6). Mean intensity when observed was 2.3±.3 across strategies directed to children and 2.4±.5 for parent strategies. Of all observed sessions, 724 (59.6%) included at least one strategy delivered to either child or parent(s) at high intensity.

Some therapeutic strategies previously identified as common to EB treatment for children with DBPs and their parents (23) (in bold text in Table 3) were observed frequently (e.g., affect education, use of positive reinforcement), whereas others were observed relatively infrequently (e.g., assigning/reviewing homework, role-playing with parents). All elements were observed at low average intensity.

Discussion

This study provides the first detailed data on the type and variability of psychotherapeutic treatment strategies observed in mental health care delivered to children with disruptive behavior problems, the most common presenting problems in mental health. Results reflect heterogeneity in amount and type of treatment, including treatment duration (0–70 sessions in 16 months), additional service use, and within-session psychotherapeutic strategies observed.

Although treatment duration varied widely, the mean number of sessions attended (22.4) is somewhat consistent with limited available data on duration of community-based care, with reported total treatment session averages ranging from 17 (31) to 23 sessions (32). Most EB treatment models for children with DBPs require a minimum of 12 weekly visits (23). In addition to out-patient sessions, the majority of patients reportedly received other types of mental health services. Almost two-thirds of these 4 to 13 year olds received some psychoactive medication, which stands in contrast to 38% of a somewhat similar sample of youths with DBPs receiving UC treatment in the Netherlands (33). The majority of children in our sample also received some type of school-based psycho-educational services, and a minority received more intensive services, including hospitalization for psychiatric reasons (9%), during the 16-month study period. Combined with the fact that 28% of children were still in treatment after 16 months, these data reflect a substantial volume of mental health services for families who engaged in UC. The extent to which these services were coordinated across treatment modalities is not known.

Within and across treatment sessions, therapists were observed delivering a wide array of clinical strategies, which is consistent with limited research indicating that UC therapists prefer an eclectic approach to psychotherapy (10,14). Although only 26% of the therapists in this study endorsed “Eclectic” as their primary theoretical orientation, the observational data indicate that eclecticism is the norm rather than the exception. The findings that almost half of sessions did not include a single therapeutic strategy delivered at high intensity, and that the mean intensity rating across all observed strategies was relatively low reflect a cursory and/or incomplete application of the treatment strategy with limited follow-through. Explanations for low intensity are largely speculative but may reflect variability in therapist training, interference of patient crises and case management challenges, assumptions or perceptions of patients’ responsiveness, and/or pragmatic constraints (e.g., not enough time). Overall, the UC psychotherapy we observed could be characterized as reflecting great breadth, but not depth, in therapeutic approaches.

Parents participated in at least part of the majority of treatment sessions, and similar therapeutic techniques were observed frequently being delivered to both children and parents (e.g., assessing problems/events, psychoeducation, establishing/reviewing goals). However, therapists were observed addressing different therapeutic content areas with parents vs. children. Specifically, two of the most common content areas addressed with parents were “child’s external care” (i.e., case management or coordination of extra-therapeutic services) and parental psychosocial issues, but these were not common with children. These findings are consistent with anecdotal reports from UC therapists who indicate that the complex, multi-determined needs of families in public sector care require significant case management, which can interfere with delivery of EB psychotherapeutic approaches such as child skill-building or parent management training. While our data do confirm that therapists are spending a great deal of time with parents on case management, the data cannot address the necessity of this case management emphasis, nor the impact on outcomes.

The second aim of this study was to examine the extent to which UC therapists employed strategies conceptually consistent with pre-determined common elements of EB treatment for this patient population. Results indicate that while several strategies common to EB practices were observed in a majority of sessions (e.g., affect education, problem solving skills, use of positive reinforcement, psychoeducation), other strategies common in EB treatment were observed relatively infrequently. Even when observed, several EB strategies were usually rated at low intensity, thus not consistent with expectations in EB treatment models. Similar observations were reported in other work comparing observational and therapist self-report methods for characterizing psychotherapy practice (34).

Many of the infrequently observed EB treatment strategies, such as assigning/reviewing homework, role-play/behavioral rehearsal, and modeling, are characterized as more directive psychotherapeutic techniques (35), and these strategies are at the core of virtually all EB treatment models for children with DBPs. We have found that therapists in UC clinics generally have positive attitudes about many of these psychotherapeutic techniques (36), yet they do not employ them often. Research on adult psychotherapy similarly finds that directive therapeutic approaches are not observed as frequently in UC compared to EB treatment models (35). Given that more directive treatment approaches have been associated with greater improvement in specific behavioral outcomes (37), more attention to this discrepancy is warranted.

Our findings raise natural questions about factors associated with therapist variability in practice. In related research, we have tested for therapist characteristics that may be associated with delivery of treatment relatively more or less consistent with elements of EB practice; few significant effects were found (38). For example, therapist experience (months practiced), discipline, and staff versus trainee status have not been significantly associated with observed intensity of delivery of EB practice elements in our sample of therapists, which includes many therapists with few years of accumulated experience. Thus, explanations for variability in practice likely require more complex investigation of training and supervision experiences. Accordingly, our results have implications for therapist training (as discussed below).

The current study findings need to be interpreted in the context of some additional strengths and limitations of the study. Representativeness of the therapist and patient sample is critical for the generalizability of these findings. The patient sample is comparable to several other clinical samples of children in publicly-funded care. Specifically, the 2:1 male to female ratio is consistent across many studies (12,20), as is the over-representation of some racial/ethnic minority youth relative to general population estimates (12,39), and the most common diagnoses (conduct disorder/ODD, ADHD, and mood disorders) (39,40). The therapist sample is also very comparable to a recent national sample of 1200 providers in children’s mental health care in terms of distribution by educational level, gender, and ethnicity (41). Trainees with limited experience are somewhat over-represented in our study, but other studies of community-based UC also report high representation of trainees (42). Our sample also includes a large representation of therapists trained in the Marriage & Family Therapy (MFT) discipline, which represents a rapidly growing segment of the workforce across the U.S., but is over-represented in California (43). Of course, the extent to which the participating therapists represent therapists in other types of service sectors (e.g., private practice) or geographic regions is unknown.

The strengths of the study methods reflect a balance of relevance and rigor. We achieved adequate inter-rater reliability on our observational measure (PRAC-TPOCS-S), which also benefited from collaborative input from practicing therapists to assure ecological validity and comprehensiveness (29). Use of such qualitative methodology to support the validity of practice measures has been strongly reinforced in previous studies (10). Despite these strengths, the resulting measure did not capture all possible therapeutic interventions. The data reflect only the observable behavior of the therapist, and do not capture therapists’ intentions, goals, or decision-making processes, patients’ responses to different intervention strategies, or additional therapeutic contacts such as communication outside the office. Finally, we do not know how observing practice may have impacted practice itself, although we attempted to minimize this effect by establishing videotaping as routine in the clinics and by using small, unobtrusive cameras.

Conclusions

In the context of recent reports of the extraordinarily high cost of mental health treatment for children and discouraging data on effectiveness, this study offers timely detail regarding the types of care being delivered in one large public system. The study provides the first glimpse into UC psychotherapy practice offices, and thus provides essential contextual data for the development of tailored efforts to improve care, as well as providing baseline data for change efforts. Our results highlight some areas of relative convergence between EB practice and UC, as well as significant discrepancies between EB treatment elements and UC. Areas of convergence may represent “common ground” upon which to build and discrepancies represent potentially potent targets for improving care. For example, training efforts designed to increase the delivery of infrequently observed directive EB treatment strategies for children and parents, such as role-playing, modeling, and assigning/reviewing homework, are needed. These efforts must also address needed improvements in intensity of treatment. Mental health clinician training is most effective when training interventions are tailored to address the existing service context (44). Further, it may be important to infuse EB training into graduate programs preparing master’s level therapists since they represent a significant proportion of the providers of community-based mental health services.

Critical next steps in this practice-based research program include examination of UC child/family outcome trajectories to determine how specific practice patterns may be associated with different clinical outcomes. More detailed analyses can examine potential moderating effects of patient characteristics (demographic and/or clinical factors), as well as therapist characteristics. Analyses of the mediating role of perceived therapeutic alliance are also needed as previous research supports the role of alliance in outcomes. Richer information about UC practice will provide essential contextual data for ongoing efforts to improve the translation of EB interventions into practice, balancing the emphasis on evidence-based practice with attention to practice-based evidence.

Acknowledgements

This work was supported by NIH R01MH66070 (A.G.), K23MH077584 (L.B.F.) and K01MH064079 (M.H.). The authors thank Scott Roesch, Ph.D. for data analysis assistance, as well as Deb Dupuis, M.P.H. and Robin Taylor for project management. In addition, this project could not have been completed without significant contributions from the Therapist Advisory Group and all participating therapists and families.

References

1. Soni A. Statistical Brief #242. Rockville, MD: Agency for Healthcare Research and Quality; 2009. Apr, The five most costly children’s conditions, 2006: estimates for the U.S. civilian noninstitutionalized children, ages 0-17. Available at http://www.meps.ahrq.gov/mepsweb/data_files/publications/st242/stat242.pdf.

2. Bickman L, Noser K, Summerfelt WT. Long-term effects of a system of care on children and adolescents. Journal of Behavioral Health Services and Research. 1999;26:185–202. [PubMed]

3. Weiss B, Catron T, Harris V, Phung TM. The effectiveness of traditional child psychotherapy. Journal of Consulting and Clinical Psychology. 1999;67:82–94. [PubMed]

4. Weisz JR, Weiss B, Han SS, et al. Effects of psychotherapy with children and adolescents revisited: a meta-analysis of treatment outcome studies. Psychological Bulletin. 1995;117:450–468. [PubMed]

5. Bickman L. The most dangerous and difficult question in mental health services research. Mental Health Services Research. 2000;2:71–72.

6. Hoagwood K, Kolko DJ. Introduction to the special section on practice contexts: a glimpse into the nether world of public mental health services for children and families. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36:35–36. [PubMed]

7. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care: a meta-analysis of direct comparisons. American Psychologist. 2006;61:671–689. [PubMed]

8. National Advisory Mental Health Council. Translating behavioral science into action: a report by the National Advisory Mental Health Council’s behavioral science workgroup. Bethesda, MD: National Institutes of Health/National Institute of Mental Health; 2000.

9. Westfall JM, Mold J, Fagnan L. Practice-Based Research—"Blue Highways" on the NIH Roadmap. Journal of the American Medical Association. 2007;297:403–406. [PubMed]

10. Baumann BL, Kolko DJ, Collins K, et al. Understanding practitioners' characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse and Neglect. 2006;30:771–787. [PubMed]

11. Garland AF, Hurlburt M, Brookman-Frazee L, et al. Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research. 2009 September; published on-line first. [PMC free article] [PubMed]

12. Zima BT, Hurlburt MS, Knapp P, et al. Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. Journal of the American Academy of Child and Adolescent Psychiatry. 2005;44:130–144. [PubMed]

13. Heaton KJ, Hill CE, Edwards LA. Comparing molecular and molar methods of judging therapist techniques. Psychotherapy Research. 1995;5:141–153.

14. Goldfried MR. Toward the delineation of therapeutic change principles. American Psychologist. 1980;35:991–999. [PubMed]

15. Garland AF, Hough R, McCabe K, et al. Prevalence of psychiatric disorders for youths in public sectors of care. Journal of the American Academy of Child and Adolescent Psychiatry. 2001;40:409–418. [PubMed]

16. Copeland WE, Miller-Johnson S, Keeler G, et al. Childhood psychiatric disorders and young adult crime: a prospective, population-based study. American Journal of Psychiatry. 2007;164:1668–1675. [PubMed]

17. Earls FJ. Violence and today's youth. Future Child. 1994;4:4–23. [PubMed]

18. Burns BJ, Hoagwood K, Mrazek PJ. Effective treatment for mental disorders in children and adolescents. Clinical Child and Family Psychology Review. 1999;2:199–254. [PubMed]

19. Carr A. What works with children and adolescents: a critical review of psychological interventions with children, adolescents and their families. Florence, KY: Taylor & Frances; 2000.

20. Eyberg SM, Nelson MM, Boggs SR. Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. Journal of Clinical Child and Adolescent Psychology. 2008;37:215–237. [PubMed]

21. Kazdin AE, Weisz JR, editors. Evidence-Based Psychotherapies for Children and Adolescents. New York, NY: Guilford Press; 2003.

22. Chorpita BF, Becker KD, Daleiden EL. Understanding the common elements of evidence-based practice: misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry. 2007;46:647–652. [PubMed]

23. Garland AF, Hawley KM, Brookman-Frazee L, et al. Identifying common elements of evidence-based psychosocial treatments for children's disruptive behavior problems. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47:505–514. [PubMed]

24. McLeod BD. The Therapy Process Observational Coding System for Child Psychotherapy. Los Angeles: University of California; 2001. unpublished instrument.

25. McLeod BD, Weisz JR. The Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale. Journal of Clinical Child and Adolescent Psychology. in press. [PubMed]

26. Garland AF, Brookman-Frazee L, McLeod BD. Scoring manual for the PRAC study therapy process observational coding system for child psychotherapy: strategies scale. San Diego: Child and Adolescent Services Research Center; 2008. unpublished coding manual.

27. Weersing VR, Weisz JR, Donenberg GR. Development of the Therapy Procedures Checklist: a therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child and Adolescent Psychology. 2002;31:168–180. [PubMed]

28. Hogue A, Liddle HA, Rowe C. Treatment adherence process research in family therapy: a rationale and some practical guidelines. Psychotherapy. 1996;33:332–345.

29. Garland AF, Plemmons D, Koontz L. Research-practice partnerships in mental health: lessons from participants. Administration and Policy in Mental Health and Mental Health Services Research. 2006;33:517–528. [PubMed]

30. Cichetti DV. Guidelines, criteria, and rule of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment. 1994;6:284–290.

31. Brannan AM, Heflinger CA. Child behavioral health service use and caregiver strain: comparison of managed care and fee-for-service Medicaid systems. Mental Health Services Research. 2005;7:197–211. [PubMed]

32. Hawley KM, Weisz JR. Youth versus parent working alliance in usual clinical care: distinctive associations with retention, satisfaction, and treatment outcome. Journal of Clinical Child and Adolescent Psychology. 2005;34:117–128. [PubMed]

33. van de Wiel NM, Matthys W, Cohen-Kettenis PT, et al. The effectiveness of an experimental treatment when compared to care as usual depends on the type of care as usual. Behavior Modification. 2007;31:298–312. [PubMed]

34. Hurlburt MS, Garland AF, Nguyen K, et al. Child and family therapy process: concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research. 2009 November; published on-line first. [PMC free article] [PubMed]

35. Malik ML, Beutler LE, Alimohamed S, et al. Are all cognitive therapies alike?: a comparison of cognitive and noncognitive therapy process and implications for the application of empirically supported treatments. Journal of Consulting and Clinical Psychology. 2003;71:150–158. [PubMed]

36. Brookman-Frazee L, Garland AF, Taylor R, et al. Therapists’ attitudes towards psychotherapeutic strategies in community-based psychotherapy with children with disruptive behavior problems. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36:1–12. [PMC free article] [PubMed]

37. Schoenwald SK, Henggeler SW, Brondino MJ, et al. Multisystemic therapy: monitoring treatment fidelity. Family Process. 2000;39:83–102. [PubMed]

38. Brookman-Frazee L, Haine A, Baker-Ericzen M, et al. Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research. 2009 October; published on-line first. [PMC free article] [PubMed]

39. Foster EM, Kelsch CC, Kamradt B, et al. Expenditures and sustainability in systems of care. Journal of Emotional and Behavioral Disorders. 2001;9:53–70.

40. Rosenblatt A, Rosenblatt J. Demographic, clinical, and functional characteristics of youth enrolled in six California systems of care. Journal of Child and Family Studies. 2000;9:51–66.

41. Glisson C, Landsverk J, Schoenwald S, et al. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35:98–113. [PubMed]

42. Hawley KM, Weisz JR. Youth versus parent working alliance in usual clinical care: distinctive associations with retention, satisfaction, and treatment outcome. Journal of Clinical Child and Adolescent Psychology. 2005;34:117–128. [PubMed]

43. Northey WF. Characteristics and clinical practices of marriage and family therapists: a national survey. Journal of Marital and Family Therapy. 2002;28:487–494. [PubMed]

44. Casper ES. The theory of planned behavior applied to continuing education for mental health professionals. Psychiatric Services. 2007;58:1324–1329. [PubMed]