PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Adm Policy Ment Health. Author manuscript; available in PMC 2010 November 1.
Published in final edited form as:
PMCID: PMC2783722
NIHMSID: NIHMS146303

Trends in Characteristics of Children Served by the Children’s Mental Health Initiative: 1994–2007

Abstract

Data from 14 years of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program were used to understand the trends of the emotional and behavioral problems and demographic characteristics of children entering services. The data for this study were derived from information collected at intake into service in 90 sites who received their initial federal funding between 1993 and 2004. The findings from this study suggest children entering services later in a site’s funding cycle had lower levels of behavioral problems and children served in sites funded later in the 14 year period had higher levels of behavioral problems. Females have consistently entered services with more severe problems and children referred from non-mental health sources, younger children, and those from non-white racial/ethnic backgrounds have entered system of care services with less severe problems. The policy and programming implications, as well as implications for local system of care program development and implementation are discussed.

Keywords: children’s mental health, CMHI, serious emotional disturbance, service trajectories, systems of care

The prevention and treatment of children’s mental health problems remains a national priority. In the United States an estimated 5% of children experience severe emotional or behavioral difficulties in any 6-month time period (CMHS, 2006), yet many do not receive treatment (Briggs-Gowan, Horowitz, Schwab-Stone, Leventhal, & Leaf, 2000; Burns, Costello, Angold, Tweed, Stangl, Farmer, et al., 1995; Costello, Burns, Costello, Edelbrock, Dulcan, & Brent, 1988; Leaf, Algeria, Cohen, Goodman, Horwitz, Hoven et al., 1996). Progress has been made over the past thirty years in raising awareness regarding gaps and limitations in the way that children’s mental health services are provided (Joint Commission on the Mental Health of Children, 1969; Knitzer, 1982; New Freedom Commission on Mental Health, 2003; USDHHS, 1999), with concentrated effort on the reduction of youth hospitalizations and the provision of community-based care (Pumariega, Winters, & Huffine, 2003). Despite this, there remains a great deal of work to do with 50% of the Healthy People 2010 mental health objectives demonstrating movement away from target at the point of midcourse review (USDHHS, 2006).

A system of care approach to the delivery of services to children with serious emotional disturbance has become common in the last 25 years with multiple and consistent federal investments in the development and implementation of community-based service infrastructures (Pires, 2002). In a period of increasingly limited resources it is important to consider whether the investment of federal dollars is being used to serve children with substantial need. A better understanding of the demographic and clinical characteristics of children served by federally funded programs can inform program policies and practices associated with service provision to children with high needs, as well as contribute to our current understanding of mental health disparities in service receipt.

Historical Development of Children’s Mental Health Systems of Care

The initiation and growth of systems of care has its origin in the Child and Adolescent Service System Program (CASSP). The National Institute of Mental Health (NIMH) initiated the CASSP in the early 1980s in an effort to provide funds and technical assistance to jurisdictions across the United States (Pires, 2002). The funds were used for the initial planning of systems of care with an emphasis on interagency collaboration across the child serving sectors. Since being codified by Stroul and Friedman in 1986 and revised in 1994 (Stroul & Friedman, 1986; Stroul & Friedman, 1994), there has been a substantial increase in the numbers of programs and strategies incorporating, or being built around, the principles that children’s mental health services should be comprehensive, community-based, child-centered, family-focused, and culturally competent (Stroul & Friedman, 1986). Given that children’s mental health needs are complex and often addressed in multiple service sectors, interagency collaboration remains an integral part of a system of care. Although many communities have recognized the need to incorporate these principles into children’s mental health services, historically a lack of resources have impeded efforts to develop and transform service delivery systems.

In response, the Comprehensive Community Mental Health Services for Children and Their Families Program (referred to as the Children’s Mental Health Initiative [CMHI]), administered by the Substance Abuse and Mental Health Services Administration, was initiated in 1993 to provide funds for communities to develop and enhance systems of care to treat and support children and youth with serious emotional disturbance (Stroul & Friedman, 1994). These funds were authorized to advance and support the system of care service delivery model as a best practice approach to providing care to children with serious emotional disturbance and their families.

Community sites were selected for funding based on their need for, and proposed ability to develop, an interagency structure and process of service provision that included, at a minimum, representatives from the major child- and family-serving agencies (i.e., mental health, juvenile justice, child welfare, education, and physical health). The grant guidance for applicants required that services provided should include diagnosis and evaluation; case management; outpatient individual, group, and family counseling; medication management; professional consultation; 24-hour emergency; intensive home-based; intensive day treatment; respite; therapeutic foster care; and transition-to-adult. Grant guidelines also required involvement of families in the governance of the grant, and applicants were required to have a local family support organization in place or to be affiliated with a statewide family network organization that had the potential to create a local organization. Funds were typically awarded to a lead agency directly (most often a community mental health agency) or indirectly through a state- or county-level agency (e.g., a state or county mental health agency). The local lead agency worked in coordination with local juvenile justice agencies (often a juvenile court or probation office), child protective service agencies, school districts (through school-based mental health service providers, etc.), community mental health (if they were not the lead agency), and others. Often service delivery was coordinated across agencies through the use of intensive case management, while in some communities various agencies chose to co-locate and cross-train staff for service delivery in multiple contexts.

Children served by the grant program had to meet the criteria for serious emotional disturbance, which typically required that a child had a diagnosable DSM–IV disorder, exhibited functional impairments, and was at risk for out-of-home placement (CMHS, 1997; 1998; 1999; 2000; 2001). In addition, in 1994 a congressionally mandated national evaluation of the CMHI began which has resulted in extensive and systematic data collection on all children referred to the CMHI since the program’s inception. It could be argued that the longevity and consistency of this funding source have augmented the existing system of care movement and solidified its standing as the prevailing mental health service delivery model for children and their families. Related principle vernacular and operational definitions (and in some instances federal funding) have been integrated across service sectors (e.g., juvenile justice and child welfare systems of care), targeted to specific populations (e.g., Native American/Alaskan Native), and promoted to address specific presenting problems (e.g., individuals with substance abuse problems).

The CMHI is the largest and longest continually funded federal children’s mental health services demonstration program to date, and the national evaluation of the program is the largest existing source of data on such an initiative. As of the start of fiscal year 2009, the federal government has committed over $1 billion to the CMHI, which in turn has funded 144 communities; and it is estimated that these communities have served nearly 90,000 children across the United States and its territories. Although usually thought of as a uniform strategy for service delivery and system reform, we do not fully understand whether the characteristics of children and adolescents served in the CMHI have changed since its inception, whether the services and collaborative efforts have changed, nor whether these changes have incorporated improvements both in treatment and our capacity to develop integrated programs of services for youth requiring multiple agency involvement.

A recently released article comparing the characteristics of youth served in funded systems of care to the population of youth in the geographic catchment areas (Miech et al., 2008) suggests that the CMHI is successfully reaching disadvantaged youth. To this end, it is important that our trend monitoring include aspects relevant to assessing disparities in children’s receipt of mental health services (New Freedom Commission on Mental Health, 2003). Specifically, understanding the racial/ethnic, gender, age, and problem behavior distributions of children entering the program over time and whether changes have occurred in the characteristics of children utilizing services through this system of care program are critically important factors for influencing CMHI policy and programming, furthering our knowledge base related to the CMHI as an infrastructure to address mental health disparities, and informing community implementation of future system of care models.

This study uses data from the national evaluation of the CMHI to examine trends in the demographic characteristics and emotional and behavioral problems of children referred to the CMHI from 1994–2007. Mandated by congress in 1994, the national evaluation of the CMHI has resulted in extensive and systematic data collection on all children referred to the CMHI since the program’s inception. From the start, the national evaluation design has included the following core components: 1) an assessment of system development, 2) a cross-sectional descriptive study, 3) a longitudinal outcome study, and 4) an assessment of services and costs. The two components relevant to the current study (cross-sectional descriptive and longitudinal outcome studies) are described below, and additional information on the remaining components is available elsewhere (CMHS, 1996; 1997; 1998; 1999; 2000; 2001; 2003; 2004; Holden, Friedman & Santiago, 2001). The cross-sectional descriptive study describes the children enrolled in the systems of care in terms of their demographics, functional status, living arrangements, diagnoses, risk factors, and mental health service histories. Family demographics, socioeconomic status, and composition are also described. The longitudinal outcome study examines how child clinical and functional status, family life, service experience, and satisfaction with services change over time for a subsample of families enrolled in the cross-sectional descriptive study. Outcomes data are collected at intake and at subsequent follow-up periods to assess change over time in the following areas: behavioral and emotional problems, child strengths, social functioning, substance abuse, school attendance and performance, delinquency and juvenile justice involvement, stability of living arrangements, family functioning, caregiver strain, service utilization, and service satisfaction.

The current study represents the first opportunity to present trends in characteristics of children served across the CMHI lifespan using a hierarchical linear modeling approach. Specifically, these multi-year data were used to investigate:

  • Whether and how the behavior problems of the children served by sites change as the sites mature and whether there were trends in the behavior problems of children served from 1994–2007?
  • Whether behavior problems of the children served vary by race/ethnicity, age, gender, and referral source?

Methodology

Data source and participants

The data for this study were derived from information collected at intake into service as part of the national evaluation of sites receiving their initial funding between 1993 and 2004, and include all data available as of January 2007 (CMHS, 1996; 1997; 1998; 1999; 2000; 2001; 2003; 2004). The data were derived from baseline information collected as part the national evaluation between 1994 and 2007 from 90 of the 96 communities initially funded between 1993 and 2004. The excluded sites did not contribute data to the national evaluation on one or more of the variables used in this study. These funding years encompass four phases of the national evaluation and corresponding evaluation protocols. Phase I communities were initially funded in 1993 and 1994, Phase II communities in 1997 and 1998, Phase III communities in 1999 and 2000, and Phase IV communities between 2002 and 2004. Communities were expected to spend the first year of funding for infrastructure development; they were expected to begin enrollment of children into services and into the national evaluation beginning in their second year of funding. Thus, for sites initially funded in 1993, data were available for children served beginning in 1994. Communities funded in Phases I – III have completed their federal funding cycles and hence are no longer contributing data to the national evaluation, whereas Phase IV communities are potentially funded through 2010. The data for this study represent 14 annual cohorts of children and families enrolled into service in the funded systems of care (Table 1).

Table 1
Number of sites and children by funding year and enrollment cohort

Between the CMHI’s inception and January of 2007, cross-sectional data has been gathered on 75,433 children at intake into service. The program targeted children aged birth to 21 years with serious emotional disturbance. While grantees were asked to provide a core set of descriptive indicators on all children entering service, only a subset of children (and families) were approached and recruited for the longitudinal outcome study. Guidance was provided to communities from the national evaluation team on the number of families to enroll, that guidance equaled approximately 300 families per site (i.e., approximately 100 children per year for three years). This guidance was based on the sample size required to maintain sufficient power to detect differences over time in community-specific samples. Enrollment in the outcome study varied across grant communities. In most communities all willing families were recruited into the outcome study, however, for communities serving larger populations an option was available for developing sampling strategies to select a sufficient number of children and families from among those who entered system of care services.1 The current study uses measures included in the outcome study protocol, hence eligible children (n=15, 226) were drawn from those enrolled in the outcome study (n=22,347). Of those eligible, children were included in the current study sample if they had valid data on age, gender, referral source, race/ethnicity, and internalizing and externalizing problem behavior scores on the Child Behavior Checklist (Achenbach, 1991; Achenbach & Rescorla, 2000) at intake into system of care service. Differences among children in the current study sample and those enrolled in the outcome sample but not included in the current study sample (n=7,121) and from children not enrolled in the outcome study (n=53,086) were statistically significant for age, race/ethnicity, gender and referral source (see Table 2). These differences, while statistically significant (likely as a result of the large sample sizes), were generally small and hence not practically significant, with the exception of race/ethnicity. Children in the current study sample were more likely to be Black and less likely to be White or Hispanic than the children in the remainder of the outcome sample. Phase I cases were disproportionately excluded from the current study analyses, likely a result of less developed study protocols and procedures – hence more missing data – being utilized in Phase I.

Table 2
Sample Comparisons

As mentioned earlier, children were included in the sample if they had valid data on age, gender, referral source, race/ethnicity, and internalizing and externalizing problems. Referral source was the data element most frequently associated with exclusion from analyses. It was missing for about 30% of the total outcome study sample, while all other variables were missing for less than 10%. Several issues likely contributed to the amount of missing data. Local sites were responsible for implementation of the evaluation including recruitment, enrollment, and data collection. They varied considerably in their evaluation capacities, particularly in regard to their experiences with data collection staffing, allocation of resources to evaluation, incentives provided to participants, and recruitment in similar populations. All the aforementioned variables (age, race/ethnicity, gender, referral source, site and year of funding) were included in the model for analysis. The possibility remains, however, that children differ in other, unmeasured, characteristics; hence inference outside the analyzed sample is not assured.

The current study sample was divided into two sub-samples (training, n=7,611 and validation, n=7,615) for the purposes of analyses. The subsamples were randomly drawn from the overall sample of n=15,226 (Further explanation of this sub-sample approach is provided subsequently in conjunction with the analytic approach). Children in the training and validation sub-samples were quite similar in their demographic and clinical characteristics. Specifically, the majority of children in both sub-samples were male (66.6% and 66.1%, respectively) with an average age of approximately 11.8 years (M=11.87 and M=11.82, respectively). Slightly more than one-half of the children in each sample were White (53.1% and 53.4%, respectively), nearly one-quarter were Black (24.8% and 24.1%, respectively), approximately 12% were Hispanic (12.3% and 12.8%, respectively), 3% Asian/Native Hawaiian/Pacific Islander (2.9% and 3.0%, respectively), and 3.5% American Indian/Alaskan Native (3.7% and 3.5%, respectively). Mental health agencies were the most common source of referral in both sub-samples (28.4% and 29.2%, respectively), followed by schools (21.0% and 21.2%, respectively), child welfare (13.0% and 13.0%, respectively), justice (14.0% and 13.3%, respectively), the family (12.3% and 12.4%, respectively), and physical health care agencies or providers (2.9% and 3.0%, respectively). Children in both samples had comparable average internalizing (M=64.19 and 65.29, respectively) and externalizing (M=68.79 and M=68.86, respectively) problems which fall in the clinical range. Finally, there was equal representation of funding phase in both subsamples, with greatest representation of Phase I funded sites (42.6% and 42.9%, respectively), followed by Phases II (23.1% and 22.3%, respectively), III (19.8% and 19.5%, respectively) and IV (14.6% and 15.3%, respectively).

Instruments and Indicators

While the national evaluation protocol – in an effort to continually improve the depth, breadth and rigor of the data collected – was enhanced and modified across the four phases; a subset of indicators and measures remained consistent and serve as the information source for this study. Specifically, this study includes the following child demographic and clinical indicators.

Child age (years), race/ethnicity and gender were collected from caregivers at the child’s intake into services. Mutually exclusive race/ethnicity categories were created for the current study and include White, Black/African-American, Hispanic, Asian/Native Hawaiian/Pacific Islander, American Indian/Alaskan Native, and Other.

The child’s cohort 1 – 6 (i.e., cohort represents the year that the child entered system of care service–) was extracted from record review. The year the site received their initial federal funding was obtained from administrative records and ranged from 1993 to 2004. Source of referral into the funded system of care was obtained via record review. A single source of referral into the system of care is typically indicated in clinical records, despite the fact that more than one child serving agency may provide independent services to the target child at the time of their referral. Referral source categories were created for the current study and include mental health, education, child welfare, juvenile justice, physical health care agency or provider, caregiver, self and other.

Internalizing and externalizing problems were measured with the Child Behavior Checklist (CBCL; Achenbach 1991; Achenbach & Rescorla, 2001). The CBCL is a widely accepted instrument used to assess caregiver perception of problem behavior in children. The CBCL/4–18 (Achenbach, 1991) was administered in Phases I–III of the national evaluation and the revised CBCL/6–18 and CBCL/1.5–5 versions (Achenbach & Rescorla, 2001) were administered in Phase IV of the national evaluation. Differences between the older and newer versions of the CBCL are for the most part related to inclusion of new age-appropriate items that replaced rarely endorsed or unscored items on the older version. Continuity between the pre-2001 CBCL and the 2001 scales facilitates longitudinal analyses of pre-2001 editions in relation to the 2001 version (Achenbach & Rescorla, 2001). There was also a slight variation in the administration methods across phases with self-administration being used in Phase I, as compared to structured interview in subsequent phases. Previous research has indicated that method of administration does not affect the ability to discriminate clinical cases (Berg, Lucas, & McGuire, 1992).

The CBCL includes 113 behavioral items rated on a 3-point scale (0 - not true, 1 - somewhat or sometimes true, or 2 - very true or often true) that are scored to create a Total Problem behavior score, eight narrow band syndrome scales, and two broad band subscale scores for externalizing and internalizing problems (Achenbach, 1991; Achenbach & Rescorla, 2001). Raw and standardized T-scores are available. Internalizing and externalizing T-scores are used in the current study. CBCL T-scores range from 50 to 100, with higher scores indicating increased behavior problems. For both the newer and older version, a T-score between 60 and 63 for Internalizing, Externalizing and Total Problems scales is considered to be in the borderline clinical range and a T-score greater than 63 is in the clinical range. The reliability and validity of the CBCL has been adequately demonstrated (Achenbach, 1991; Achenbach & Rescorla, 2001).

Analytic Approach

The relationship between CBCL externalizing and internalizing problem scores, a group of demographic characteristics, and time was explored in this study. We were particularly interested in distinguishing two time trends, one associated with funding year (i.e., the year in which the site was initially funded) and another one associated with the enrollment cohort (i.e., the year of a child’s intake into services during a site’s six years of funding).

In order to formulate a statistical model well suited to both the research questions and the characteristics of the data, several alternative models were analyzed. This process of model formulation, using the data as an input, can compromise statistical inference if the same dataset is used to accomplish both purposes. As mentioned earlier, to prevent this problem the dataset was divided randomly in two halves: one of the samples -the “training” sample- was used for model formulation, while the second sample was reserved for inferential purposes (see Fox, 1997, pp. 514–518 for details on this particular method of cross-validation).

Ordinary least squares (OLS) regression was used to get a first estimation of the importance of demographic variables, cohort trend and site differences in CBCL problem scores, as well as to explore several interactions and alternative ways to model the relationship with continuous covariates (like age and enrollment cohort). In this framework, differences across sites can be incorporated as a set of fixed effects. It is not possible, however, to model these differences as a function of other variables, like funding year. Both GEE (General Estimating Equation) and HLM (Hierarchical Linear Model) were used to overcome this limitation by “moving” the site effect to the random part of the model. Though in both cases similar estimations were obtained for the fixed effects, HLM was finally selected because it allows a more detailed and realistic representation of the random components of the model (Goldstein, 1995, p. 38, includes a more technical comparison of these approaches).

The final fit includes both individual- and site-level models. The individual-level model can be written as:

Yij=β0j+β1j*Cohortij+Demographicsij*β2+eij

Where indexes i and j denote children and site respectively: there are i = 1,2,…, nj children within site j and j = 1,2, …, J sites. The dependent variable Yij denotes, alternatively, the CBCL externalizing and internalizing T-scores of child i in site j. For a given site (say, j=1), β0, j = 1 represents the average CBCL score for the initial cohort of children when all other covariates are zero, while β1, j = 1 captures the rate of change across successive cohorts of children. To simplify the exposition, we grouped together all the individual level covariates in a matrix called Demographicsij. This matrix includes the child’s age (both a linear and a quadratic term), gender, race/ethnicity and source of referral. β2 represents, therefore, a group of coefficients capturing systematic differences in CBCL scores by demographic segment. In the case of the categorical variables (gender, race/ethnicity and source of referral) the most frequently endorsed category (boys, white and mental health agency, respectively) serves as a reference against which the other categories are compared. Age was centered at its mean, which further specifies the meaning of the intercept. Finally, eij is a random error assumed to be normally distributed with a mean of zero and a variance of σ2.

Both the intercept and cohort’s slope are different for each site and can be thought of as outcomes of a second model. This site-level model can be written as:

β0j=γ00+γ01*FundingYearj+Demographics¯j*γ02+u0jβ1j=γ10+Demographics¯j*γ11+u1j

Where γ00and γ10 are the grand mean intercept and slope, i.e. the average value of intercept and slope across sites when site-level covariates are zero. We were particularly interested in the estimation of γ01, which captures differences in intercepts across sites by site’s funding year. Site’s aggregate indicators of individual level predictors were included in the site level-model of both intercept and slope mainly to ensure independence between site and individual level errors (Bafumi & Gellman, 2006). Specifically, the Demographics¯j matrix includes the following site-level variables: proportion of boys, average age, proportion of Whites, proportion referred from mental health provider and average enrollment cohort. γ02 and γ11 are vectors of coefficients capturing systematic differences between sites’ intercepts and slopes associated with differences in sites’ demographic composition - sometimes called “contextual” or “compositional” effects (Raudenbush & Bryk; 2002; p.139). Since site-level covariates are also centered at their mean, the grand mean intercept and slope correspond to a site with average demographic characteristics. Finally, u0j and u1j are site-level random errors, assumed to be bivariate normally distributed, each with a mean of zero, variance τu0and τu1 and covariance τu0u1.

All procedures were implemented using R 2.5.1 (R Development Core Team, 2007). The nmle package (Pinheiro et al., 2007) was used to fit the various hierarchical models.

Results

Table 3 and Table 4 summarize the results of fitting the previously described model to the validation sample. The complete set of results, including results from the training sample, is available upon request.

Table 3
Individual−level model (*)
Table 4
Site-level model (*)

Age, race/ethnicity, and referral source are significant predictors of children’s CBCL externalizing and internalizing T-scores (p<0.01). CBCL scores increased with age at a decreasing rate. At age 12, for internalizing scores, and age 10, for externalizing scores (using point estimates), the relationship reversed, such that CBCL scores decreased with age. Generally, children from non-White races/ethnicities have lower externalizing scores. This is also true for Black and American Indian children in the case of internalizing scores. For instance, Black children are estimated to have a 3.18 point (95% CI 2.43 – 3.93) lower internalizing score and a 1.21 point (95% CI 0.48 – 1.94) lower externalizing score than White children. Gender is a significant predictor of externalizing scores (p<0.01), but not of internalizing scores. Girls are estimated to have a 1.77 point (95% CI 1.25 – 2.30) higher externalizing score than boys at baseline. Finally, children referred from sources other than a mental health agency were, in general, estimated to have lower CBCL internalizing and externalizing scores. That is particularly the case for school and child welfare referral sources, which are estimated with internalizing scores that were 1.79 points (95%CI 0.96 – 2.61) and 3.01 points (95%CI 2.09 –3.93) lower, respectively; and with externalizing scores that were 1.69 points (95%CI 0.89 –2.50) and 2.34 points (95%CI 1.45 – 3.23) lower, respectively. It is also worth noting that the difference in internalizing scores of children referred from juvenile justice were estimated to be 3.94 points (95%CI 2.96 – 4.93) lower than among children referred from mental health agency(See Figure 1).

Figure 1
CBCL variation between children

Mean CBCL internalizing and externalizing T-scores have increased across the funding phases of the CMHI when comparing children with the same demographic characteristics and source of referral (See Figure 2 and Table 4). Specifically, there is a significant and positive linear trend in the initial site average of CBCL scores, both externalizing and internalizing, by site funding year (p<0.01). The estimations of the rate of change by funding year for internalizing and externalizing scores are similar (0.30 [95%CI 0.11–0.49] and 0.20 [95%CI 0.06–0.41], respectively). Thus, sites funded one year apart are expected to have a 0.30 point difference in the initial mean internalizing scores and a 0.20 point difference in the initial mean externalizing scores of their children; and over the 12 years of initial funding covered in these analyses, there is a 3.3 point difference (roughly one-third of a standard deviation in CBCL T-scores) between the initial mean internalizing score and a 2.2 point difference (roughly one-fifth of a standard deviation) between mean externalizing score for sites initially funded in 1993 versus sites initially funded in 2004.

Figure 2
CBCL cross-year variation between sites

After the initial year of funding, sites’ mean CBCL scores follow different trajectories as the sites served successive enrollment cohorts of children (See Figure 3 and Table 4). These trajectories can be described as multiple linear trends with different slopes (whose variation is estimated at 0.51 and 0.42 for internalizing and externalizing scores, respectively). On average, there is a downward trend estimated at −0.37 (95% CI −0.74 - 0.00) for internalizing scores and −0.48 (95% CI −0.83 – −0.14) for externalizing scores, though it is only marginally significant in the first case (p ~ .05). In the case of externalizing scores, and based on point estimates, children from the last cohort had scores that were, on average, 2.41 points lower than those of children from the first cohort.

Figure 3
CBCL cross-year variation within sites

Finally, site aggregates of child-level demographic characteristics are generally not associated with CBCL scores after controlling for child-level differences (results not shown but available upon request). The only exception is race/ethnicity for internalizing CBCL scores (p<0.05). A difference of 32% in the percentage of Whites between two sites -which represents one standard deviation was associated with a difference of 0.82 (95% CI 0.04–1.60) points on the average internalizing CBCL scores.

Discussion

This study represents the first attempt at understanding the characteristic trends of children with serious emotional disturbance entering federally funded systems of care between 1994 and 2007. The ability to understand these trends through sophisticated analytic modeling provides the unique opportunity for a program-in-review which can be used to inform future federal programming, as well as the alternative ways that the existing system of care infrastructure can be used to meet the most needs of the most children and families. Furthermore, it provides the necessary characteristic context to understand child and family level outcome trends.

The findings from this study suggest that demographic variables and referral source are significant predictors of children’s CBCL scores, both externalizing and internalizing, across the 14 year service delivery period. Children with the same demographic characteristics and referred by the same source, however, have different CBCL scores depending on the site in which they are served. In particular, children entering services later in the site’s funding cycle had lower CBCL scores and children served in sites funded in later years had higher CBCL scores. Specifically and historically, females have entered services with more severe problems while children referred from non-mental health sources, younger children, and those from non-white racial/ethnic backgrounds have entered system of care services with less severe problems.

Given the relative scarcity of data related to children’s mental health problems across time (generally or specific to the time period of interest for this study), it is difficult to definitively contextualize these findings within (inter)national trends (Farmer, Mustillo, Burns, & Costello; 2005). With that said however, available – albeit not ideal – previous literature suggests that the behavior problems of both boys and girls in the US and abroad have remained relatively stable or declined in the last decade (Achenbach, Dumenci & Rescorla, 2003; Federal Interagency Forum on Child and Family Statistics, 2007, 2008; Sourander, Santalathi, Haavisto, Piha, Ikaheimo & Helenius, 2004). Specifically, Achenbach and colleagues (2003) found in representative samples of American 4–16 year olds, that regardless of child gender, parent reports of significant problems on the CBCL declined between 1989 and 1999. During that same period of time, investigators in Finland reported no increase in rate of overall problem behavior among children 8 to 9 years old, but did note that while boys demonstrated a decline in psychiatric symptoms, girls’ symptom levels remained relatively unchanged (Sourander et al., 2004). Most recently, the 2007 and 2008 America’s Children Key National Indicators of Children’s Well-Being reports stated that between 2001 and 2006 the percentage of children 4 to 17 years old with serious emotional and behavioral problems has remained relatively stable (Federal Interagency Forum on Child and Family Statistics, 2007; 2008). These reports do note, however, that males and older children consistently had higher rates of serious emotional disturbance. Collectively these findings suggest stability in problem behaviors for children overall, whereas in this study the children entering systems of care funded in the more recent years had greater problem behaviors. Reconciliation of this contradiction is potentially at least two-fold in that 1) the federal proposal solicitation process across the funding phases has been iteratively fine-tuned in an effort to deliberately target and support particularly high-need populations and regions of the country; and that 2) outreach efforts at the local level have likely become more sophisticated over the years as a community of practice has grown and expanded related to system of care implementation (Kutash, Duchnowski, & Friedman, 2005).

System of care sites – regardless of year of initial funding – are serving children with the most serious behavioral problems in their earliest years of funding. This pattern of children with lower levels of severity being served in later years of a site’s funding could potentially be driven by annual enrollment expectations associated with funding, annual recruitment expectations associated with the national evaluation, or simply by serving children with the greatest needs in a given community’s program catchment area in the early program years. Regardless of the mechanism of action, the notion that the maximum load on the system’s service provision is occurring during its period of greatest immaturity is worth considering. Recent findings regarding system of care development suggest a set of seven interconnected and complex critical factors (Ferreira, Hodges, Kulka-Acevedo, & Mazza; April 2008) that contribute to successful system implementation and provision of service which require dedicated time and process to articulate and execute. Systems that begin serving children with the serious needs before they had the opportunity to develop and execute these critical and complex factors may not be optimally prepared to provide best practice system of care supports and interventions.

Despite the noted trends associated with overall problem behavior across funding phases and within funding years, findings indicate that the culturally-specific (i.e., gender, age, race/ethnicity, referral source) problem thresholds for entering services have remained stable since the CMHI began. It appears that girls must reach a higher maximum threshold of problem behavior before entering (or being referred into) service and that children of color, along with those from non-mental health sectors and young children, are entering services (or being referred in) with lower overall problem levels. The value judgment associated with these consistent disparities must be considered. For example, are these thresholds desired in order to avoid stigma (related to girls with behavior problems) or promote early intervention (for young children or children of color) or do they represent bias and stereotyping in the assessment of problem behavior and subsequent referral for services? Earlier evidence which suggests that systems of care are indeed reaching the most disadvantaged youth (Miech et al., 2008) forces us to evaluate whether the consistent findings of lower problem behavior among young, non-white and non-mental health referred children is indicative of the system of care infrastructure opening avenues of entry for these oft-overlooked youth or whether these systems of care are receiving referrals motivated out of relative bias (e.g., measurement or cultural).

Study Limitations

There are several limitations that must be considered in the interpretation of these findings. First, the data presented were collected via different national evaluation protocols and retrospectively combined to create a cross-phase data set. While these evaluation protocols varied as a function of phase, detailed attention was paid to ensuring the variables selected for inclusion in the study could be reliably combined and reported. Second, a caregiver report measure was used to assess problem behavior without the opportunity to triangulate response by the youth or clinician; however, the measure and reporting sources were consistent across time and should not introduce bias into the trend analyses. Furthermore, historical threats to internal validity might be potential sources of bias in caregiver ratings (e.g., Columbine shootings). While this was not directly addressed in the current study, Achenbach and colleagues (2003) did analytically investigate this possibility and found no evidence for support. Third, while several significant differences were found between the study sample children and the larger samples from which they were drawn, these difference were relatively small in magnitude and likely attributable to the large sample sizes. Fourth, these data were gathered from children entering federally funded system of care programs and hence may not generalize to other community-based children’s mental health populations. A related limitation lies in the fact that the evaluation was not funded to collect community-level data on prevalence of serious emotional disturbance in the population of youth in funded communities. Thus, the existing level of need in the population and how it changes over years of funding, as well as the program’s level of penetration, remain unknown. Next, while this study does indeed answer the question of how time and demographic characteristics are related to the behavioral and emotional problems of children entering system of care services, it does not answer how time itself may have affected the demographic composition of the children entering services. It was determined that this question – albeit important – is best addressed through a separate study. Lastly, the nature of the federal request for proposals for grant funding guides and shapes the way that sites target and build their service systems, and ultimately their outreach and referral mechanisms.

Policy and Program Implications

Despite limitations these findings have important policy and programming implications, as well as implications for local system of care program development and implementation. As the federal initiative continues to grow, sustained attention must be directed to the cultural sensitivity of these systems as more evidence accumulates to suggest that indeed the system of care infrastructure is poised and ready to address health disparities among disadvantaged youth. Legislation that continues to fund efforts aimed at eliminating disparities is needed, as well as funding demonstration projects and research to better understand the mechanisms through which disparities occur. To this end, a better understanding of the context in which children enter into mental health services could result in initiatives that improve the health and well being of specific groups. For example, as discussed earlier, findings from this study suggest that a higher threshold may exist for recognizing and/or labeling behaviors as symptomatic of mental health problems in girls than in boys. If this is the case, then a campaign to increase awareness of behaviors in girls indicative of mental health problems could then increase and facilitate referrals into treatment for girls.

Large scale technical assistance efforts should be directed to support system of care development and service delivery across target populations, but also during times of greatest system strain – for example during earlier years of federal funding. As future federal funding is allocated and dedicated to this initiative, considering these critical trajectory points will ensure adequate programming support is available when it is most needed. For example, in the early years of funding, when sites are serving the most severe children, systems may want to improve assessment and service planning functions to maximize allocation of services; ensure sufficient provider capacity given the potential need for lower case loads; and assess their planned service arrays in light of the likely needs associated with this level of severity. When these systems develop their budget and staffing policies, they will need to anticipate and plan for serving children with the most severe problems in the initial years of their funding. Furthermore, if the trajectory to date continues with future phases of funded communities serving more and more severe children, a realignment of service requirements (both quantity and quality) and an inventory of available resources must be undertaken to ensure that all children served through systems of care, regardless of their problem severity, receive the best care possible; and that future funded systems of care have the requisite capacity and resources to serve these children and families. From an evaluation perspective, this could be aided by incorporating a data collection component to provide community-level prevalence estimates of serious emotional disturbance in the funded communities’ youth populations. This would shed light on the remaining uncertainty regarding the nature of the decreases in clinical severity of youth in subsequent enrollment cohorts over the years of grant funding in the local communities.

Local program planning and system development – as it relates to cultural competency, staffing capacity, and outreach efforts – must consider these consistent patterns across sites and phases, because they provide insight into characteristic trends (i.e., fluctuation and stability) that have been experienced relative to the level of problem behavior among children entering services. Each and every community past and present, by virtue of the incredible diversity of funded sites, is not expected to see relevance in these findings. However, this sort of information can be critically useful when forecasting staffing needs, portals of entry into service, enrollment goals, and technical assistance needs.

As the CMHI continues to grow in size, this program-in-review provides the opportunity to better understand who has been served and how the clinical characteristics of the served populations has changed over time, and speculate as to what has influenced both the stability and fluctuation in these trends. The knowledge-gain associated with these trends provides the potential for data to drive and shape the future of the Initiative while simultaneously providing the necessary context for more in-depth understanding of the trends associated with child characteristics and client outcomes.

Acknowledgement

This research was supported by National Institute of Mental Health Grant # 1R01MH075828. The views expressed are the opinions of the authors and not those of the Substance Abuse and Mental Health Services Agency, the National Institute of Mental Health, the National Institute of Health, or the federal government.

Footnotes

1Many site-specific factors, such as obtaining a large enough sample using sampling strategies and maintaining the sampling frame across time, influenced the ability of local evaluation teams to maintain a random sampling strategy.

Contributor Information

Christine Walrath, ICF Macro, 116 John Street, Suite 800, New York, NY 10038, USA.

Lucas Godoy Garraza, ICF Macro, 116 John Street, Suite 800, New York, NY 10038, USA.

Robert Stephens, ICF Macro, Atlanta, GA, USA.

Melissa Azur, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health Baltimore, MD, USA.

Richard Miech, Department of Health and Behavioral Sciences, University of Colorado, Denver Campus, Denver, CO, USA.

Philip Leaf, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health Baltimore, MD, USA.

References

  • Achenbach T. Manual for the Child Behavior Checklist/4–18 and 1991 Profile. Burlington, VT: Department of Psychiatry, University of Vermont; 1991.
  • Achenbach TM, Dumenci L, Rescorla LA. Are American children’s problems still getting worse? A 23-year comparison. Journal of Abnormal Child Psychology. 2003;31(1):1–11. [PubMed]
  • Achenbach TM, Rescorla LA. Manual for the AESBA school-age forms & profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families; 2000.
  • Bafumi J, Gelman A. Fitting Multilevel Models When Predictors and Group Effects Correlate; Prepared for the 2006 Annual Meeting of the Midwest Political Science Association; Chicago, Il. 2006. Available at http://www.stat.columbia.edu/~gelman/research/unpublished/Bafumi_Gelman_Midwest06.pdf.
  • Berg I, Lucas C, McGuire R. Measurement of behavior difficulties in children using standard scales administered to mothers by computer: Reliability and validity. European Child and Adolescent Psychiatry. 1992;1(1):14–23.
  • Briggs-Gowan MJ, Horwitz SM, Schwab-Stone ME, Leventhal JM, Leaf PJ. Mental health in pediatric settings: Distribution of disorders and factors related to service use. Journal of the American Academy of Child and Adolescent Psychiatry. 2000;39(7):841–849. [PubMed]
  • Burns BJ, Costello EJ, Angold A, Tweed D, Stangl D, Farmer EM, Erkanali A. Children's mental health service use across service sectors. Health Affairs. 1995;14(3):147–159. [PubMed]
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1996. 1996. Unpublished manuscript.
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1997. Atlanta, GA: Macro International Inc; 1997.
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1998. Atlanta, GA: Macro International Inc; 1998.
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1999. Atlanta, GA: ORC Macro; 1999.
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 2000. Atlanta, GA: ORC Macro; 2000.
  • Center for Mental Health Services. Annual report to Congress on the evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 2001. Atlanta, GA: ORC Macro; 2001.
  • Center for Mental Health Services. The Comprehensive Community Mental Health Services for Children and Their Families Program: Evaluation findings–Annual report to Congress, 2002–2003. Atlanta, GA: ORC Macro; 2003.
  • Center for Mental Health Services. The Comprehensive Community Mental Health Services for Children and Their Families Program: Evaluation findings–Annual report to Congress, 2004. Atlanta, GA: Macro International Inc; 2004.
  • Manderscheid RW, Berry JT, editors. Center for Mental Health Services. Mental Health, United States, 2004. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2006. DHHS Pub no. (SMA)-06-4195.
  • Costello EJ, Burns BJ, Costello AJ, Edelbrock C, Dulcan M, Brent D. Service utilization and psychiatric diagnosis in pediatric primary care: The role of the gatekeeper. Pediatrics. 1988;82(3 Pt 2):435–441. [PubMed]
  • Farmer EMZ, Mustillo S, Burns BJ, Costello EJ. The epidemiology of mental health problems and service use in youth: Results from the Great Smoky Mountains study. In: Epstein MH, Kutash K, Duchnowski A, editors. Outcomes for children and youth with emotional and behavioral disorders and their families: programs and evaluation best practices. 2nd ed. Austin, TX: Pro-Ed; 2005. pp. 23–44.
  • Federal Interagency Forum on Child and Family Statistics. America’s Children: Key National Indicators of Well-Being. Washington, DC: U.S. Government Printing Office; 2007. Federal Interagency Forum on Child and Family Statistics.
  • Federal Interagency Forum on Child and Family Statistics. America’s Children in Brief: Key National Indicators of Well-Being. Washington, DC: U.S. Government Printing Office; 2008. Federal Interagency Forum on Child and Family Statistics.
  • Ferreira K, Hodges S, Kukla-Acevedo S, Mazza J. System implementation issue brief #5—What works: System development strategies across communities. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health; 2008.
  • Fox J. Thousand Oaks, CA: Sage; 1997. Applied Regression Analysis, Linear Models, and Related Methods.
  • Goldstein H. 2nd Ed. London: Edward Arnold; 1995. Multilevel statistical models. Available at http://www.cmm.bristol.ac.uk/team/HG_Personal/multbook1995.pdf.
  • Holden E, Friedman R, Santiago R. Overview of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Journal of Emotional and Behavioral Disorders. 2001;9:4–12.
  • Crisis in child mental health: Challenge for the 1970s. New York: Harper & Row; 1969. Joint Commission on the Mental Health of Children.
  • Knitzer J. Unclaimed Children: The failure of public responsibility to children and adolescents in need of mental health services. Washington, D.C.: Children's Defense Fund; 1982.
  • Kutash K, Duchnowski AJ, Friedman RM. The system of care twenty years later. In: Epstein MH, Kutash K, Duchnowski AJ, editors. Outcomes for children with emotional and behavioral disorders and their families: Program and evaluation best practices. 2nd ed. Austin, TX: Pro-Ed; 2005. pp. 3–22.
  • Leaf PJ, Alegria M, Cohen P, Goodman SH, Horwitz SM, Hoven CW, Narrow WE, Vaden-Kiernan M, Regier DA. Mental health service use in the community and schools: Results from the four-community MECA study. Journal of the American Academy of Child and Adolescent Psychiatry. 1996;35(7):889–897. [PubMed]
  • Miech R, Azur M, Dusablon T, Jowers K, Goldstein AB, Stuart EA, Walrath C, Leaf PJ. The Potential to Reduce Mental Health Disparities Through the Comprehensive Community Mental Health Services for Children and Their Families Program. Journal of Behavioral and Health Services Research. 2008;35(3):253–264. [PMC free article] [PubMed]
  • Pires S. National Technical Assistance Center for Children’s Mental Health. Building Systems of Care: A Primer. Washington, DC: Georgetown University Child Development Center; 2002.
  • New Freedom Commission on Mental Health. Achieving the Promise: Transforming Mental Health Care in America. Final Report. 2003. DHHS Pub. No. SMA-03-3832.
  • Pinheiro J, Bates D, DebRoy S, Sarkar D. the R Core team. nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1-86. 2007
  • Pumariega AJ, Winters NC, Huffine C. The evolution of systems of care for children's mental health: forty years of community child and adolescent psychiatry. Community Mental Health Journal. 2003;39(5):399–425. [PubMed]
  • R Development Core Team. R Foundation for Statistical Computing. Vienna, Austria: 2007. R: A language and environment for statistical computing. ISBN 3-900051-07-0, URL http://www.R-project.org.
  • Raudenbush SW, Bryk AS. 2nd Edition. Thousand Oaks, CA: Sage; 2002. Hierarchical Linear Models: Applications and data analysis methods.
  • Sourander, Santalathi, Haavisto, Piha, Ikaheimo, Helenius Have there been changes in children’s psychiatric symptoms and mental health service use? A 10-year comparison from Finland. Journal of the American Academy of Child and Adolescent Psychiatry. 2004;43(9):1134–1145. [PubMed]
  • Stroul B, Friedman R. A system of care for children and youth with severe emotional disturbances. Washington, DC: Georgetown University Child Development Center; 1986.
  • Stroul BA, Friedman RM. A system of care for children and youth with severe emotional disturbances. (Revised edition) Washington, D.C: Georgetown University Child Development Center; 1994.
  • U.S. Department of Health and Human Services. Mental health: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health; 1999.
  • U.S. Department of Health and Human Services. Healthy People 2010 Midcourse Review. Washington, DC: U.S. Government Printing Office, December; 2006.