PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Psychol Assess. Author manuscript; available in PMC 2011 September 1.
Published in final edited form as:
PMCID: PMC2936710
NIHMSID: NIHMS231715

Assessment of Dysregulated Children Using the Child Behavior Checklist: A Receiver Operating Characteristic Curve Analysis

Abstract

Disorders of self-regulatory behavior are common reasons for referral to child and adolescent clinicians. Here, we sought to compare two methods of empirically-based assessment of children with problems in self-regulatory behavior. Using parental reports on 2028 children (53% boys) from a U.S. national probability sample of the Child Behavior Checklist, we applied Receiver Operating Characteristic (ROC) Curve analysis to compare scores on the Post-traumatic Stress Problems Scale (PTSP) of the CBCL to the CBCL-Dysregulation Profile, identified using latent class analysis of the Attention Problems, Aggressive Behavior, and Anxious/Depressed scales of the CBCL. The CBCL-PTSP score demonstrated an Area Under the Curve (AUC) of between 0.88-0.91 for predicting membership in the CBCL-DP profile for boys and for girls. These findings suggest that the CBCL-PTSP, which others have shown does not uniquely identify children who have been traumatized, does identify the same profile of behavior as the CBCL-DP. Therefore, we recommend renaming the CBCL-PTSP the Dysregulation Short Scale and provide some guidelines for the use of the CBCL-DP scale and the CBCL-PTSP in clinical practice.

Keywords: CBCL, Pediatric Bipolar Disorder, Dysregulation, PTSD, Child, Bipolar Affective Disorder

Introduction

Dysregulation in Clinical Child Psychology and Psychiatry

There is a growing need for measures that reliably characterize children with profound disorders of self regulation. While there is little doubt about the existence of children who demonstrate impairment in multiple regulatory domains including affective regulation, behavioral regulation, and cognitive (e.g., attentional) regulation, controversy swirls regarding the proper conceptualization of these problems. Some researchers have described these children as “highly co-morbid” psychopathology (Carlson, 2007), while other have used labels such as hyperkinetic conduct disorder (Holtmann, Goth, Wockel, Poustka, & Bolte, 2008) or severe mood dysregulation (Leibenluft, Charney, Towbin, Bhangoo, & Pine, 2003). Perhaps receiving the most attention is the contention that these symptoms represent a broad phenotype of pediatric bipolar disorder (NIMH, 2001), or chronic bipolar disorder with an explosive modifier (Staton, Volness, & Beatty, 2008). Inconsistencies and disagreements as to the assessment, classification, and treatment of these youth are particularly problematic. Despite disagreement as to how these children might be best described, however, there is consensus that the severity of their psychopathology and functional impairment represent a serious clinical and public health problem. The clinical and public health ramifications have spurred the National Institute of Mental Health, the American Academy of Child and Adolescent Psychiatry (AACAP), and other groups to call for the study of these broadly dysregulated children in greater detail (Akinso, 2007; McClellan, Kowatch, & Findling, 2007; NIMH, 2001) and the American Psychiatric Association (APA) has recently proposed a diagnosis that may encompass some of these children, Temper Dysregulation Disorder with Dysphoria, which captures many of the severe mood dysregulation criteria of Leibenluft and colleagues.

Assessment of Dysregulation Using the Child Behavior Checklist

The CBCL is one of the most widely used parent-report measure of child behavioral issues. It has been translated into 85 languages and is administered in more than 35 cultures (Berube & Achenbach, 2009). It allows the clinician and researcher to measure behavior in an analytic environment that is normed by informant, age, and sex. In the absence of a “gold standard” for measuring self-regulatory behavior in children, CBCL profiles have been used to measure various disorders where self-regulation is thought to be impaired. Researchers have been using the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2001) to attempt to describe these children using empirically-derived and standardized methods. Specifically, it has been demonstrated that children with a profile characterized by co-occurring elevations in attention problems, aggressive behavior, and anxious-depressed scales of the CBCL have increased rates of pediatric bipolar disorder (Faraone, Althoff, Hudziak, Monuteaux, & Biederman, 2005; Mick, Biederman, Pandina, & Faraone, 2003), suicidality (Althoff, Rettew, Faraone, Boomsma, & Hudziak, 2006; Holtmann et al., 2008; Volk & Todd, 2007), comorbid psychiatric disorders (Holtmann et al., 2008; Volk & Todd, 2007), and a high rate of adult psychopathology (Meyer et al., 2008). This profile of high attention problems, aggressive behavior, and anxious-depression has been termed the CBCL-Pediatric Bipolar Disorder profile, the CBCL-Juvenile Bipolar Disorder profile, or the CBCL-mania proxy (Galanter et al., 2003; Hudziak, Althoff, Derks, Faraone, & Boomsma, 2005; Mick et al., 2003). However, because of a lack of specificity in predicting narrow phenotypic DSM mania in comparison to children with impairment in multiple self-regulatory domains (Diler et al., 2009; Kahana, Youngstrom, Findling, & Calabrese, 2003; Volk & Todd, 2007), we have suggested that the name “CBCL-Dysregulation Profile” or CBCL-DP better describes the children meeting this profile (Althoff, Rettew, Ayer, & Hudziak, submitted; Althoff, Rettew, Ayer, Sulman, & Hudziak, 2008; Ayer et al., 2009).

Recently, other subscales based on the CBCL have emerged to examine other specific problem areas. The CBCL-Post Traumatic Stress Problems scale (CBCL-PTSP) is a modified version of scales used by Wolfe and colleagues (Wolfe, Gentile, & Wolfe, 1989); (Wolfe & Birt, 1997). While the initial research supported its sensitivity among youth with histories of trauma, its sensitivity and, by extension, its use as a screening tool for PTSD, was subsequently questioned (Ruggiero & McLeer, 2000) . Ruggiero and McLeer (2000) noted that the 14 items associated with post-traumatic symptoms did not discriminate well between PTSD children and other children with high levels of overall problems. Added to other research (Sim et al., 2005) and clinical observations that many children without histories of trauma were rated in the clinical range on the CBCL-PTSP scale, the work by Ruggiero and McLeer (2000) led us to examine this scale more closely. This study reflected the clinical experience in our center that the vast majority of children with elevations on the CBCL-PTSP scale did not have an identified trauma history. Moreover, clinicians were beginning to excessively search for a history of trauma based on this scale. Because children with PTSD often present with self-regulatory problems and a mix of internalizing and externalizing problems, and because the CBCL-PTSP scale appears to identify children with broad and severe self-regulatory problems in the clinic, it was hypothesized that the CBCL-PTSP and the CBCL-DP scales were measuring the same children. To test this hypothesis, correlations between the CBCL-PTSP and summed total of the attention problems, aggressive behavior, and anxious-depressed scales (the CBCL-DP summed score) were first estimated (Ayer et al. 2009). The CBCL-DP summed score has been used in genetic studies (Boomsma et al., 2006; Hudziak et al., 2005) and Receiver Operating Characteristic Curve (ROC) analyses (Faraone et al., 2005) as a continuous measure of the CBCL-DP profile. Results revealed that the correlation between the CBCL-PTSP and the CBCL-DP summed score was 0.9. Further, structural equation modeling determined that this pattern could be explained by a single latent variable of “dysregulation” even when total problems, overlapping items, and other confounders were taken into account (Ayer et al., 2009). While this finding provides convincing evidence that the two scales appear to be measuring the same construct, it doesn't speak to whether the CBCL-PTSP scale is identifying the same children with the pattern of high attention problems, aggressive behavior, and anxious-depression that are captured by the CBCL-DP profile. The research presented here used ROC analysis in combination with latent class analysis to determine if scores on the CBCL-PTSP accurately predicted individuals with a profile consistent with the CBCL-DP. We hypothesized that the CBCL-PTSP score for an individual would accurately place them into a CBCL-DP latent class with high attention problems, aggressive behavior, and anxious-depression.

Latent Class Analysis

Rather than classify children into the CBCL-DP profile on the basis of an arbitrary cutpoint, we used latent class analysis (LCA). LCA is a form of categorical data analysis which attempts to identify a smaller number of mutually exclusive, but internally homogeneous classes to account for the observed symptom (or item) endorsement profiles. It is a form of person-centered analysis that presupposes the existence of discrete latent categories or classes of responding. Characteristics of an individual (in the current analysis, items on the CBCL) are placed into the analysis and, using maximum likelihood estimation, latent variables are defined that capture the response profile to more or less of a degree (Goodman, 1974). In the current work, this estimation is performed via an iterative Expectation-Maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977) where an expectation is first created for the log-likelihood of an observed variable and the estimated latent variable followed by a procedure which attempts to maximize that log-likelihood. Each time a model is run, the two measures of importance that emerge are (1) probabilities of class membership assignment for each individual and (2) conditional item endorsement probabilities for each item within a given latent class (McCutcheon, 1987). Each class is assumed to be functionally independent from the others within a particular solution. Researchers can choose the number of latent classes to be fit to the model and can do so either by allowing for no restrictions on the conditional item endorsement probabilities within a class and no restrictions on the probability of class membership (exploratory LCA) or with conditions on one or both of these metrics (confirmatory LCA) (McCutcheon, 1987). Decisions about the number of classes can be made on the basis of fit indices. Because this is a log-likelihood-based analysis, it should be possible to compare log-likelihoods between models and to determine significance based on their distribution as a chi-square statistic. However, in an analytic environment with sparse data matrices, these log-likelihood-based estimates from the model cannot be fully trusted and other measures must be used (Nylund, Asparouhov, & Muthen, 2007). Specifically, it is recommended in this situation that researchers use log-likelihood-based measures that also consider the rules of parsimony like the Bayesian Information Criterion (BIC) and/or use nonparametric bootstrapping to determine significance (Nylund et al., 2007). Researchers can compare the BIC of one model as compared to the model with an additional latent class with reduction of the BIC indicating a better model fit. For example, if there is an improvement in the BIC when moving from a model with M number of classes to a model with M+1 number of classes, then the M+1 class solution is accepted. The procedure is repeated until the optimal class solution is found. The final result is a set of independent classes that represent different levels of item endorsement probabilities, which can be represented graphically (see Figures 1a and 1b) with the item on the X axis and the conditional item endorsement probability within that class on the Y axis. For any given individual, they are placed into the class for which they have the highest probability of class membership. In this way, we determined the profile of response consistent with the CBCL-Dysregulation Profile. This allowed for an empirical, data-driven manner of grouping those individuals with high attention problems, aggressive behavior, and anxious-depression. Placement into these classes was considered as the outcome variable for the Receiver Operating Characteristic Curve Analysis.

Figure 1Figure 1
Part A: LCA curves for Classes 1, 3, and 4 (classes with zero, 1, or 2 problem areas). Part B: LCA curves for Classes 2, 5, 6, and 7 (classes with three problem areas – those most similar to the DP latent class).

Receiver Operating Characteristic Curve Analysis

Because we hypothesized that children placed into the CBCL-Dysregulation Profile latent class would also be well characterized by the CBCL-PTSP scale score, we aimed to quantify the ability of the CBCL-PTSP score to classify children in the CBCL-DP latent class. One could simply choose an arbitrary cutpoint for the CBCL-PTSP and see how many children rated over that threshold were also in the CBCL-DP latent class. However, the choice of the threshold would determine the number of children placed into the CBCL-DP latent class (and how many children from other classes were also “positive” on the CBCL-PTSP). If one chooses too low a threshold, all children from all classes will be called “positive” (i.e. there will be a great number of false positives). Conversely, if one chooses too high a threshold, no children with the CBCL-DP will be called “positive” (i.e. there will be a great number of false negatives). The Receiver Operating Characteristic (ROC) curve analysis allows researchers to determine the optimal sensitivity (“the proportion of cases in which a disorder is detected when it is in fact present”) and specificity (“the proportion of cases for which a diagnosis of disorder is rejected when rejection is warranted”) ( AERA, APA & NCME, 1999) at which a given continuous trait identifies a disorder (Krzanowski & Hand, 2009). ROC analysis assesses the diagnostic efficiency of tests and is used to adjust cut-points for clinical or research purposes (McNeil & Hanley, 1984). It is frequently used to assess the accuracy of diagnostic tests (Siegel, Vukicevic, Elliott, & Kraemer, 1989; Siegel, Vukicevic, & Spitzer, 1990; Swets, 1982). By plotting true positives (Sensitivity) versus false positives (1-Specificity), the ROC curve characterizes the relation between belonging in a particular population (in the current work, belonging in a particular latent class) across a range of thresholds for the continuous classifier (in the current work, the CBCL-PTSP score). The higher the graph extends toward the upper left corner of the graph, the higher the discriminatory power of the test. This overall function is summarized by the area under the curve (AUC) statistic. The AUC can range from 0 to 1.0. If a test predicts the disorder perfectly the AUC will be 1. If a test predicts the disorder at random, the AUC will be 0.5. The AUC is the probability that a randomly selected member of the latent class will have a more extreme CBCL-PTSP score than a randomly selected member not in that class (Colditz, Miller, & Mosteller, 1988; Hanley & McNeil, 1982). Here, we predicted that an ROC curve analysis would demonstrate that the CBCL-PTSP used as a classifier would perform well as an indicator for membership in the CBCL-DP latent class.

Methods

Subjects

Participants for this study come from the 1999 normative sample used to calculate the norms for the most recent version of the CBCL. The 1999 CBCL national sample was carefully designed to be representative of the 48 contiguous states (Achenbach & Rescorla, 2001). We have reported on the demographics of this sample previously (Ayer et al., 2009). Briefly, parents (72% mothers, 22% fathers, and 6% other caregivers) of 2028 children, including 276 children who had been referred for mental health services in the preceding 12 months, completed the CBCL during home interviews. Children ranged in age from 6 to18, with an overall mean age of 11.98 (SD = 3.53). Race distribution was 60% Caucasian, 20% African-American, 9% Latino, and 11% Mixed or Other (Achenbach & Rescorla, 2001). Socioeconomic status (SES) was calculated based on the Hollingshead index (Hollingshead, 1975 ), and demonstrated that the sample consisted of 33% Upper, 51% Middle, and 16% Lower SES. All participants provided informed consent.

Assessment of PTSP and CBCL-DP

Caregivers completed the CBCL (Achenbach & Rescorla, 2001), rating their children on a three-point scale for 118 behavior problems. Specifically, a parent rates the behavior as a “0” if it is not present, a “1” if the child sometimes exhibits the symptom, and a “2” if the child frequently demonstrates the symptom. The raw PTSP scale score is calculated by adding ratings on fourteen items (see Table 1). The CBCL-DP profile is calculated from the ratings on items from the Attention Problems (AP), Aggressive Behavior (AB), and Anxious/Depressed (AD) scales. Because the LCA was performed as part of a multi-informant study of children with self-regulatory problems, the 38 items from these three scales were the items that existed on all three of the CBCL, the Youth Self Report, and the Teacher Report forms. Seven items overlap between the CBCL-PTSP and the CBCL-DP profile (Table 1). To maximize the applicability of these findings to clinical practice, in our main analysis we allowed for the overlap between these subscales in this analysis. In a secondary analysis, we removed the 7 overlapping items from the CBCL-DP LCA and re-ran the analysis.

Table 1
CBCL-PTSP Items.

Data Analysis

All analyses for descriptive statistics were performed in SPSS (Version 15.0). To identify the CBCL-DP group we used Latent Class Analysis (LCA). Latent class models were previously fitted for these data by means of an Expectation Maximization (EM) algorithm (Dempster et al., 1977; Heinen, 1996) using the program Latent Gold (Vermunt & Magidson, 2000) using the methods described above with nonparametric bootstrapping as a measure of the significance of the model solution and a reduction in the BIC as the indication of best model fit. The EM algorithm included 10 random sets, a convergence tolerance of 1×10-8 and an EM tolerance of 0.01. Bootstrapping was performed with 500 replications to assess model fit because of the relative sparseness of the data. In these analyses, a bootstrapped p-value greater than 0.05 indicates that the null hypothesis (that the specific model holds true in the population) is valid and the fit of the model is good. Comparisons of models using reduction in the BIC (as noted above) as index of the best model were then performed. One-class through 8-class models were fit. Age and sex covariates were included in the model and then dropped to examine their effect on model fit.

Once an appropriate model was chosen, the output of the LCA was examined for the probability of class membership into different classes and individuals were placed into the latent class with the highest probability of class membership. ROC curve analysis was then performed using SPSS (Version 15.0) with CBCL-PTSP score as the classifier used to predict membership in a given latent class. AUC and 95% confidence intervals around the AUC were computed for each latent class separately for boys and girls.

Results

The graphical results of the LCA are shown in Figures 1a and 1b. Results of the model fitting are presented fully in supplemental Tables 1 and 2. For this sample of caregiver reports a 7-class solution fit the data best. As the number of classes increased, improvements were made in the goodness of fit demonstrated by a decrease in the BIC (from 77318.4 in a 1-class solution to a nadir of 66291.5 in a 7-class solution and rising to 66348.6 in an 8-class solution). Dropping age as a covariate from the 7-class model resulted in further lowering of the BIC (to 66257.0) while dropping sex as a covariate worsened the BIC (66353.6). Consequently, a 7-class model with sex as a covariate was chosen as the best fitting model. For the purposes of this analysis, we were most interested in the two classes which demonstrated elevations on all three scales (AP, AB, and AD). These were Class 6, consisting of 10% of the boys and 6% of the girls, and Class 7, consisting of 4% of the boys and 10% of the girls. Both of these classes had elevations on all three scales, but Class 7 had lower average item endorsement of items that index direct aggression like threatening, destroying property, fighting and attacking but had elevated item endorsement probabilities of items that index more “relational” aggression like being stubborn, disobedient, and screaming. Consistent with work on relational and direct aggression (Crick & Grotpeter, 1995), more boys were placed into the Class 6 (AP, A/D and both direct and relational aggression) while more girls were placed into Class 7 (AP, A/D and mainly relational aggression). Age and probability of class membership for each class are presented in Table 2, separately for males and females. Age was significantly different overall across classes in the model F (6,13) = 2.113, p = 0.047 but post-hoc analysis with Games-Howell correction demonstrated no significant differences for age across the individual classes. There were differences across the classes in the sex of children, Cramer's V = 0.28, p <0.001. Consequently, ROC analyses were done independently for boys and girls.

Table 2
Demographic, Class Names, and AUC Statistics

ROC plots for the score on the CBCL-PTSP scale versus the 7 classes are provided for boys (Figure 2a) and girls (Figure 2b) and the AUC statistics for each latent class are provided in Table 2. The CBCL-PTSP significantly predicts being in Classes 2, 6, or 7. The CBCL-PTSP score significantly predicts not being in Classes 1 or 3 and is at chance for the Classes 4 and 5. The level of the AUC is worth noting. The AUC for Classes 6 and 7 is above 0.9. While these classes are not significantly different (on the basis of overlapping 95% CI), they are significantly higher than the AUC for Class 2. Sensitivities and specificities for various levels of the CBCL-PTSP to predict membership in Classes 6 and 7 for males and females are provided in Table 3. In the secondary analysis where placement into the DP Class was determined without the overlapping CBCL-PTSP items, the resultant ROC analysis demonstrated similar findings for Class 1-5. For the DP class, the AUC remained high for girls (AUC = 0.92, 95% CI = 0.89 – 0.94) and for boys (AUC = 0.93, 95% CI = 0.90 – 0.96) but the AUC for the DP without violence class (class 7) was significantly lower for boys (AUC = 0.83, CI = 0.78-0.87) and for girls (AUC = 0.85, CI = 0.82-0.87), distinguishing it from the DP Class in this analysis.

Figure 2Figure 2
Part A: ROC curves for Latent Classes using CBCL-PTSP scores – males. Each line represents the ROC analysis for a particular Latent Class versus the CBCL-PTSP separately. Part B: ROC curves for Latent Classes using CBCL-PTSP scores – females. ...
Table 3
Sensitivities and specificities for various levels of CBCL-PTSP to predict presence in class 6 or 7

Discussion

These results demonstrate that a deviant score on the CBCL-PTSP scale is a good marker for the profile of high attention problems, aggressive behavior, and anxious/depression that has been called the CBCL-DP. Coupled with the demonstration of Ayer et al. (2009) that the CBCL-PTSP and the CBCL-DP scales can be explained by either two highly correlated latent variables or, more parsimoniously, by one underlying latent variable, the current results provide further convincing evidence that elevations on the CBCL-PTSP scale predict the same behaviors as the CBCL-DP profile. The finding of two classes that closely resemble the CBCL-DP may at first seem to raise questions about the specificity of the profile. However, it is clear that individuals in Class 6 are distinguished from individuals in Class 7 on the basis of different types of aggression – relational versus direct (or overt) (Crick & Grotpeter, 1995). This distinction has been used to clarify that all aggression does not have to result in direct physical attacks on other individuals, but can include verbal, oppositional, and other relational behaviors. The items which distinguish Classes 6 and 7 in the current study are exactly the same items that emerged when our group performed a factor analysis of the aggression scale in a Dutch sample (Ligthart, Bartels, Hoekstra, Hudziak, & Boomsma, 2005). In that work, sex differences were larger for direct aggression than for relational aggression. Consistent with this finding, the current study demonstrates that girls are more likely to be in the class with relational aggression (Class 7) while boys are more likely to be in the class that has both relational and direct aggression (Class 6, the DP Class). The CBCL-PTSP appears to be slightly more predictive of the DP Class than the DP Class without violence when overlapping items are removed, but remains highly predictive of both of these types of children with profound problems with dysregulation. What distinguishes the CBCL-DP from the CBCL-PTSP scale, therefore, is that the CBCL-DP also captures the three components of self-regulation separately and especially identifies those children with more direct aggression. The CBCL-PTSP, on the other hand, also identifies children in the CBCL-DP without direct aggression class until the overlapping items are removed.

This finding has important clinical implications for the children who have been identified with both the CBCL-DP profile and the CBCL-PTSP scale. As noted above, the CBCL-DP has been utilized mainly as a research tool to identify children with pediatric bipolar disorder (Faraone et al., 2005). In contrast, the CBCL-PTSP scale has been hypothesized as a screening device for PTSD in children, or at least a measurement of the severity of symptoms in children with PTSD (Wolfe & Birt, 1997). On the basis of these findings and earlier work (Ayer et al., 2009), we would hypothesize that the reason for the high scores on the CBCL-PTSP in children with PTSD and the reason for the presence of the CBCL-DP profile in samples that have many broad phenotype pediatric bipolar children is because children with both these diagnoses have primary problems with self-regulatory behaviors. This is not to say that children with PTSD and bipolar disorder have the same disorder. Rather, that this scale and this profile can be used as an empirical measure of dysregulation, rather than being tied to specific disorders. Given these findings and previous work suggesting that many if not most children who score highly on the CBCL-PTSP scale do not have trauma histories, we recommend that the name of the CBCL-PTSP scale should be changed to the “CBCL-Dysregulation Short Scale” or CBCL-DSS in order to avoid the use of this scale as a diagnostic tool for PTSD. This recommendation comes from the evidence that the scale appears to function best as a marker of broad dysregulation that can be present in a range of diagnoses, including but not limited to PTSD and bipolar disorder. Further, we recommend this scale be used as a marker of dysregulation rather than to predict or track PTSD symptoms. Similarly, the use of the name CBCL-Dysregulation Profile should discourage the use of this profile to identify individuals with narrow phenotype pediatric bipolar disorder. It does not appear useful in that regard. It may, however, be useful in identifying children with a diagnosis being considered for DSM-V, namely “Temper Dysregulation Disorder with Dysphoria.” This proposed disorder is designed to identify children with severe mood problems but without narrow phenotype pediatric bipolar disorder. Whether the CBCL-DP identifies these children separately from children with other disorders is a topic for further research.

The renaming of a construct has diagnostic and treatment implications, particularly in this period in which the DSM-V is being prepared. Construction of reliable, empirically-based measures of poor self-regulation in children that can be measured across informants, across cultures, across ages, and across gender can inform the DSM revision process (Hudziak, Achenbach, Althoff, & Pine, 2007). The identification here of both a short scale and a longer profile that can screen for “dysregulated” children should then be incorporated within a DSM framework. The idea of “dysregulation” as a construct worthy of identification on its own and without the encumbrance of being attached to other diagnoses bears consideration. The term “dysregulation” is often used clinically in a very general way to describe children who have problems controlling their behaviors, moods, and cognitions despite research demonstrating some separation of these domains (Fitzsimons & Bargh, 2004). Because there has been no accepted “gold standard” for what a “dysregulated” child looks like, both the ability to describe these children clinically and the ability to research them as a group has been hindered. By describing these children with the most commonly used assessment device for child behavior problems, we hope to promote additional discussion and empirical investigation by scientists and practitioners about evidence-based assessment and identification of youth with core regulatory deficits. In our opinion, this profile and this scale appear to apply best as measures of disordered self-regulation with concomitant disturbance in the regulation of affect, behavior, and cognition. Thus, the proposal to name the profile of high attention problems, aggressive behavior, and anxious/depression the CBCL-DP reflects its composition of three empirically derived scales which each measure a component of self-regulatory behavior (attention problems = cognitive dysregulation, aggressive behavior = behavioral dysregulation, anxious/depression = affective dysregulation). Because the CBCL-PTSP measures these same children and the same latent construct, we recommend a befitting name.

Limitations

This research has limitations. The most important limitation is that we have not measured other self-regulatory behaviors in this sample. Other ways to measure self-regulatory capacity include the use of emotional, attentional, and behavior regulation tasks in a sample with the CBCL-DP and/or CBCL-DSS to determine the degree to which these three components of self-regulation are impaired. Although Ayer and colleagues (2009) demonstrated that the highest factor loadings on items loading onto the single dysregulation construct spanned all three dimensions, it will be necessary to conduct additional research to compare these CBCL scales to better-established measures of affective, attentional, and behavioral dysregulation. This research is underway. This study also did not specifically assess for trauma to provide a quantitative estimate of the number of children who score highly on the PTSP scale who do not have a documented trauma history. Further, LCA might not be easily applied in clinical practice. However, the concept behind it is easily implemented. Rather than concentrating only on elevations on individual subscales on the CBCL (or on any other instrument of child behavior for that matter), we recommend an examination of the pattern of responding. If the child has elevations on attention problems, aggressive behavior, and anxious-depression in the borderline or clinical range, they would be considered to have this CBCL-DP profile. The LCA approach simply captures the child who has elevations in the clinical range on attention problems and aggressive behavior but who may have a T-score 1 unit below the cutpoint for anxious-depression. Rather than considering this child qualitatively different from the child who had exactly the same scores on attention problems and aggressive behavior but who was rated 1 point higher on anxious-depression, LCA would classify them together. We encourage this more dimensional approach to examining profiles of behavior (Hudziak et al., 2007). We are further encouraged that future versions of the scoring algorithms for the CBCL may incorporate a return to a profile-based approach that were present in earlier versions of the CBCL (Achenbach, 1993). This is an area of active research that we are performing with Dr. Achenbach and his team. Overall, these data provide evidence that two widely used scales that have been previously associated with specific diagnoses of PTSD and bipolar disorder, respectively, may be better conceptualized as measuring a single broad dysregulation disorder.

Supplementary Material

Supp Data

Acknowledgments

This research was supported in part by National Institute of Mental Health grants K08 MH082116 (R. Althoff, PI) and K08 MH069562 (D. Rettew, PI). The authors thank Dr. Thomas Achenbach for generously sharing his data.

Footnotes

Publisher's Disclaimer: The following manuscript is the final accepted manuscript. It has not been subjected to the final copyediting, fact-checking, and proofreading required for formal publication. It is not the definitive, publisher-authenticated version. The American Psychological Association and its Council of Editors disclaim any responsibility or liabilities for errors or omissions of this manuscript version, any version derived from this manuscript by NIH, or other third parties. The published version is available at www.apa.org/pubs/journals/pas.

Contributor Information

Robert R. Althoff, University of Vermont, Burlington, Vermont.

Lynsay A. Ayer, Medical University of South Carolina, Charleston, South Carolina.

David C. Rettew, University of Vermont, Burlington, Vermont.

James J. Hudziak, University of Vermont, Burlington, Vermont.

References

  • Achenbach TM. Empirically based taxonomy: How to use syndromes and profile types derived from the CBCL/4-18, TRF, and YSR. University of Vermont, Department of Psychiatry; Burlington, VT: 1993.
  • Achenbach TM, Rescorla LA. Manual for the ASEBA School-Age Forms & Profiles. University of Vermont Research Center for Children, Youth, & Families; Burlington, VT: 2001.
  • Akinso W. Rates of Bipolar Diagnosis in Youth Rapidly Climbing, Treatment Patterns Similar to Adults. National Institutes of Health Radio. 2007
  • Althoff RR, Rettew DC, Ayer LA, Hudziak JJ. Latent Class Cross-informant Agreement of the Child Behavior Checklist Dysregulation Profile. submitted. [PMC free article] [PubMed]
  • Althoff RR, Rettew DC, Ayer LA, Sulman JS, Hudziak JJ. Requiem to the CBCL-mania proxy.. Paper presented at the Pediatric Bipolar Conference; Boston, MA. Mar 28-29, 2008.
  • Althoff RR, Rettew DC, Faraone SV, Boomsma DI, Hudziak JJ. Latent class analysis shows strong heritability of the child behavior checklist-juvenile bipolar phenotype. Biol Psychiatry. 2006;60(9):903–911. [PubMed]
  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education Standards for Educational and Psychological Testing. American Educational Research Association; Washington, D.C.: 1999.
  • Ayer L, Althoff R, Ivanova M, Rettew D, Waxler E, Sulman J, et al. Child Behavior Checklist Juvenile Bipolar Disorder (CBCL-JBD) and CBCL Posttraumatic Stress Problems (CBCL-PTSP) scales are measures of a single dysregulatory syndrome. J Child Psychol Psychiatry. 2009;50(10):1291–1300. [PubMed]
  • Boomsma DI, Rebollo I, Derks EM, van Beijsterveldt TC, Althoff RR, Rettew DC, et al. Longitudinal stability of the CBCL-juvenile bipolar disorder phenotype: A study in Dutch twins. Biol Psychiatry. 2006;60(9):912–920. [PubMed]
  • Carlson GA. Who are the children with severe mood dysregulation, a.k.a. “rages”? Am J Psychiatry. 2007;164(8):1140–1142. [PubMed]
  • Colditz GA, Miller JN, Mosteller F. Measuring gain in the evaluation of medical technology. The probability of a better outcome. Int J Technol Assess Health Care. 1988;4(4):637–642. [PubMed]
  • Crick NR, Grotpeter JK. Relational aggression, gender, and social-psychological adjustment. Child Dev. 1995;66(3):710–722. [PubMed]
  • Dempster A, Laird N, Rubin D. Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B. 1977;39:1–38.
  • Diler RS, Birmaher B, Axelson D, Goldstein B, Gill M, Strober M, et al. The Child Behavior Checklist (CBCL) and the CBCL-bipolar phenotype are not useful in diagnosing pediatric bipolar disorder. J Child Adolesc Psychopharmacol. 2009;19(1):23–30. [PMC free article] [PubMed]
  • Faraone SV, Althoff RR, Hudziak JJ, Monuteaux M, Biederman J. The CBCL predicts DSM bipolar disorder in children: a receiver operating characteristic curve analysis. Bipolar Disorders. 2005;7(6):518–524. [PubMed]
  • Fitzsimons GM, Bargh JA. Automatic Self-Regulation. In: Baumeister RF, Vohs KD, editors. Handbook of Self-Regulation. Guilford Press; New York: 2004. pp. 151–170.
  • Galanter CA, Carlson GA, Jensen PS, Greenhill LL, Davies M, Li W, et al. Response to methylphenidate in children with attention deficit hyperactivity disorder and manic symptoms in the multimodal treatment study of children with attention deficit hyperactivity disorder titration trial. J Child Adolesc Psychopharmacol. 2003;13(2):123–136. [PubMed]
  • Goodman L. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika. 1974;61:215–231.
  • Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. [PubMed]
  • Heinen T. Latent class and discrete latent trait models: Similarities and differences. Sage; Thousand Oaks, CA: 1996.
  • Hollingshead AB. Department of Sociology. Yale University; 1975. Four factor index of social status.
  • Holtmann M, Goth K, Wockel L, Poustka F, Bolte S. CBCL-pediatric bipolar disorder phenotype: severe ADHD or bipolar disorder? J Neural Transm. 2008;115(2):155–161. [PubMed]
  • Hudziak JJ, Achenbach TM, Althoff RR, Pine DS. A dimensional approach to developmental psychopathology. International Journal of Methods in Psychiatric Research. 2007;16(S1):S16–S23. [PubMed]
  • Hudziak JJ, Althoff RR, Derks EM, Faraone SV, Boomsma DI. Prevalence and genetic architecture of Child Behavior Checklist-juvenile bipolar disorder. Biol Psychiatry. 2005;58(7):562–568. [PubMed]
  • Kahana SY, Youngstrom EA, Findling RL, Calabrese JR. Employing parent, teacher, and youth self-report checklists in identifying pediatric bipolar spectrum disorders: an examination of diagnostic accuracy and clinical utility. J Child Adolesc Psychopharmacol. 2003;13(4):471–488. [PubMed]
  • Krzanowski WJ, Hand DJ. Monographs on Statistics and Applied Probability. Vol. 111. Taylor and Francis Group, LLC; Boca Raton, FL: 2009. ROC curves for continuous data.
  • Leibenluft E, Charney DS, Towbin KE, Bhangoo RK, Pine DS. Defining clinical phenotypes of juvenile mania. Am J Psychiatry. 2003;160(3):430–437. [PubMed]
  • Ligthart L, Bartels M, Hoekstra RA, Hudziak JJ, Boomsma DI. Genetic contributions to subtypes of aggression. Twin Res Hum Genet. 2005;8(5):483–491. [PubMed]
  • McClellan J, Kowatch R, Findling RL. Practice parameter for the assessment and treatment of children and adolescents with bipolar disorder. J Am Acad Child Adolesc Psychiatry. 2007;46(1):107–125. [PubMed]
  • McCutcheon AL. Latent class analysis. Sage Publications; Newbury Park: 1987.
  • McNeil BJ, Hanley JA. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making. 1984;4(2):137–150. [PubMed]
  • Meyer SE, Carlson GA, Youngstrom E, Ronsaville DS, Martinez PE, Gold PW, et al. Long-term outcomes of youth who manifested the CBCL-Pediatric Bipolar Disorder phenotype during childhood and/or adolescence. J Affect Disord. 2008 [PubMed]
  • Mick E, Biederman J, Pandina G, Faraone SV. A preliminary meta-analysis of the child behavior checklist in pediatric bipolar disorder. Biol Psychiatry. 2003;53(11):1021–1027. [PubMed]
  • NIMH National Institute of Mental Health research roundtable on prepubertal bipolar disorder. J Am Acad Child Adolesc Psychiatry. 2001;40(8):871–878. [PubMed]
  • Nylund KL, Asparouhov T, Muthen BO. Deciding on the number of clases in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling. 2007;14(4):535–569.
  • Ruggiero KJ, McLeer SV. PTSD scale of the Child Behavior Checklist: concurrent and discriminant validity with non-clinic-referred sexually abused children. J Trauma Stress. 2000;13(2):287–299. [PubMed]
  • Siegel B, Vukicevic J, Elliott GR, Kraemer HC. The use of signal detection theory to assess DSM-III-R criteria for autistic disorder. J Am Acad Child Adolesc Psychiatry. 1989;28(4):542–548. [PubMed]
  • Siegel B, Vukicevic J, Spitzer RL. Using signal detection methodology to revise DSM-III-R: re-analysis of the DSM-III-R national field trials for autistic disorder. J Psychiatr Res. 1990;24(4):293–311. [PubMed]
  • Sim L, Friedrich WN, Davies WH, Trentham B, Lengua L, Pithers W. The Child Behavior Checklist as an indicator of posttraumatic stress disorder and dissociation in normative, psychiatric, and sexually abused children. J Trauma Stress. 2005;18(6):697–705. [PubMed]
  • Staton D, Volness LJ, Beatty WW. Diagnosis and classification of pediatric bipolar disorder. J Affect Disord. 2008;105(1-3):205–212. [PubMed]
  • Swets JA. Sensitivities and specificities of diagnostic tests. JAMA. 1982;248(5):548–550. [PubMed]
  • Vermunt JK, Magidson J. Latent GOLD User's Guide. Statistical Innovations Inc.; Belmont, MA: 2000.
  • Volk HE, Todd RD. Does the Child Behavior Checklist juvenile bipolar disorder phenotype identify bipolar disorder? Biol Psychiatry. 2007;62(2):115–120. [PubMed]
  • Wolfe VV, Birt JH. Child sexual abuse. In: Mash E, Terdal L, editors. Assessment of childhood disorders. 3rd ed. Guilford; New York: 1997. pp. 569–623.
  • Wolfe VV, Gentile C, Wolfe DA. The impact of sexual abuse on children: A PTSD formulation. Behav Therapy. 1989;20:215–228.