The effects of latent variables in the development of comorbidity among common mental disorders

doi:10.1002/da.20760

Depress Anxiety. Author manuscript; available in PMC 2012 January 1.

Published in final edited form as:

Depress Anxiety. 2011 January; 28(1): 29–39.

doi: 10.1002/da.20760

PMCID: PMC3228602

NIHMSID: NIHMS254463

The effects of latent variables in the development of comorbidity among common mental disorders

Ronald C. Kessler, PhD,¹ Brian J. Cox, PhD,² Jennifer Greif Green, PhD,³ Johan Ormel, PhD,⁴ Katie A. McLaughlin, PhD,¹ Kathleen Ries Merikangas, Ph.D,⁵ Maria Petukhova, PhD,¹ Daniel S. Pine, MD,⁶ Leo J. Russo, PhD,⁷ Joel Swendsen, PhD,⁸ Hans Ulrich Wittchen,⁹ and Alan M. Zaslavsky, PhD¹

¹Department of Health Care Policy, Harvard Medical School, Boston, MA USA

²Departments of Psychiatry, Psychology and Community Health Sciences at the University of Manitoba, Winnipeg, Manitoba, Canada

³Special Education Program, School of Education, Boston University, Boston, MA

⁴Department of Psychiatry, University Medical Center Groningen; Graduate School of Behavioural and Cognitive Neuroscience & Graduate School for Experimental Psychopathology, University of Groningen, the Netherlands

⁵Intramural Research Program of the National Institutes of Health, National Institute of Mental Health, Bethesda, Maryland, USA

⁶Section on Development and Affective Neuroscience, NIMH-Intramural Research Program, US National Institute of Mental Health, Bethesda, MD USA

⁷Shire Pharmaceuticals Research & Development; Chesterbrook, PA

⁸National Scientific Research Center, Bordeaux, France

⁹Institute of Clinical Psychology and Psychotherapy, Technische Universitaet Dresden, Germany

Correspondence: Ronald C. Kessler, Ph.D., Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston MA 02115 Tel. (617) 432-3587, Fax: (617) 432-3588, Email: kessler/at/hcp.med.harvard.edu

The publisher's final edited version of this article is available at Depress Anxiety

Abstract

Background

Although numerous studies have examined the role of latent predispositions to internalizing and externalizing disorders in the structure of comorbidity among common mental disorders, none examined latent predispositions in predicting development of comorbidity.

Methods

A novel method was used to study the role of latent variables in the development of comorbidity among lifetime DSM-IV disorders in the National Comorbidity Surveys. Broad preliminary findings are briefly presented to describe the method. The method used survival analysis to estimate time-lagged associations among 18 lifetime DSM-IV anxiety, mood, behavior, and substance disorders,. A novel estimation approach examined the extent to which these predictive associations could be explained by latent canonical variables representing internalizing and externalizing disorders.

Results

Consistently significant positive associations were found between temporally primary and secondary disorders. Within-domain time-lagged associations were generally stronger than between-domain associations. The vast majority of associations were explained by a model that assumed mediating effects of latent internalizing and externalizing variables, although the complexity of this model differed across samples. A number of intriguing residual associations emerged that warrant further investigation.

Conclusions

The good fit of the canonical model suggests that common causal pathways account for most comorbidity among the disorders considered. These common pathways should be the focus of future research on the development of comorbidity. However, the existence of several important residual associations shows that more is involved than simple mediation. The method developed to carry out these analyses provides a unique way to pinpoint these significant residual associations for subsequent focused study.

Keywords: Anxiety Disorders, Comorbidity, Epidemiology, Epidemiologic Methods

INTRODUCTION

Comorbidity is the norm among common mental disorders, as more than 50% of people with a mental disorder in a given year meet criteria for two or more disorders.^[1,2] The structure of this comorbidity has been the subject of considerable interest over the past decade. Beginning with an influential paper by Krueger,^[3] numerous researchers have documented that bivariate associations among hierarchy-free anxiety, mood, behavior, and substance disorders can be accounted for by correlated latent predispositions to internalizing and externalizing disorders with division of the internalizing dimension into secondary dimensions of fear (e.g., panic, phobia) and distress (e.g., major depressive episode, generalized anxiety disorder).^[4-9]

These results have been used to argue for a reorganization of the classification of mental disorders in the DSM and ICD diagnostic systems.^[10-14] However, some data suggest that this theoretical structure is insufficiently robust to serve as the basis for such a reorganization.^[4,15] For example, the distinction between fear and distress disorders emerges in some samples but not in others^[4,15-17] and the model fit deteriorates when additional disorders are added or when the model is estimated separately among people at different life-course stages.^[13,15] Nonetheless, the results regarding latent variable associations can be useful in investigating the extent to which risk factors for individual disorders are more accurately conceptualized as risk factors for broad predispositions to multiple disorders, as modeled by the latent dimensions underlying these disorders. Kramer and colleagues,^[18] for example, found that observed gender differences in a number of internalizing and externalizing disorders became statistically insignificant when controls were included for latent internalizing and externalizing dimensions. Although the use of latent variable models in this way is only in its infancy, evidence such as this has the potential to be very valuable in distinguishing between specific and nonspecific risk factors.

One obvious application of this sort involves focusing on the development of comorbidity itself. In particular, while the cross-sectional structure of comorbidity has been examined in a number of studies, we are unaware of attempts to investigate the role of latent dimensions in accounting for the development of comorbidity. Several studies of comorbidity used longitudinal data to determine whether the structure of internalizing and externalizing disorders is stable over time,^[9,15,16] but none investigated whether the presumed underlying structure accounts for the associations between temporally primary disorders and subsequent first onset of comorbid disorders. A number of other longitudinal studies examined temporal progression^[19-22] or sequencing^[23-27] between earlier and later mental disorders, documenting strong persistence of individual disorders over time and significant predictive associations between some but not other temporally primary and later disorders. For example, Fergusson and colleagues^[19] found that childhood conduct disorder but not ADHD predicted subsequent onset of substance disorders, while Beesdo et al. found that temporally primary social anxiety disorder predicted subsequent onset and persistence of major depression.^[28] None of these studies, though, investigated the extent to which associations of earlier disorders with onset of later disorders were mediated by latent variables. Such an analysis could be very useful in identifying potentially modifiable risk pathways.^[29,30]

The methodology used up to now to study latent variable associations underlying the structure of comorbidity are too inflexible to study the development of comorbidity, as the latter requires the use of survival analysis methods in which temporally primary disorders are time-varying covariates. We consequently developed a new method to study the mediating effects of latent variables in accounting for the development of comorbidity. This method was used to analyze data in the National Comorbidity Survey (NCS) and the WHO World Mental Health (WMH) Surveys. The method is described in the current report. Although analysis is still underway, broad preliminary findings are briefly described to illustrate the substantive value of the method.

MATERIALS AND METHODS

Samples

The method described here was applied to three surveys in the NCS family of surveys: the NCS-R, NCS-2, and NCS-A. The NCS-R (National Comorbidity Survey Replication) is a national household survey of the prevalence and correlates of DSM-IV^[31] mental disorders among English-speaking adults in the continental US carried out between 2001-03. A total of 9,282 adults (ages 18+) were interviewed face-to-face. The response rate was 70.9%. Informed consent was obtained before interviewing respondents. Respondents were given a $50 incentive for participation. A probability sub-sample of hard-to-recruit pre-designated respondents was administered a brief telephone non-respondent survey (for a $100 incentive), results of which were used to weight the main sample for non-response bias. The Human subjects committees of Harvard Medical School and the University of Michigan approved procedures for recruitment, consent, and protection of subjects. NCS-R design, field, and weighting procedures are described in more detail elsewhere.^[32]

The NCS-2 is a panel sample obtained by interviewing respondents in the baseline NCS^[33] a decade after the initial 1990-02 assessment. The baseline NCS was a nationally representative US household survey of 8,098 respondents aged 15–54. The response rate was 82.4%. Further details about the NCS design and weighting are reported elsewhere.^[33] NCS-2 sought to trace and re-interview an enriched probability subsample of 5,877 NCS respondents, of whom 5,463 were successfully traced and 5,001 re-interviewed (of the remainder, 166 were deceased and the other 710 either not traced or refused to be interviewed), for a conditional response rate of 87.6%. A propensity score adjustment weight^[34] corrected for baseline discrepancies between the full NCS and the NCS-2.

The NCS-A is a face-to-face survey of adolescents ages 13–17 administered between February 2001 and January 2004 in a dual-frame sample of the continental US.^[35,36] An NCS-A household sample is made up of 904 adolescents from households in the NCS-R (86.8% adolescent response rate). An NCS-A school sample is made up of 9,244 adolescents from a sample of schools in the NCS-R counties (82.6% adolescent response rate). One parent or surrogate (henceforth referred to as a parent) was asked to complete a self-administered questionnaire (SAQ) about each participating adolescent. The SAQ conditional response rate was 82.5–83.7% in the household-school samples. A total of 6,483 adolescent-parent pairs provided data on both adolescent interviews and parent SAQs.

Diagnostic assessment

Diagnoses in the NCS-2, NCS-R, and NCS-A were based on the WHO Composite International Diagnostic Interview (CIDI),^[37] a fully structured lay-administered interview that generates diagnoses according to both ICD-10 and DSM-IV criteria. The 18 lifetime diagnoses considered here include mood disorders (bipolar I-II and sub-threshold disorder [BPD], major depressive episode/dysthymia), anxiety disorders (agoraphobia with or without panic disorder, generalized anxiety disorder [GAD], panic disorder with or without agoraphobia, post-traumatic stress disorder [PTSD], separation anxiety disorder, social phobia, specific phobia), behavior disorders (attention-deficit disorder, referred to below as AD; hyperactivity disorder, referred to below as HD; conduct disorder with covert symptoms [e.g., lying, shoplifting; referred to below as CD1]); conduct disorder with overt symptoms [e.g., bullying, being physically cruel to people, referred to below as CD2]; intermittent explosive disorder [IED]; oppositional-defiant disorder [ODD]), and substance disorders (alcohol abuse with or without dependence, drug abuse with or without dependence). Lifetime prevalence and age-of-onset were retrospectively assessed in the NCS and NCS-A, while NCS-2 respondents were administered an expanded version of the baseline interview that assessed onset and course of disorders between the two surveys. As detailed elsewhere,^[38,39] blinded clinical reappraisal interviews found generally good concordance between DSM-IV diagnoses based on the CIDI and those based on the Structured Clinical Interview for DSM-IV^[40] in the NCS-2 and NCS-R and with the Schedule for Affective Disorders and Schizophrenia for School-Age Children Lifetime Version^[41] in the NCS-A. Organic exclusions but not diagnostic hierarchy rules were used in making diagnoses. The CIDI included retrospective disorder age-of-onset reports based on a special question sequence that has been shown experimentally to improve recall accuracy.^[42] These were used to date lifetime disorder age-of-onset (AOO) retrospectively in all surveys. The NCS-2, in addition, has prospective data on order of onset obtained from the two-waves of panel interviews.

Analysis methods

Defining the content of the latent variables

Exploratory principal axis tetrachoric factor analysis with promax rotation was used to examine bivariate comorbidity to determine if the same structure was found as in previous studies of the structure of comorbidity among the NCS disorders.^[3,43] Clear internalizing and externalizing dimensions were found in all surveys. This structure was very similar to that found in a comparison of the NCS-R data with parallel data obtained in 13 other countries in the WHO World Mental Health (WMH) surveys.^[44] In addition, evidence for sub-factors of fear disorders (panic, phobia, IED) and distress disorders (depression, GAD, PTSD, SAD) within the internalizing factor and for sub-factors of behavior disorders and substance disorders within the externalizing factor was found in the NCS-A, although not in the NCS-2 or NCS-R.

Predicting onset of comorbidity

Discrete-time survival analysis^[45] with person-year as the unit of analysis and a logistic link function^[46] was then used to study associations of temporally primary lifetime disorders with subsequent first onset of later disorders. Each model predicted first onset of one of the 18 DSM-IV/CIDI disorders from information about prior lifetime occurrence of the other 17 disorders, which were treated as time-varying covariates, controlling for respondent age, sex, and race/ethnicity. As noted above, retrospective age-of-onset reports were used to define the predictor disorders as time varying and to define age of onset of the outcome disorders. In the NCS-2, which is a panel sample, results were replicated and compared for prospective assessments (i.e., disorders with first lifetime onsets in the decade between T1 and T2 that were reported for the first time at T2 predicted by lifetime disorders reported at T1) and for pooled prospective and retrospective assessments (i.e., adding to the prospective data information about associations between temporally primary and secondary lifetime disorders that were both reported at T1 or were both reported at T2 to have occurred in the decade between T1 and T2) in an effort to determine if retrospective recall bias led to any distortion in results. As there were 18 disorders in these analyses, a total of 306 coefficients (18 × 17) were estimated (first onset of each disorder predicted by prior lifetime occurrence of the 17 other disorders).

Estimating the mediating effects of latent variables

We then estimated a latent variable model that constrained these 306 coefficients by assuming that they are all mediated by time-varying latent internalizing and externalizing variables or, in the case of the NCS-A, latent fear, distress, behavior, and substance disorder variables. The coefficients in the observed variable model were constrained in the sense that the latent variable model used a smaller number of independent coefficients to reproduce these same associations. This is easiest to grasp by comparing a visual representation of the model for the associations involving observed variables (Figure 1), which has 306 coefficients, with a representation of the model for the two-factor latent internalizing-externalizing model (Figure 1), which has 36 independent coefficients. The latter includes 10 coefficients (one of which is non-independent because the 10 coefficients together perfectly predict the time t latent variable) for 10 time t lifetime internalizing disorders predicting the time t latent internalizing variable, 8 coefficients (one of which is non-independent because the 8 coefficients together perfectly predict the time t latent variable) for the 8 time t lifetime externalizing disorders predicting the time t latent externalizing variable, 4 coefficients (two of which are non-independent because the pair of time t latent variables perfectly predict each of the two time t+1 latent variables) for the time t latent internalizing and externalizing variables predicting the time t+1 latent internalizing and externalizing variables, 10 coefficients for the time t+1 latent internalizing variable predicting first onsets of the 10 time t+1 internalizing disorders, and 8 coefficients for the time t+1 latent externalizing variable predicting first onsets of the 8 time t+1 externalizing disorders. Using similar logic, there are 44 independent coefficients in the four-factor model of fear, distress, behavior, and substance disorders.

Figure 1

Schematic of the multivariate observed variable model¹

It is important to recognize that these constrained latent variable models cannot be estimated using the standard covariance structure analysis programs used in previous studies of the structure of comorbidity, as the number of person-years in the survival analysis data files varies across the 18 outcomes. It is also important to note that identification in this model was different than in the conventional model in that we did not assume the structure assumed in a factor analysis model; that is, a structure in which the latent variables cause the observed disorders and the prediction errors for the observed disorders are assumed to be conditionally independent. Instead, we assumed that the observed disorders are the predictors of the latent variables and that the observed disorders can be exogenously correlated. The basic notion here is that the latent variables, which are referred to as canonical variables,^[47] represent common pathways by which the predictors influence multiple outcomes, with the identification of the common pathways indicated by consistencies in ratios of metric regression coefficients for the predictors across the range of outcomes.

Conventional canonical analysis is carried out using matrix manipulation methods similar to those used in the confirmatory factor analysis models that have previously been used to study the structure of comorbidity.^[48] The situation is much more complex, though, when we are dealing with survival models across a range of outcomes that have first onsets that vary from person to person and year to year. As a result, a special approach was needed to estimate the variant on the standard canonical model used in our analysis. Specifically, iterative methods were used to estimate the coefficients in the model by considering each of the three main parts of the model separately (time t observed variables predicting time t latent variables, time t latent variables predicting time t+1 latent variables, time t+1 latent variables predicting time t+1 observed variables), estimating the coefficients only in one of these three parts at a time while fixing the coefficients in the other two parts of the model to their values in the most recent iteration, and then repeating this process sequentially until the estimates converged. This procedure yields maximum-likelihood estimates of the model parameters. A SAS macro was written to implement these procedures.^[49] Once the coefficients in the model were estimated, a likelihood-ratio χ² test was used to compare the fit of the latent variable model with the observed variable model with 270 (306-36) degrees of freedom in the case of the two-factor model and 264 (306-44) degrees of freedom in the case of the four-factor model.

Importantly, the latent variable model provided a better fit than the observed variable model in all three datasets based on a wide range of conventional fit indices. This was also true when we applied the same approach to the analysis of the pooled cross-national dataset in the WMH surveys.^[44] This then led to the issue of how to interpret the coefficients in the three-part latent variable models. This is a rather complex matter in that none of the individual coefficients can be interpreted directly, as it is the products of the coefficients that are determined rather than the coefficients themselves in the model. For example, referring to Figure 2, the product of coefficients a₁ × b₁ × c₄ is the estimated survival coefficient for the association between temporally primary internalizing disorder I₁ (which, for example, might be specific phobia) and the subsequent first onset of internalizing disorder I₄ (which, for example, might be major depressive episode). However, if all a_n coefficients were multiplied by 2 and all b_n coefficients were divided by 2, this product would be the same. As a result, we need to consider how to make sense of these component coefficients. The focus of this report is on how to do that and on the substantive meaning of these coefficients in the preliminary results obtained up to now in the analysis of the NCS-2 and NCS-A data.

Figure 2

Schematic of the multivariate latent variable model¹

Examining residual effects

We also wanted to go beyond the global comparison of latent and observed variable models to determine if any of the 306 pair-wise associations among predictor disorders and outcome disorders remained significant even after controlling for the latent variables. In other words, we were interested in determining whether there were any particularly strong associations between a specific temporally prior disorder and a later comorbid disorder that were not explained by the general internalizing and externalizing pathways. This was done by generating empirical estimates of the time t latent variables from the a_n coefficients in the models and using these as control variables in 306 separate bivariate survival equations that predicted first onset of each disorder from prior history of each of the other disorders. One problem in using this approach, though, is that simple pair-wise significance tests were inappropriate to evaluate the significance of residual coefficients because probability of false positives increases as number of tests increases. We would expect more than a dozen false positives out of 306 tests using .05-level pair-wise significance tests. A stronger standard of evidence is needed to prevent this problem from occurring. The Bonferroni method and its extensions have been developed to adjust significance levels in pair-wise tests to deal with this problem,^[50] but these are low-power tests that make it difficult to detect true associations unless they are very large.^[51] To address this problem, we used an internal sub-sampling strategy to pinpoint particular pair-wise associations for further investigation. Specifically, we generated the bivariate survival models in the NCS-2 separately among men and women and at four independent life course stages (childhood: ages 4–12; adolescence: ages 13–19; young adulthood: ages 20–29; middle adulthood: ages 30–44). We considered a given pair-wise association substantively significant only if it was statistically significant at the .05 level both in the total sample and either among both males and females or in at least two independent sub-samples with consistent sign patterns and odds-ratios (exponentiated survival coefficients) either greater than or equal to 2.0 or less than or equal to 0.5. In the NCS-A, where we had a restricted age range (13–17), we only considered consistency of results by gender. As the NCS-2 and NCS-A data are both clustered and weighted, the design-based method of jackknife repeated replications (JRR)^[52] was used to calculate standard errors and assess statistical significance. A SAS macro was used for this purpose.^[49]

RESULTS

Bivariate and multivariate associations between earlier and later disorders

As noted above, data analysis is still in progress and we consequently only provide a general overview of preliminary results here to illustrate the uses of the method. When we examined the 306 bivariate survival equations (each with first onset of one of the 18 lifetime outcome disorders predicted by the prior occurrence of one of the other 17 disorders), we found well over 90% to be positive and significant in each of the three datasets, with most odds ratios (exponentiated survival coefficients) in the range 2.0–4.0. Within-domain ORs (e.g., internalizing disorders predicting other internalizing disorders) were generally larger than between-domain ORs (e.g., internalizing disorders predicting externalizing disorders). This largely positive pattern persisted in somewhat attenuated form in the multivariate models that included information about time t history of all other 17 disorders to predict subsequent first onset of each of the 18 disorders.

Multivariate associations in the latent variable model

As noted above in the section on analysis methods, the latent variable model fit the observed data better than did the observed variable model in all three datasets as indicated by standard measures of comparative model fit.^[53] This means that the predicted values of the 306 survival coefficients generated by the coefficients in the latent variable models did not, as a set, differ substantially from the observed values generated from the 18 unrestricted survival equations. Given that the latent variable models are much more parsimonious than the observed variable models, it makes sense to focus on the former rather than the latter in interpreting the data. A difficulty in doing this, though, was noted above in the section on analysis methods: that none of the individual coefficients in the latent variable models can be interpreted directly because of the fact that although the products of the coefficients are determined, the individual values of the coefficients are arbitrary.

Three observations are especially useful in making sense of the latent variable coefficients in the face of this arbitrary nature of the individual coefficients: first, the time t latent variables represent weighted composites of the time t predictor disorders; second, the associations between the time t latent variables and the time t+1 latent variables represent differential multipliers of the effects of the disorders in the time t composites on what has been referred to in the literature^[25] as homotypic (i.e., within-composite) and heterotypic (i.e., between-composite) comorbidity; and, third, the differential importance of the time t composites in predicting the diverse outcomes is represented by the product of the b and c coefficients. We use these three observations to organize our discussion of the latent variable results in the next three subsections.

Time t latent variables as weighted composites

Inspection of Figure 2 makes it clear that the time t latent variables represent weighted composites of the time t predictor disorders. For example, latent variable I_t is defined as the sum of the a coefficients for the lifetime internalizing disorders that the respondent has as of time t. The weights define the relative importance of the predictor disorders (as given by their survival coefficients in predicting the time t latent variables) in predicting the outcome disorders. The model assumes that this relative importance is constant across all outcomes, an assumption that is shown to be consistent with the data by the fact that the model provides a good fit to the observed data. This being the case, the most useful metric in which to consider the a coefficients is one that highlights the disorders with the largest coefficients in each set and that describes the coefficients associated with the other disorders as fractions of these largest coefficients.

Preliminary results regarding the relative magnitude of a coefficients show a number of important consistencies across surveys. For one, temporally primary major depression (MD) is consistently much more important than generalized anxiety disorder (GAD) in predicting onset of later disorders in all the surveys. In a similar way, temporally primary specific phobia (SP) is consistently the most important fear disorder and panic disorder (PD) consistently less important in predicting onset of later disorders in all surveys. These results are true despite the existence of high comorbidities between MD and GAD and between SP and PD. Furthermore, in all surveys substance disorders are by far the least important externalizing disorders in predicting onset of later disorders. Finally, between-disorder variation in the size of a coefficients in all surveys is considerably greater for internalizing than externalizing disorders. In particular, the coefficients associated with temporally primary ADHD, conduct disorder, and oppositional-defiant disorder are fairly comparable in magnitude, suggesting that these disorders have similar predictive effects, whereas the predictive effects of internalizing disorders are more variable.

Differential prediction of homotypic and heterotypic comorbidity

Inspection of Figure 2 makes it clear that the coefficients linking the time t and time t+1 latent variables (the b coefficients) are differential multipliers of the associations between time t predictor disorders in different composites and the subsequent onset of secondary disorders in these same and different composites. For example, if we standardize a coefficients and c coefficients to have weighted (by prevalence of disorders) means of 1.0, the resulting values of b coefficients can be interpreted as averages of the survival coefficients linking temporally primary disorders in a given time t composite to subsequent first onset of disorders in a given time t+1 composite. Based on the results of previous studies,^[25,27] we would expect these coefficients to be larger in predicting homotypic than heterotypic associations. This is, in fact, what we found in all the samples. Temporally primary internalizing disorders were more powerful predictors of subsequent onset of other internalizing disorders than of externalizing, while temporally primary externalizing disorders were more powerful predictors of subsequent onset of other externalizing disorders than of internalizing disorders. However, the specificity was greater for externalizing than internalizing disorders. That is, externalizing disorders were much more powerfully predicted by earlier externalizing than internalizing disorders, whereas internalizing disorders were predicted only slightly more by earlier internalizing than externalizing disorders.

Differential prediction of time t+1 disorders

Comparison of the c coefficients within each composite tells us about the differential effects of the time t composites on first onset of the time t+1 disorders. A striking finding in preliminary analyses of all three datasets is that the c coefficients are remarkably similar in magnitude within each composite. This means that odds of onset of all internalizing disorders are predicted with similar strength by prior history of other disorders as of time t and that all externalizing disorders are predicted with similar strength (i.e., magnitude of the survival coefficients by history of other time t disorders). It is noteworthy in this regard that the c coefficients are much less variable in magnitude than the a coefficients, which means that the disorders considered here differ much more in their importance as predictors of later disorders than as consequences of earlier disorders.

The issue of whether time t+1 internalizing disorders are predicted as strongly as time t+1 externalizing disorders cannot be evaluated by looking only at the c coefficients, as these cross-composite comparative predictive associations are a function of the product of the b coefficients and the c coefficients. Give that, as noted above, the b coefficients are stronger in predicting homotypic than heterotypic comorbidity, it follows that the survival coefficients linking the time t latent composites with the time t+1 disorders will be stronger for the time t internalizing composite than the time t externalizing composite predicting the time t+1 internalizing disorder and vice versa in predicting the time t+1 externalizing disorders.

Variation over the life course

An important consideration in evaluating the results reported so far is that they were based on analyses that assume consistency of coefficients over the entire life course. It’s not clear that this assumption makes sense, especially given the fact that the age-of-onset (AOO) distributions of the disorders considered here differ markedly from the typically very early AOO of some disorders (e.g., ADHD, specific phobia, separation anxiety disorder) to comparatively late AOO of others disorders (e.g., major depression, GAD, substance use disorders).^[54,55] In order to investigate this issue, we replicated the analysis in the prospective NCS-2 data in sub-samples of person-years defined by life course stage (childhood: ages 4–12; adolescence: 13–24; young adulthood: 25-–39; later adulthood: 40+). Results were clear in showing that predictive associations exist in all the age ranges studied, that these associations generally decrease in magnitude over the life course, and that the associations involving prediction of behavior disorders are very unstable beyond adolescence due to the fact that the vast majority of behavior disorders have their first onsets in childhood or adolescence.

Perhaps the most striking aspect of these disaggregated analyses is that we found significant predictive associations between temporally primary behavior disorders and the subsequent first onset of internalizing disorders even in later adulthood despite the fact that other NCS-A analyses showed that the majority of these behavior disorders remitted either in adolescence or early adulthood. Further investigation of this pattern showed that it was due to prior history of remitted behavior disorders predicting first onset of distress disorders in later adulthood. We did not control in these analyses for the various adverse consequences of prior behavior disorders that might still be present in later adulthood (e.g., low educational attainment and consequent low employment status, low earnings, and high financial distress) and still exert an active effect in promoting first onset of other disorders. Nor did we investigate the possibility that sub-threshold manifestations of the ostensibly remitted behavior disorders continued to exist in adulthood and accounted for the predictive effects found in these analyses. As a result, we have no firm basis for saying that some latent predisposition for behavior disorders still present in adulthood promotes onset of later disorders even when the behavior disorders remit. Further analyses aimed at discovering and documenting the effects of intervening adverse consequences are needed to advance our understanding of this data pattern.

Residual associations between specific pairs of disorders

About a dozen residual (i.e., controlling the latent variables) pair-wise time-lagged associations between prior lifetime disorders and subsequent first onset of later disorders passed our test of statistical significance in one or more of the surveys. Most of these involved either within-disorder reciprocal effects (e.g., covert conduct disorder predicting overt conduct disorder), well-known linked pairs of disorders (e.g., panic predicting agoraphobia, depression predicting GAD), or possible diagnostic confusions (e.g., agoraphobia predicting bipolar disorder). In the NCS-A, in comparison, virtually all of these significant residual associations disappeared when we estimated a more complex latent variable model that allowed for four dimensions (distress, fear/anger, behavior, substance). We attribute this to the fact that the NCS-A model was more fine-grained than the model used in the other datasets. There are hints in the data of a small number of other potentially important residual associations existing that are substantively meaningful, but they appear to be few in number and their stability is unclear from the analyses we have carried out so far.

DISCUSSION

Four limitations need to be noted in interpreting the above results. First, diagnoses were based on fully structured lay interviews with the CIDI. The CIDI typically produces more reliable diagnoses than those based on semi-structured clinical interviews^[56] and their prevalence estimates typically correspond well with those based on clinical reappraisal interviews.^[57] However, fully structured interviews, unlike semi-structured clinical interviews, are unable to clarify symptom responses or check questions across disorders to facilitate differential diagnosis, potentially leading to inflated estimates of comorbidity. As comorbidity is the focus of the current analysis, this is an especially important limitation that should be kept in mind when interpreting results. Second, lifetime diagnoses were based on retrospective reports rather than prospective data, probably leading to recall bias that under-estimated prevalence^[58] and distorted age-of-onset estimates^[59] despite special memory-priming methods used in both surveys.^[42] Third, the models were based on the simplifying assumption that the time-lagged associations among mental disorders are additive. Fourth, the models assumed that there are no differential predictive associations related to age of onset or time since onset of the temporally primary disorders. More in-depth analyses to investigate these assumptions go beyond the scope of this preliminary report.

In the context of these limitations, our finding of a two-factor internalizing-externalizing structure among the DSM-IV disorders is consistent with previous research,^[3-5,7-9] but the NCS-2 and NCS-R results do not support the distinction found in the NCS-A and some other previous studies between distress (e.g., depression, GAD, PTSD) and fear (i.e., panic and phobias) disorders within the internalizing domain. Others also failed to find a distinction between distress and fear disorders.^[3,4,17] This less differentiated structure might be due to our focus on lifetime disorders, whereas 12-month disorders were the focus of most studies that distinguished distress and fear disorders. The fact that the more differentiated picture appears in the NCS-A despite the focus on lifetime disorders might be due to a greater proportion of lifetime disorders are being recent disorders in studies of youth than adults.

Our finding of significant time-lagged associations across virtually all pairs of disorders considered here is broadly consistent with evidence of associations between earlier and later disorders in previous longitudinal studies,^{[19,22-25,27]} although most previous studies focused on prevalent cases whereas we studied first onsets. We found, again consistent with previous studies, stronger and more consistent time-lagged associations within (homotypic) than between (heterotypic) the internalizing and externalizing domains. However, again like previous studies, we also found significant between-domain time-lagged associations.^[25,27]

Our analysis then went beyond previous studies to investigate the role of latent variables in the development of comorbidity. We showed that the vast majority of the 306 pair-wise time-lagged associations among the 18 disorders considered here can be explained by a model that assumes the existence of mediating latent internalizing and externalizing variables. The temporally primary disorders constituting the mediating predictor variables vary substantially in importance in predicting secondary disorders, but the good fit of the model shows that the relative importance of these disorders is quite consistent in predicting a wide range of secondary disorders. This suggests that common pathways are involved in these many predictive associations.

Despite the stability of relative importance of predictor disorders within domains, we also found significant between-domain variation in strength of prediction, with consistently stronger predictive associations within (homotypic) than between (heterotypic) domains. However, temporally primary externalizing disorders were more consistently significant predictors of the subsequent onset of internalizing disorders, with predictive effects nearly as large as those of temporally primary internalizing disorders, than temporally primary internalizing disorders were in predicting subsequent onset of externalizing disorders.

Despite the good fit of the latent variable model, we found a number of significant residual associations between particular pairs of temporally primary and secondary disorders. However, these residual associations were much more common in the NCS-2 and NCS-R data, where the best-fitting model assumed the existence of two latent variables (internalizing, externalizing) than in the NCS-A data, where the best-fitting model assumed the existence of four latent variables (fear, distress, behavior, substance). This raises the possibility that the residual associations represent traces of more differentiated dimensions underlying internalizing and externalizing disorders.

Some of the significant residual associations found in the data are quite intriguing and cannot be accounted for by a simple division of internalizing disorders into separate fear and distress sub-domains or externalizing disorders into behavior and substance domains. For example, a stable negative association was found between temporally primary intermittent explosive disorder (IED) and subsequent drug abuse after controlling for scores on the latent internalizing disorder variable. This negative association could be due to externalizing disorders being made up of multiple dimensions, one or more of which is significantly more strongly related to drug abuse than to IED. Consistent with this possibility, evidence exists that multiple dimensions account for associations among externalizing disorders^[60] and that these underlying dimensions are somewhat different for impulsive aggression (i.e., IED) than substance abuse, with sensation seeking, risk taking, and antisocial personality more strongly related to substance abuse than to impulsive aggression, and global psychopathology more strongly related to impulsive aggression than to substance abuse.^[61,62] A more differentiated latent variable model that includes these externalizing sub-dimensions or a model that includes explicit measures of these predispositions might explain the negative association between IED and subsequent drug abuse in our less differentiated latent variable model.

It is important to caution, though, that these few unique significant residual pair-wise associations should be treated as no more than preliminary due to the problem of multiple testing even though we required a high standard of proof to select them. Replication in other datasets is needed before these associations should be considered reliable. Furthermore, even if they are subsequently found to be reliable, their existence should not deflect attention from our main finding: the consistently significant comorbidities found among the 306 disorder-pairs considered in the three datasets are likely due to common underlying processes that should be the focus of future research on the development of comorbidity. More subtle processes doubtlessly exist, but are likely to become manifest more clearly by controlling the effects of the broadly defined latent variables studied here and searching for consistencies in residual associations that can be fleshed out in more focused studies. Implicitly, our results also caution against interpreting pair-wise associations as unique in focused analyses of particular disorder pairs without first demonstrating, as we did here, that they are specific rather than mere realizations of larger nonspecific processes involving a larger set of internalizing or externalizing disorders.

Acknowledgments

We thank Adrian Angold and Jane Costello for helpful comments on an earlier draft of the paper. Data collection for the baseline NCS was supported by the National Institute of Mental Health (NIMH; R01MH46376). The NCS-2 data collection was supported by the National Institute on Drug Abuse (NIDA; R01DA012058). Data collection for the NCS-A was funded by the National Institute of Mental Health (NIMH; U01-MH60220, R01-MH66627, and U01MH060220-09S1) with supplemental support from the National Institute on Drug Abuse (NIDA), the Substance Abuse and Mental Health Services Administration (SAMHSA), the Robert Wood Johnson Foundation (RWJF; grant 044780), and the John W. Alden Trust.

Data analysis for this paper was additionally supported by Shire Pharmaceuticals and NIMH grants R01MH070884, R01MH077883, and U01MH060220, with supplemental support from the Substance Abuse and Mental Health Services Administration (SAMHSA), the Robert Wood Johnson Foundation (RWJF; grant 044780), and the John W. Alden Trust. The work of Merikangas and Pine on this report was supported by the NIMH Intramural Research Program. The work of Zaslavsky on this report was supported by NIMH grant R01-MH66627. The work of Cox on this report was supported by a Canadian Research Chair and an operating grant from the Canadian Institutes of Health Research. A complete list of NCS, NCS-2, and NCS-A publications can be found at http://www.hcp.med.harvard.edu/ncs/.

The NCS, NCS-2, and NCS-A were carried out and are being analyzed in conjunction with the World Health Organization World Mental Health (WMH) Survey Initiative. We thank the staff of the WMH Data Collection and Data Analysis Coordination Centres for assistance with instrumentation, fieldwork, and consultation on data analysis. The WMH Data Coordination Centres have received support from NIMH (R01-MH070884, R13-MH066849, R01-MH069864, R01-MH077883), NIDA (R01-DA016558), the Fogarty International Center of the National Institutes of Health (FIRCA R03-TW006481), the John D. and Catherine T. MacArthur Foundation, the Pfizer Foundation, and the Pan American Health Organization. The WMH Data Coordination Centres have also received unrestricted educational grants from Astra Zeneca, BristolMyersSquibb, Eli Lilly and Company, GlaxoSmithKline, Ortho-McNeil, Pfizer, Sanofi-Aventis, Shire, and Wyeth. A complete list of WMH publications can be found at http://www.hcp.med.harvard.edu/wmh/. The views and opinions expressed in this report are those of the authors and should not be construed to represent the views of any of the sponsoring organizations, agencies, or US government.

Role of the Sponsors: The sponsors had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; or the preparation, review, and approval of the manuscript.

Footnotes

Author Contributions: Dr. Kessler had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Financial Disclosure: As noted above, support for the WMH Surveys was obtained from a variety of government agencies, academic institutions, foundations, and private companies. In addition, Dr. Kessler has been a consultant for GlaxoSmithKline Inc., Kaiser Permanente, Pfizer Inc., Sanofi-Aventis, Shire Pharmaceuticals, and Wyeth-Ayerst; has served on advisory boards for Eli Lilly & Company, Johnson & Johnson Pharmaceuticals, and Wyeth-Ayerst; and has had research support for his epidemiological studies from Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Johnson & Johnson, Ortho-McNeil, Pfizer, and Sanofi-Aventis. Dr. Russo is an employee of Shire Pharmaceuticals. The remaining authors report nothing to disclose.

Conflict of Interest: Dr. Kessler has been a consultant for GlaxoSmithKline Inc., Kaiser Permanente, Pfizer Inc., Sanofi-Aventis, Shire Pharmaceuticals, and Wyeth-Ayerst; has served on advisory boards for Eli Lilly & Company, Johnson & Johnson Pharmaceuticals, and Wyeth-Ayerst; and has had research support for his epidemiological studies from Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Johnson & Johnson, Ortho-McNeil, Pfizer, and Sanofi-Aventis. Dr. Russo is an employee of Shire Pharmaceuticals.

References

1. Demyttenaere K, Bruffaerts R, Posada-Villa J, et al. Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health surveys. JAMA. 2004;291:2581–2590. [PubMed]

2. Kessler RC, Chiu WT, Demler O, et al. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62:617–627. [PMC free article] [PubMed]

3. Krueger RF. The structure of common mental disorders. Arch Gen Psychiatry. 1999;56:921–926. [PubMed]

4. Beesdo K, Hofler M, Gloster A, et al. The structure of common mental disorders: a replication study in a community sample of adolescents and young adults. Int J Methods Psychiatr Res. 2009;18:204–220. [PubMed]

5. Cox BJ, Swinson RP. Instrument to assess depersonalization-derealization in panic disorder. Depress Anxiety. 2002;15:172–175. [PubMed]

6. Krueger RF, Markon KE. Reinterpreting comorbidity: a model-based approach to understanding and classifying psychopathology. Annu Rev Clin Psychol. 2006;2:111–133. [PMC free article] [PubMed]

7. Lahey BB, Rathouz PJ, Van Hulle C, et al. Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. J Abnorm Child Psychol. 2008;36:187–206. [PubMed]

8. Slade T, Watson D. The structure of common DSM-IV and ICD-10 mental disorders in the Australian general population. Psychol Med. 2006;36:1593–1600. [PubMed]

9. Vollebergh WA, Iedema J, Bijl RV, et al. The structure and stability of common mental disorders: the NEMESIS study. Arch Gen Psychiatry. 2001;58:597–603. [PubMed]

10. Andrews G, Goldberg DP, Krueger RF, et al. Exploring the feasibility of a meta-structure for DSM-IV and ICD-11: could it improve utility and validity? Psychol Med. 2009;39:1993–2000. [PubMed]

11. Goldberg DP, Krueger RF, Andrews G, et al. Emotional disorders: cluster 4 of the proposed meta-structure for DSM-IV and ICD-11. Psychol Med. 2009;39:2043–2059. [PubMed]

12. Krueger RF, Markon KE. Understanding Psychopathology: Melding Behavior Genetics, Personality, and Quantitative Psychology to Develop an Empirically Based Model. Curr Dir Psychol Sci. 2006;15:113–117. [PMC free article] [PubMed]

13. Watson D. Rethinking the mood and anxiety disorders: a quantitative hierarchical model for DSM-V. J Abnorm Psychol. 2005;114:522–536. [PubMed]

14. Wittchen HU, Beesdo K, Gloster AT. A new meta-structure of mental disorders: a helpful step into the future or a harmful step back to the past? Psychol Med. 2009;39:2083–2089. [PubMed]

15. Wittchen HU, Beesdo-Baum K, Gloster A, et al. The structure of mental disorders re-examined: is it developmentally stable and robust against additions? Int J Methods Psychiatr Res. 2009;18:189–203. [PubMed]

16. Krueger RF, Caspi A, Moffitt TE, et al. The structure and stability of common mental disorders (DSM-III-R): a longitudinal-epidemiological study. J Abnorm Psychol. 1998;107:216–227. [PubMed]

17. Krueger RF, Finger MS. Using item response theory to understand comorbidity among anxiety and unipolar mood disorders. Psychol Assess. 2001;13:140–151. [PubMed]

18. Kramer MD, Krueger RF, Hicks BM. The role of internalizing and externalizing liability factors in accounting for gender differences in the prevalence of common psychopathological syndromes. Psychol Med. 2008;38:51–61. [PubMed]

19. Fergusson DM, Horwood LJ, Ridder EM. Conduct and attentional problems in childhood and adolescence and later substance use, abuse and dependence: results of a 25-year longitudinal study. Drug Alcohol Depend. 2007;88(Suppl 1):S14–26. [PubMed]

20. Merikangas KR, Zhang H, Avenevoli S, et al. Longitudinal trajectories of depression and anxiety in a prospective community study: the Zurich Cohort Study. Arch Gen Psychiatry. 2003;60:993–1000. [PubMed]

21. Orvaschel H, Lewinsohn PM, Seeley JR. Continuity of psychopathology in a community sample of adolescents. J Am Acad Child Adolesc Psychiatry. 1995;34:1525–1535. [PubMed]

22. Stein MB, Fuetsch M, Muller N, et al. Social anxiety disorder and the risk of depression: a prospective community study of adolescents and young adults. Arch Gen Psychiatry. 2001;58:251–256. [PubMed]

23. Burke JD, Loeber R, Lahey BB, et al. Developmental transitions among affective and behavioral disorders in adolescent boys. J Child Psychol Psychiatry. 2005;46:1200–1210. [PubMed]

24. Copeland WE, Shanahan L, Costello EJ, et al. Childhood and adolescent psychiatric disorders as predictors of young adult disorders. Arch Gen Psychiatry. 2009;66:764–772. [PMC free article] [PubMed]

25. Costello EJ, Mustillo S, Erkanli A, et al. Prevalence and development of psychiatric disorders in childhood and adolescence. Arch Gen Psychiatry. 2003;60:837–844. [PubMed]

26. Feehan M, McGee R, Williams SM. Mental health disorders from age 15 to age 18 years. J Am Acad Child Adolesc Psychiatry. 1993;32:1118–1126. [PubMed]

27. Newman DL, Moffitt TE, Caspi A, et al. Psychiatric disorder in a birth cohort of young adults: prevalence, comorbidity, clinical significance, and new case incidence from ages 11 to 21. J Consult Clin Psychol. 1996;64:552–562. [PubMed]

28. Beesdo K, Bittner A, Pine DS, et al. Incidence of social anxiety disorder and the consistent risk for secondary depression in the first three decades of life. Arch Gen Psychiatry. 2007;64:903–912. [PubMed]

29. Angold A, Costello EJ, Erkanli A. Comorbidity. J Child Psychol Psychiatry. 1999;40:57–87. [PubMed]

30. Jensen PS. Comorbidity and child psychopathology: recommendations for the next decade. J Abnorm Child Psychol. 2003;31:293–300. [PubMed]

31. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. Fourth Edition (DSM-IV) Washington, DC: American Psychiatic Association; 1994.

32. Kessler RC, Berglund P, Chiu WT, et al. The US National Comorbidity Survey Replication (NCS-R): design and field procedures. Int J Methods Psychiatr Res. 2004;13:69–92. [PubMed]

33. Kessler RC, McGonagle KA, Zhao S, et al. Lifetime and 12-month prevalence of DSM-III-R psychiatric disorders in the United States. Results from the National Comorbidity Survey. Arch Gen Psychiatry. 1994;51:8–19. [PubMed]

34. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41–55.

35. Kessler RC, Avenevoli S, Costello EJ, et al. National comorbidity survey replication adolescent supplement (NCS-A): II. Overview and design. J Am Acad Child Adolesc Psychiatry. 2009;48:380–385. [PMC free article] [PubMed]

36. Kessler RC, Avenevoli S, Costello EJ, et al. Design and field procedures in the US National Comorbidity Survey Replication Adolescent Supplement (NCS-A) Int J Methods Psychiatr Res. 2009;18:69–83. [PMC free article] [PubMed]

37. Kessler RC, Üstün TB. The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI) Int J Methods Psychiatr Res. 2004;13:93–121. [PubMed]

38. Kessler RC, Abelson J, Demler O, et al. Clinical calibration of DSM-IV diagnoses in the World Mental Health (WMH) version of the World Health Organization (WHO) Composite International Diagnostic Interview (WMHCIDI) Int J Methods Psychiatr Res. 2004;13:122–139. [PubMed]

39. Kessler RC, Avenevoli S, Green J, et al. National comorbidity survey replication adolescent supplement (NCS-A): III. Concordance of DSM-IV/CIDI diagnoses with clinical reassessments. J Am Acad Child Adolesc Psychiatry. 2009;48:386–399. [PMC free article] [PubMed]

40. First MB, Spitzer RL, Gibbon M, et al. Structured Clinical Interview for DSM-IV Axis I Disorders, Research Version. Non-patient Edition (SCID-I/NP) New York, NY: Biometrics Research, New York State Psychiatric Institute; 2002.

41. Kaufman J, Birmaher B, Brent D, et al. Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL): initial reliability and validity data. J Am Acad Child Adolesc Psychiatry. 1997;36:980–988. [PubMed]

42. Knäuper B, Cannell CF, Schwarz N, et al. Improving accuracy of major depression age-of-onset reports in the US National Comorbidity Survey. Int J Methods Psychiatr Res. 1999;8:39–48.

43. Kessler RC. The prevalence of psychiatric comorbidity. In: Wetzler S, Sanderson WC, editors. Treatment Strategies for Patients with Psychiatric Comorbidity. New York: Wiley; 1997. pp. 23–48.

44. Kessler RC, Ormel J, Petukhova M, et al. Development of lifetime comorbidity in the WHO World Mental Health (WMH) Surveys. Arch Gen Psychiatry. in press.

45. Efron B. Logistic regression, survival analysis, and the Kaplan-Meier curve. J Am Stat Assoc. 1988;83:414–425.

46. Willett JB, Singer JD. Investigating onset, cessation, relapse, and recovery: why you should, and how you can, use discrete-time survival analysis to examine event occurrence. J Consult Clin Psychol. 1993;61:952–965. [PubMed]

47. Tofallis C. Model building with multiple dependent variables and constraints. J Royal Stat Soc Series D: The Statistician. 1999;48:371–378.

48. Thompson B. Canonical Correlation Analysis: Uses and Interpretation. Thousand Oaks, CA: Sage Publications; 1984.

49. SAS Institute Inc. SAS/STAT® Software, Version 9.1 for Windows. Cary, NC: SAS Institute Inc; 2002.

50. Shaffer JP. Multiple Hypothesis Testing. Ann Rev Psychology. 1995;46:561–584.

51. Perneger TV. What’s wrong with Bonferroni adjustments. BMJ. 1998;316:1236–1238. [PMC free article] [PubMed]

52. Wolter KM. Introduction to Variance Estimation. New York: Springer-Verlag; 1985.

53. Bollen KA, Long JS, editors. Testing Structural Equation Models. Newbury Park, CA: Sage; 1993.

54. Kessler RC, Amminger GP, Aguilar-Gaxiola S, et al. Age of onset of mental disorders: a review of recent literature. Curr Opin Psychiatry. 2007;20:359–364. [PMC free article] [PubMed]

55. Kessler RC, Berglund P, Demler O, et al. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62:593–602. [PubMed]

56. Wittchen HU. Reliability and validity studies of the WHO--Composite International Diagnostic Interview (CIDI): a critical review. J Psychiatr Res. 1994;28:57–84. [PubMed]

57. Haro JM, Arbabzadeh-Bouchez S, Brugha TS, et al. Concordance of the Composite International Diagnostic Interview Version 3.0 (CIDI 3.0) with standardized clinical assessments in the WHO World Mental Health surveys. Int J Methods Psychiatr Res. 2006;15:167–180. [PubMed]

58. Moffitt TE, Caspi A, Taylor A, et al. How common are common mental disorders? Evidence that lifetime prevalence rates are doubled by prospective versus retrospective ascertainment. Psychol Med. 2009:1–11. [PMC free article] [PubMed]

59. Simon GE, VonKorff M. Recall of psychiatric history in cross-sectional surveys: implications for epidemiologic research. Epidemiol Rev. 1995;17:221–227. [PubMed]

60. Farmer RF, Seeley JR, Kosty DB, et al. Refinements in the hierarchical structure of externalizing psychiatric disorders: Patterns of lifetime liability from mid-adolescence through early adulthood. J Abnorm Psychol. 2009;118:699–710. [PMC free article] [PubMed]

61. Helfritz L, Stanford M. Personality and psychopathology in an impulsive aggressive college sample. Aggressive Behav. 2006;32:28–37.

62. Sher KJ, Trull TJ. Personality and disinhibitory psychopathology: alcoholism and antisocial personality disorder. J Abnorm Psychol. 1994;103:92–102. [PubMed]