PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Abnorm Child Psychol. Author manuscript; available in PMC 2018 January 1.
Published in final edited form as:
PMCID: PMC5075270
NIHMSID: NIHMS780912

Clinical Decision Making Following Disasters: Efficient Identification of PTSD Risk in Adolescents

Abstract

The present study aimed to utilize a Receiver Operating Characteristic (ROC) approach in order to improve clinical decision-making for adolescents at risk for the development of psychopathology in the aftermath of a natural disaster. Specifically we assessed theoretically-driven individual, interpersonal, and event-related vulnerability factors to determine which indices were most accurate in forecasting PTSD. Furthermore, we aimed to translate these etiological findings by identifying clinical cut-off recommendations for relevant vulnerability factors. Our study consisted of structured phone-based clinical interviews with 2,000 adolescent-parent dyads living within a 5-mile radius of tornados that devastated Joplin, MO, and northern Alabama in Spring 2011. Demographics, tornado incident characteristics, prior trauma, mental health, and family support and conflict were assessed. A subset of youth completed two behavioral assessment tasks online to assess distress tolerance and risk taking behavior. ROC analyses indicated four variables that significantly improved PTSD diagnostic efficiency: Lifetime depression (AUC=.90), trauma history (AUC=.76), social support (AUC=.70), and family conflict (AUC=.72). Youth were 2–3 times more likely to have PTSD if they had elevated scores on any of these variables. Of note, event-related characteristics (e.g., property damage) were not related to PTSD diagnostic status. The present study adds to the literature by making specific recommendations for empirically-based, efficient disaster-related PTSD assessment for adolescents following a natural disaster. Implications for practice and future trauma-related developmental psychopathology research are discussed.

Keywords: Stress disorders, evidence-based assessment, traumatic stress, adolescents, PTSD risk assessment

Youth exposed to disasters are vulnerable to trauma-related psychological problems (Udwin, Boyle, Yule, Bolton, & O'Ryan, 2000; Yule et al., 2000). Nearly half of disaster-exposed youth develop clinically elevated posttraumatic stress disorder (PTSD) symptoms (Adams et al., 2014; Furr, Comer, Edmunds, & Kendall, 2010; La Greca et al., 2013). Untreated PTSD in adolescence can persist into adulthood and result in pervasive functional impairment (Morgan, Scourfield, Williams, Jasper, & Lewis, 2003; Yule et al., 2000). It is crucial to identify youth at highest risk for developing PTSD following disasters and stem this deleterious trajectory via intervention. Similarly, given that many youth are also resilient following exposure to a traumatic event, such as a disaster, and may not warrant intervention, a parallel goal for post-disaster assessment is to identify those who should not be referred to unnecessary services. Many promising trauma-focused treatments exist to address youth psychological problems after disasters among those who are identified in need (Pfefferbaum, Sweeton, Newman, Varma, Nitiéma et al., 2014; Pfefferbaum, Sweeton, Newman, Varma, Noffsinger et al., 2014), but disruptions to community infrastructures can impede connecting these adolescents with mental health services. To meet youth mental health needs and guide resource allocation in the post-disaster context, the field needs clinical decision making tools that are efficient to administer and provide sound information for triage, diagnosis, case conceptualization, and treatment planning.

One approach to stratifying youth by PTSD risk is to assess well-established risk factors for each youth (Adams et al., 2014; Furr et al., 2010; La Greca et al., 2013; Norris et al., 2002). Guided by conceptual models rooted in a bioecological, developmental framework (La Greca, Silverman, Vernberg, & Prinstein, 1996; Weems & Overstreet, 2008), vulnerability factors identified in prior studies (Furr et al., 2010; Norris et al., 2002) can be organized across multiple levels of influence. Individual- or youth-level PTSD vulnerability factors include female gender; ethnic minority status; poverty; sustaining personal injury or severe threat to life; living in a highly disrupted community; high levels of secondary stress; prior trauma history; pre-disaster psychiatric problems (e.g., depression), and poor coping. Some youth-level PTSD risk factors can be assessed via computer-based performance tasks, such as distress tolerance (i.e., ability to persist through affective distress; Lejuez, Kahler, & Brown, 2003) and risk-taking propensity (i.e., tendency to engage in potentially harmful behavior with unpredictable rewards and punishments; Lejuez et al., 2002), which have been linked to PTSD symptoms in maltreated youth (Danielson, Ruggiero, Daughters, & Lejuez, 2010) and trauma-exposed adults (Vujanovic, Bonn-Miller, Potter, Marshall, & Zvolensky, 2011). Interpersonal factors include interpersonal conflict (Meiser-Stedman, Yule, Dalgleish, Smith, & Glucksman, 2005; Pelcovitz, Kaplan, DeRosa, Mandel, & Salzinger, 2000; Roussos et al., 2005; Udwin et al., 2000; Wickrama & Kaspar, 2007) and poor social resources (Stallard & Smith, 2007; Udwin et al., 2000; Vernberg, La Greca, Silverman, & Prinstein, 1996). Event-level factors included extreme, widespread damage (Goenjian et al., 1995; Goenjian et al., 2005); serious, ongoing financial hardship (Wickrama & Kaspar, 2007); and high injury and fatality rates (Bryant, Salmon, Sinclair, & Davidson, 2007; Landolt, Vollrath, Timm, Gnehm, & Sennhauser, 2005; Udwin et al., 2000). Prior findings highlight the importance of evaluating multiple types and sources of influence in predicting post-disaster psychopathology in adolescents.

Studies on post-disaster psychopathology typically report correlations summarizing associations between individual, interpersonal, and event level vulnerability factors and PTSD symptoms, or compare samples with versus without PTSD to find associated vulnerability factors. Such results identify candidate etiological factors, but are less helpful in providing post-disaster healthcare workers a risk-assessment framework. Moreover, a comprehensive assessment of all known vulnerability factors is typically not feasible in the post-disaster milieu. Healthcare providers must select variables and assessment tools that balance reliability and decision-making utility against time and resource burdens for patients and themselves. Thus, guidance is needed to develop efficient screening approaches by integrating multiple pieces of data to inform decisions about clinical care.

The use of receiver operating characteristic (ROC) analyses and related data-driven diagnostic efficiency statistics (Kraemer, 1992a; Youngstrom, 2014) represent a strategy for translating basic, post-disaster vulnerability studies into personalized, clinically useful information for decision-making. Rooted in signal detection theory, Bayesian statistics (Swets, Dawes, & Monahan, 2000) and evidence-based medicine (Chambless & Ollendick, 2001; Cohen, Sanborn, & Shiffrin, 2008; Jenkins, Youngstrom, Washburn, & Youngstrom, 2011), ROC analyses facilitate making empirically-derived algorithms to estimate probabilities of meeting diagnostic criteria for each person in light of his or her clinical profile (Straus, Glasziou, Richardson, & Haynes, 2011). This approach to prediction is more consistent, reliable, and efficient than clinical judgment, which often relies on arbitrary or idiosyncratic strategies for weighting multiple pieces of information to make clinical decisions (Meehl, 1954; Rettew, Lynch, Achenbach, Dumenci, & Ivanova, 2009).

Using ROC-based methods in post-disaster settings can reduce assessment-burden by prioritizing the most diagnostically predictive factors within the assessment protocol (Youngstrom & Frazier, 2013). Utilizing an assessment framework with optimized screening tools can address front-line psychiatrists', psychologists', and other disaster health workers' need to efficiently triage large groups of patients and establish clear plans of action for each person. Through the use of ROC procedures, improved assessment protocols have been developed for a wide range of pediatric mental health disorders including bipolar disorder (Youngstrom, Freeman, & Jenkins, 2009), mood disorders (Youngstrom, 2014), and anxiety disorders (Elkins, Carpenter, Pincus, & Comer, 2014; Van Meter et al., 2014). Most relevant to the present study, ROC approaches have also been applied to improving assessment protocols for PTSD in community (Suliman, Kaminer, Seedat, & Stein, 2005) and outpatient settings (You, Youngstrom, Feeny, Youngstrom, & Findling, in press), paving the way for more targeted referrals and interventions for individual who suffer from these symptoms. The present study seeks to build off these past studies by a) assessing PTSD specifically within a post-disaster context and b) including a wide-range of individual, interpersonal, and event-related vulnerabilities as part of the protocol. Broadening the assessment protocol beyond self-report PTSD symptoms may provide a more accurate assessment of post-trauma functioning (O'Donnell et al., 2008).

This study's overarching aim was to inform decision-making for clinicians and health workers following disasters to identify youth at highest risk for PTSD. The first goal was to identify the most potent predictive factors from a set of multi-method (caregiver report, self-report, behavioral tasks), theoretically and empirically justified candidates applied to disaster-exposed adolescents. The second goal was to generate clinical diagnostic likelihood ratios (DLRs; Jaeschke, Guyatt, & Sackett, 1994; Straus, Glasziou et al., 2011) for PTSD risk. As an extension of this latter goal, we aimed to provide a template for how the results can be used clinically. Specifically, we provide an example of how clinicians can apply DLRs by starting with the likelihood of having a diagnosis of PTSD in the general population (the base rate of PTSD in our sample, reported previously in Adams et al., 2014) and adjusting that estimate based on a vulnerable score on the factors found to be most predictive of PTSD risk. Given different base rates of PTSD for boys (6%), girls (8%), younger (ages 12-14; 6%), and older (ages 15-17; 8%) adolescents in our sample (Adams et al., 2014), we provide separate results for the increased likelihood of experiencing PTSD in each of these groups.

Method

Participants

Participants (n=2,000; 50.9% female) came from communities affected by the Spring 2011 tornadoes (Joplin, Missouri; Alabama). Participants were 12 to 17 years old (M=14.5, SD=1.7), 71% White/Caucasian, 26% Black/African-American, and 4% other per adolescent self-report. See (Adams et al., 2014) for further demographic and disaster details.

Procedure

Study design and procedures are detailed in Ruggiero et al. (2015). Selection criteria prioritized identification of families directly impacted by tornadoes. Latitude/longitude coordinates were used to obtain mailing addresses for households within a 5 mile radius of the impact areas (NOAA, 2011). Eligibility criteria included: 1) family residing at their address at the time of the tornado, 2) an adolescent aged 12 to 17 years and legal guardian, and 3) reliable home Internet access.1 Reliable home Internet access was required because data were collected for a randomized controlled trial evaluating a web-based intervention for disaster-related mental health problems (Ruggiero et al., 2015). Addresses matched to a published phone number were contacted and screened by telephone. Addresses unmatched to a phone number were sent a letter explaining the study and a screening questionnaire that assessed inclusion criteria and requested telephone contacts. Adolescent- parent dyads independently completed structured, standardized computer-assisted telephone interviews conducted in a single interview session by well-trained professional interviewers. These interviews assessed PTSD and depression symptoms, substance use, trauma history, disaster impact, social support, and family conflict approximately 9 months (M=8.8, SD=2.6) following the tornado. The overall cooperation rate--(i.e., number screened/number screened + screen-outs + unknown eligibility)--was 61%. Adolescents received a $15 incentive for the interview. All adolescents were provided log-in information to a study website upon completion of the interview.

When adolescents signed in to the study website subsequent to the interview, they were invited to complete two brief game-like computer tasks (BART-Y and BIRD, described below). In total, 352 adolescents completed these tasks. Completers did not differ from noncompleters on any demographic or clinical variables (all ps>.10).

Measures

Main outcome: PTSD

The National Survey of Adolescents-Replication PTSD module (Resnick, Kilpatrick, Dansky, Saunders, & Best, 1993; Ruggiero et al., 2015) assessed the 17 DSM-IV (APA, 2004) PTSD symptom criteria in adolescents. Only symptoms that had been present at least 2 weeks and since the tornado were assessed. Total scores were then dichotomized as either having PTSD or not based on whether symptom counts in each diagnostic cluster were met and functional impairment was endorsed (see Kilpatrick et al., 2003 for psychometric details).

Event related vulnerability: Disaster impact

Caregivers reported on the family's experiences with the tornado, including property damage (to house, vehicle, furniture, personal item and pet), loss of services for over a week (water, electricity, clean clothing, food, shelter, transportation, and spending money), and interpersonal impact (present for the tornado, feared death, feared for safety of others, fear of injury to self, feared for pets). Responses were dichotomized (0=no/1=yes) and summed; higher scores meant worse severity (range = 0–17; M=3.71; SD=2.61).

Interpersonal-related vulnerability

Social support

The Social Support for Adolescents Scale (SSAS; Seidman et al., 1995) assessed the extent to which youth could turn to their mothers, fathers, siblings, close friends, and peers for 1) emotional social support; 2) instrumental social support; and 3) fun. Responses used a 3-point Likert scale ranging from Not at all to A great deal. Higher scores indicated elevated social support. Global support scores were summed across relationships and averaged across social support types. Total scores ranged from 1.67 to 10 (M=7.09; SD=1.64). Reliability (α=.80) was good, consistent with past research (Birman, Trickett, & Vinokurov, 2002).

Conflict

The Conflict Belief Questionnaire Short Form (CBQ; Prinz, Foster, Kent, & O'Leary, 1979; Robin & Foster, 1989). Adolescents completed the CBQ Short Form, a 20-item true/false measure of parent-adolescent conflict. A total score is computed; higher scores indicate greater conflict. Total scores ranged between 0 and 20 across administrations (M=2.78; SD=3.91). Similar to past research using the CBQ Short Form (Kane & Garber, 2009), this scale had excellent internal consistency in our sample (α=.90).

Individual-related vulnerability

Trauma history

Adolescents reported their experience of five different types of potentially traumatic events including physical assault, physical abuse, witnessing domestic violence, witnessing community violence, and serious accidents. Behaviorally specific prompts probed each trauma type, consistent with reliable and valid questions used in the National Survey of Adolescents (NSA; Kilpatrick et al., 2000). A count of the potentially traumatic event types endorsed by each adolescent in his or her lifetime indexed trauma history severity (range =0–5; M=1.03; SD=1.13).

Distress tolerance

The Behavioral Indicator of Resiliency to Distress (BIRD; Danielson et al., 2010; Lejuez, Daughters, Danielson, & Ruggiero, 2006) is a behavioral task that measures distress tolerance (DT). Adolescents selected a series of moving targets on a computer screen, getting points (when successful) or loud, unpleasant noises (when unsuccessful). Adolescents could exchange their points for a prize at the end of the task. Trials became faster, more challenging, and more aversive as the task progressed. During the final level of testing, adolescents could elect to stop at any time, but quitting hypothetically influenced the magnitude of their prizes. DT was measured by persistence (i.e., total time in minutes to quit) on the final level (range=0.03–7.30; M=2.51; SD=1.93). Higher times indicated higher levels of DT (Lejuez et al., 2006).

Risk taking

The Balloon Analog Risk Task-Youth (BART-Y; Lejuez et al., 2007) assessed risk-taking propensity by accounting for Participants' inflated virtual balloons via mouse click. Each pump earned the participant one point. If a balloon was pumped past its pre-set explosion point, the participant lost all that balloon's points. Each pump increased the likelihood of popping. Adolescents could stop pumping and bank each trial's points. The total number of pumps across 30 balloon trials quantified risk-taking propensity (range=1–203; M=101.17; SD = 45.05).

Substance Use

The CRAFFT Substance Abuse Screening Tool (Knight, Sherritt, Shrier, Harris, & Chang, 2002) is a six-item (yes/no) measure of risky substance use behavior. Higher summed scores indicate greater risk. The present study assessed for lifetime substance abuse. CRAFFT total scores ranged between 0 and 5 (M=0.18; SD=0.65). The CRAFFT reliability (α=0.54) was consistent with prior work using the CRAFFT (Dhalla, Zumbo, & Poole, 2011).

Depression

The National Survey of Adolescents-Replication MDD module (Resnick et al., 1993) assessed the 9 DSM-IV (APA, 2004) depression symptom criteria in adolescents. Symptoms were scored dichotomously (1=present/0=absent). Analyses used lifetime MDD symptom count as a dimensional variable (Range 0–9; M=1.35; SD=1.94). Reliability (α=.80) was comparable to previous research with this measure (Boscarino, Galea, Ahern, Resnick, & Vlahov, 2002).

Data Analytic Approach

Receiver Operating Characteristics (ROC) analyses followed recommendations by Youngstrom (2014). Back-to-back histograms investigated the degree of degeneracy across indicators (i.e., high scores on an indicator variable corresponding to no diagnosis; Van Meter et al., 2014). Areas under the curve (AUC) analyses quantified how well each predictor classified PTSD diagnoses (i.e., sensitivity) and identified cases that did not have PTSD (i.e., specificity). Per Swets' (1988) recommendations, AUC values above 0.70 were deemed adequate predictors of PTSD status. Hanley and McNeil's (1983) test of dependent AUCs tested which indicator had the best predictive accuracy for PTSD diagnoses. For the most discriminating predictors (AUC>.70), we created multilevel diagnostic likelihood ratios based on tertiles (i.e., balanced groups of high, medium, and low scores; Straus, Tetroe, & Graham, 2011). DLRs less than 1 indicate that corresponding predictor scores are less likely to lead to a diagnosis of PTSD. For DLR scores between 0.10 and 0.25, one can be moderately certain that the person does not have a PTSD diagnosis in most settings (Straus, Tetroe, & Graham, 2011). Scores above 1 indicate an increased risk for PTSD diagnosis. In cases, where age and sex correlate to the outcome, logistic regressions were used to examine if the influence of predictors varied as a function of these demographic variables. Similarly, as PTSD-risk following a natural disaster is multifaceted (Furr et al., 2010), and indicators for psychological distress should be studied in combination rather than isolation (Beauchaine & Gatzke-Kopp, 2012), interaction terms between predictor variables were created and logistic regression analyses tested if sets of predictors showed incremental value in identifying PTSD (i.e., interactions between predictor variables were tested). Finally, posterior probabilities were calculated for each vulnerable score range on significant indicators for PTSD. Posterior probabilities represent the increased risk for presenting with PTSD once accounting for a vulnerable score on a given indicator. Posterior probabilities were then used to complete standardized nomograms to demonstrate how clinicians could translate ROC analyses into real time decision making regarding patient-risk for PTSD especially for patients who present with more than one indicator of risk.

Results

Descriptive data is provided in Table 1 and Figure 1. Table 1 presents bivariate correlations between all of the study variables. Of note, females and older adolescents were more likely to report a diagnosis of PTSD since the natural disaster (p < .05). Figure 1 illustrates a series of population pyramids comparing the distribution of different vulnerability scores for cases without PTSD versus those with a PTSD diagnosis. Very few examples of degeneracy (i.e., a high vulnerability score corresponding to no diagnosis) were identified across the indicators.

Figure 1
Population pyramids for predictor variable distributions for those with a diagnosis of PTSD (green bars, right of center axis) versus no diagnosis of PTSD (blue bars, left of center axis). Predictors represented in each cell are as follows: (A) Number ...
Table 1
Correlations between study variables

Of the 2,000 adolescents completing baseline assessments, 126 met diagnostic criteria for PTSD, well exceeding the recommended minimum of 20 cases for estimating diagnostic efficiency parameters (Kraemer, 1992b). Table 2 presents AUC statistics and corresponding Cohen's d scores. Distress tolerance and risk taking propensity did not discriminate PTSD significantly better than chance (p > .05). Furthermore, disaster impact (AUC=.62) and lifetime substance abuse (AUC=.68) were deemed poor predictors (i.e., AUC <.70) of PTSD. The remaining four indicators (lifetime depression, trauma history, social support, family conflict) performed fair or better (Swets, 1988), with effect sizes in the medium (d >.50) to large (d >.80) range (Cohen, 1988). Analyses suggested that none of the indicators varied by sex (p > .05) or age (p > .20). Thus, the potential consequences for each significant indicator were consistent for boys, girls, and all adolescents within our population.

Table 2
Diagnostic efficiency of predictors of PTSD diagnosis

It was next tested which of these four constructs (lifetime depression, trauma history, social support, and family conflict) forecasted the greatest risk for PTSD within our sample (Hanley & McNeil, 1983). Lifetime depression predicted PTSD diagnosis significantly better than trauma history (z=7.07, p<.001), social support (z=10.00, p<.001), and family conflict (z=6.04, p<.001). Trauma history outperformed social support (z=2.34, p<.05); however, post hoc correction rendered this finding nonsignificant. Thus, lifetime depression performed best, and all other variables were functionally equivalent in predicting PTSD diagnosis.

We then examined how different score ranges on each significant indicator corresponded to the likelihood of presenting with a diagnosis of PTSD. Specifically, diagnostic likelihood ratios (DLRs) across three equal cutpoints (tertiles) were calculated (Straus et al., 2011). As shown in Table 3, there is a linear pattern of increasing DLRs (i.e., heightened risk for PTSD) with more vulnerable scores. Scores in the low and medium range across variables indicated that adolescents with these scores were less likely to experience PTSD compared to the base rate in the population (DLRs < 1). Meanwhile, adolescents who scored in the high range were at greater risk for presenting with a diagnosis of PTSD (DLRs > 1; Straus et al., 2011).2

Table 3
Diagnostic Likelihood Ratios Predicting PTSD

To better understand the level of risk associated with a vulnerable score, DLRs from the most vulnerable group were taken and inserted into Table 4. Table 4 demonstrates how the DLRs can be interpreted clinically by transforming the likelihood of having a diagnosis of PTSD in the general population (i.e., prior probability) to the percentage chance of having PTSD with a vulnerable score (posterior probability; Youngstrom, 2014). Because of different base rates for boys, girls, younger (ages 12-14), and older (ages 15-17) adolescents in our sample (Adams et al., 2014), separate results are reported for the increased likelihood of experiencing PTSD in each of these groups in light of vulnerable scores on each indictor. As seen in the posterior probabilities in Table 4, findings were both statistically and clinically significant. Youth were approximately 2 to 3 times more likely to have PTSD if they had elevated scores on any of the four predictor variables.

Table 4
Pretest Probability and Posttest Probability for PTSD Diagnosis based on Vulnerable Scores

Finally, two-way interactions between predictors were examined to test whether having a vulnerable score on more than one indicator for PTSD conferred additional risk. The interaction between social support and family conflict (B=.03, SE=.02; W=4.59, p=.03) was the only significant interaction between the indicator variables (all other ps>.10). Results suggested youth who endorsed low social support (score <6.33) and high levels of conflict with their parents (scores ≥3) were at increased risk for PTSD following natural disasters. Similar to Table 4, Figure 2 displays two probability nomograms that can also be used to highlight how DLRs can be used to interpret increased risk for a given diagnosis. In cases where there are multiple vulnerabilities that explain risk above and beyond one of the vulnerabilities independently (i.e., a significant interaction) clinicians can line probability nomograms side by side (Youngstrom, 2014). As seen in Figure 2, the left side of the left nomogram represents the general population's prior probability (i.e., base rate) for PTSD (7% in our study; Adams et al., 2014). The middle line of the nomogram represents the calculated DLR for a given predictor of PTSD. For instance, in the case of family conflict, the highest tertile of family conflict corresponded to a DLR of 2.04. With this information, clinicians would then draw a straight line from the prior probability through the DLR to calculate the posterior probability. In this case, individuals with high family conflict have a 13% of presenting with PTSD (also shown in Table 4). This posterior probability now becomes the prior probability in the second nomogram (displayed on the right). The DLR for this second nomogram comes from the second significant indicator in the interaction, in this case low social support (DLR = 2.20). We conclude by then repeating the step of drawing a line from the prior probability through the DLR to understand the posterior probability of having both vulnerability scores. As shown in Figure 2, an adolescent high in parental conflict goes from having a 13% probability of having a diagnosis of PTSD to a 25% probability if they also report low levels of social support (nearly a four-fold increase).3

Figure 2
Nomograms for adolescents with high conflict and low social support

Discussion

Given the deleterious effects of PTSD in adolescence, there is great public health utility in reducing risk for its onset in the aftermath of a disaster. Mental health service resources are scarce in disaster-stricken communities, underscoring the value of empirically-based, efficient clinical-decision making tools. Present data show that, in the wake of a natural disaster, lifetime depression and trauma history, as well as social support and family conflict, effectively predict PTSD status among disaster-exposed youth. Depression emerged as the most diagnostically powerful, increasing the odds of PTSD 3.5-fold among youth reporting more than two depression symptoms. Assessing depression history may be highly informative for post-disaster PTSD risk screening. Results also underscore the importance of assessing other psychosocial factors such as interactions with family members, and peer social support. The prominent roles low social support and high family conflict showed for PTSD risk prediction are consistent with a recent meta-analysis of pediatric PTSD risk factors (Trickey, Siddaway, Meiser-Stedman, Serpell, & Field, 2012), supporting potential protective effects of social connectedness in the aftermath of trauma (Allen, Marcelin, Schmitz, Hausman, & Shultz, 2012; Landau, 2013; Norris & Stevens, 2007). The current study extends research by providing specific cut score ranges and DLRs to promote individualized risk probability estimation for disaster-exposed youth.

Although it is not surprising that trauma and mental health history are clinically useful in predicting post-disaster PTSD onset, it was surprising that individual-related factors would have greater diagnostic utility than impact of the disaster itself (i.e., event-related vulnerability). These findings highlight the utility of the ROC approach. A plethora of research has shown that traumatic event incident characteristics, such as degree of property damage (Goenjian et al., 1995; Goenjian et al., 2005; van den Berg, Wong, van der Velden, Boshuizen, & Grievink, 2012), are significantly related to post-disaster mental health symptoms (Goenjian et al., 2001; Parson, Fussell, Rhodes, & Waters, 2012; Zhang, Shi, Wang, & Liu, 2011). The current study suggests that while such disaster incident characteristics may contribute to the etiology and/or maintenance of PTSD per prior research, assessing these variables in post-disaster screenings may not improve prediction of PTSD beyond chance—and other variables are more effective for identifying those at highest risk.

It was also somewhat surprising that other individual characteristics (distress tolerance and risk-taking propensity) that have been previously empirically linked with PTSD risk did not improve diagnostic accuracy in this post-disaster sample. The extant literature suggests these factors may still be relevant in post-disaster mental health. Assessment of these characteristics may help in intervention tailoring (Zatzick et al., 2011) and/or detecting other mental health outcomes (e.g., externalizing problems; Daughters et al., 2009). There has been a recent emphasis on the utility of behavioral assessment tasks, including those that incorporate physiological measures, to potentially help inform empirically based assessment in mental health (Youngstrom & de Los Reyes, 2015). Additional research is needed to ultimately determine the utility of behavioral tasks and physiological measures in PTSD risk prediction.

Study strengths include the behaviorally-specific and comprehensive assessment of trauma history and the calculation of PTSD rates by gender and age group in our sample. The ROC approach also reminds clinicians to consider base rates of target problems, which vary by setting and context (Meehl, 1954; Straus, Tetroe, & Graham, 2011). For example, the base rate of PTSD in a clinical setting (e.g., 7.2–14.4%; Zimmerman & Mattia, 1999) is higher than community base rates (5%; Kilpatrick et al., 2003; McLaughlin et al., 2013), including among disaster-exposed community populations (6.7% in this sample; Adams et al., 2014). The DLRs provide locally contextualized interpretation in the form of a Bayesian posterior probability.

Despite these strengths, there are also noteworthy limitations. First, the length of time since the tornado for the initial post-disaster assessment in the current study (average of 8 months) prevented assessment of pre-existing vulnerability factors and symptoms immediately following the disaster. It is possible that certain factors such as social support or family dynamics may have been strong before the storm or in its immediate aftermath, then deteriorated by the time the assessments took place. Alternatively, families may have experienced temporary increases or breakdowns in support and conflict immediately following the storm that resolved by the assessments in this study. Thus the utility of these factors in predicting PTSD may vary by timing of the assessment—a hypothesis that should be tested in future studies. Second, a relatively lower number of youth who completed the behavioral assessment tasks in comparison to those who completed the interviews, which may prevent definitive conclusions from being drawn about the utility of behavioral assessment tasks in PTSD risk assessment, as noted above. Relatedly, while the multi-method approach of the study was a strength, findings indicated that only self-report measures predicted self-reported PTSD diagnostic status. Thus, shared method variance between the measures may have contributed to the strong effects seen for many of the self-report measures, and the null findings concerning distress tolerance and risk-taking propensity in predicting PTSD status. Because most assessment batteries rely on self-reported data, understanding which self-report data is most valuable in forecasting PTSD within a post-disaster context carries significant clinical value.

Post-Disaster PTSD Risk Assessment Practice Recommendations

One of the most effective strategies for achieving accurate and efficient mental health risk prediction is the use of psychological measures that are low burden in their cost and administration time and high yield in their output (i.e., low burden:high yield, LB:HY). These considerations are particularly important following disasters, when efficient resource allocation is key during recovery. Empirically-based clinical decision making—ROC analyses in particular—help achieve LB:HY goal by winnowing down the potential list of variables to screen to those most likely to inform PTSD risk status. Our ROC analyses here show that youth with high (problematic) scores across four key risk factors (lifetime depression, trauma history, social support, and family conflict) can be identified via brief sets of questions. Consideration of these four factors also will allow public health workers to triage a large group of people into a low risk category correctly, providing more fine-grained probability estimates for higher risk individuals based on presence or absence of specific risk factors.

Practical considerations for how to assess the four key post-disaster PTSD risk factors identified here (depression, trauma history, social support, family conflict) will depend somewhat on the resources (e.g., time, personnel, cost, computer and/or reliable internet access) available in the particular post-disaster context in which the recommendations are being applied. These four variables were assessed in the current study via phone-based interview. Each of the key risk factors identified here also may be feasibly and validly assessed using paper-and-pencil questionnaires or tablet-based surveys. Once these values are obtained, scores can be compared to the vulnerability scores reported in Tables 3 and and44 to yield a DLR and a Posttest Probability, as described earlier, to guide clinical decisions about intervention and referrals. Future studies should evaluate the correspondence between other publically available standardized measures for assessing these constructs and the measures used in the current study in predicting PTSD risk. Additional work may involve development of tools that can automate the process of calculating risk scores for high volume screening events in the post-disaster context.

Implications for Trauma-Related Developmental Psychopathology Research

Current findings also inform future trauma-related developmental psychopathology research directions to further evaluate and improve PTSD risk assessment among youth. Given the relevance of developmentally sensitive windows with regard to trauma exposure, the four identified potent predictors in this study, and their respective cut-scores, should be evaluated in prospective investigations across different age groups of youth with future disasters and mass traumatic events. Future studies can similarly inform the ideal timing of post-disaster PTSD risk assessment, which is a commonly posed question in practice (Pfefferbaum et al., 2014ab). In addition, given that changes to the PTSD diagnostic criteria have occurred since the execution of the current study (e.g., DSM-5 PTSD includes four clusters versus three clusters of symptoms), it will be important to determine if the results presented here replicate in a post-disaster youth sample assessed with the DSM-5 PTSD criteria. Finally, ROC analyses may be applied to other research data sets and other types of traumatic events to similarly derive clinically applicable information for a variety of problems and populations (Youngstrom, Choukas-Bradley, Calhoun, & Jensen-Doss, 2014).

ACKNOWLEDGEMENTS

This work was supported by 1R21MH086313 (PI: Danielson) and 1R01MH081056 (PI: Ruggiero) from the National Institute of Mental Health (NIMH). The preparation of this manuscript was supported by R01DA031285 (PI: Danielson from the National Institute on Drug Abuse (NIDA), P50AA010761 (PI: Becker) from the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and T32MH018869 (PIs: Danielson & Kilpatrick) from the NIMH. Views expressed herein are those of the authors and do not necessarily represent the official views of NIMH, NIDA, or NIAAA.

Footnotes

The authors declare they have no conflict of interest.

Ethical approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent: Informed consent was obtained from all individual participants included in the study.

1Only 2.1% of all adolescents contacted were ruled out due to Internet access, suggesting that our inclusion criteria did not have an adverse impact on the generalizability of our results.

2In addition to the methods outlined by Strauss and colleagues regarding tertiles, Kraemer's (1992a) method to optimize thresholds was also examined. Results using Kraemer's approach were remarkably similar to the cutpoint in the high categories (low for social support) using the initial method. The one exception was depressive symptoms, wherein a cutpoint of greater than 3, opposed to 2, symptoms resulted in the highest kappa (0.70).

3It should be noted that this new probability is not a true assessment of the interaction between social support and conflict. Instead, Figure 2 provides a quick heuristic for interested clinicians to calculate an estimated range of the additive consequences of having a significant interaction between two vulnerabilities.

References

  • Adams ZW, Sumner JA, Danielson CK, McCauley JL, Resnick HS, Gros K, Ruggiero KJ. Prevalence and predictors of PTSD and depression among adolescent victims of the Spring 2011 tornado outbreak. Journal of Child Psychology and Psychiatry. 2014;55:1047–1055. [PMC free article] [PubMed]
  • Allen A, Marcelin LH, Schmitz S, Hausman V, Shultz JM. Earthquake impact on Miami Haitian Americans: The role of family/social connectedness. Journal of Loss and Trauma. 2012;17:337–349.
  • APA . Diagnostic and Statistical Manual of Mental Disorders. revised 4th ed. American Psychiatric Association; Washington, DC: 2004.
  • Beauchaine TP, Gatzke-Kopp LM. Instantiating the multiple levels of analysis perspective in a program of study on externalizing behavior. Development and Psychopathology. 2012;24:1003–1018. [PMC free article] [PubMed]
  • Birman D, Trickett EJ, Vinokurov A. Acculturation and adaptation of Soviet Jewish refugee adolescents: Predictors of adjustment across life domains. American Journal of Community Psychology. 2002;30:585–607. [PubMed]
  • Boscarino J, Galea S, Ahern J, Resnick H, Vlahov D. Utilization of mental health services following the September 11th terrorist attacks in Manhattan, New York City. International Journal of Emergency Mental Health and Human Resilience. 2002;4:143–155. [PubMed]
  • Bryant RA, Salmon K, Sinclair E, Davidson P. Heart rate as a predictor of posttraumatic stress disorder in children. General Hospital Psychiatry. 2007;29:66–68. [PubMed]
  • Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology. 2001;52:685–716. [PubMed]
  • Cohen AL, Sanborn AN, Shiffrin RM. Model evaluation using grouped or individual data. Psychonomic Bulletin & Review. 2008;15:692–712. [PubMed]
  • Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Lawrence Earlbaum Associates; Hillsdale, NJ: 1988.
  • Danielson CK, Ruggiero KJ, Daughters SB, Lejuez CW. Distress tolerance, risk-taking propensity, and PTSD symptoms in trauma-exposed youth: Pilot study. The Behavior Therapist. 2010;33:28–34.
  • Daughters SB, Reynolds EK, MacPherson L, Kahler WW, Danielson CK, Zvolensky M, Lejuez CW. Negative reinforcement and early adolescent externalizing and internalizing symptoms: The moderating role of gender and ethnicity. Behavior Research and Therapy. 2009;47:198–205. [PMC free article] [PubMed]
  • Dhalla S, Zumbo BD, Poole GA. A review of the psychometric properties of the CRAFFT instrument: 1999–2010. Current drug abuse reviews. 2011;4:57–64. [PubMed]
  • Elkins RM, Carpenter AL, Pincus DB, Comer JS. Inattention symptoms and the diagnosis of comorbid attention-deficit/hyperactivity disorder among youth with generalized anxiety disorder. Journal of Anxiety Disorders. 2014;28:754–760. [PMC free article] [PubMed]
  • Furr JM, Comer JS, Edmunds JM, Kendall PC. Disasters and youth: A meta-analytic examination of posttraumatic stress. Journal of Consulting and Clinical Psychology. 2010;78:765–780. [PubMed]
  • Goenjian AK, Molina L, Steinberg AM, Fairbanks LA, Alvarez ML, Goenjian HA, Pynoos RS. Posttraumatic stress and depressive reactions among Nicaraguan adolescents after Hurricane Mitch. American Journal of Psychiatry. 2001;158:788–794. [PubMed]
  • Goenjian AK, Pynoos R, Steinberg A, Najarian L, Asarnow J, Karayan I, Fairbanks LA. Psychiatric comorbidity in children after the 1988 earthquake in Armenia. Journal of the American Academy of Child & Adolescent Psychiatry. 1995;34:1174–1184. [PubMed]
  • Goenjian AK, Walling D, Steinberg AM, Karayan I, Najarian LM, Pynoos R. A prospective study of posttraumatic stress and depressive reactions among treated and untreated adolescents 5 years after a catastrophic disaster. The American Journal of Psychiatry. 2005;162:2302–2308. [PubMed]
  • Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148:839–843. [PubMed]
  • Jaeschke R, Guyatt G, Sackett D. Users' guides to the medical literature: III. How to use an article about a diagnostic test: B: What are the results and will they help me in caring for my patients? Journal of the American Medical Association. 1994;271:703–707. [PubMed]
  • Jenkins MM, Youngstrom EA, Washburn JJ, Youngstrom JK. Evidence-based strategies improve assessment of pediatric bipolar disorder by community practitioners. Professional Psychology: Research and Practice. 2011;42:121–129. [PMC free article] [PubMed]
  • Kane P, Garber J. Parental depression and child externalizing and internalizing symptoms: Unique effects of fathers' symptoms and perceived conflict as a mediator. Journal of Child and Family Studies. 2009;18:465–472.
  • Kilpatrick DG, Acierno R, Saunders BE, Resnick HS, Best CL, Schnurr PP. Risk factors for adolescent substance abuse and dependence: Data from a national sample. Journal of Consulting and Clinical Psychology. 2000;68:19–30. [PubMed]
  • Kilpatrick DG, Ruggiero KJ, Acierno RE, Saunders BE, Resnick HS, Best CL. Violence and risk of PTSD, major depression, substance abuse/dependence, and comorbidity: Results from the National Survey of Adolescents. Journal of Consulting and Clinical Psychology. 2003;71:692–700. [PubMed]
  • Knight JR, Sherritt L, Shrier LA, Harris SK, Chang G. Validity of the CRAFFT substance abuse screening test among adolescent clinic patients. Archives of Pediatric & Adolescent Medicine. 2002;156:607–614. [PubMed]
  • Kraemer HC. Evaluating medical tests: Objective and quantitative guidelines. Sage; Newbury Park, CA: 1992a.
  • Kraemer HC. Reporting the size of effects in research studies to facilitate assessment of practical or clinical significance. Psychoneuroendocrinology. 1992b;17:527–536. [PubMed]
  • La Greca AM, Lai BS, Llabre MM, Silverman WK, Vernberg EM, Prinstein MJ. Children's postdisaster trajectories of PTS symptoms: Predicting chronic distress. Child & Youth Care Forum. 2013;42:351–369. [PMC free article] [PubMed]
  • La Greca AM, Silverman WK, Vernberg EM, Prinstein MJ. Symptoms of posttraumatic stress in children after Hurricane Andrew: A prospective study. Journal of Consulting and Clinical Psychology. 1996;64:712–723. [PubMed]
  • Landau JL. Handbook of Family Resilience. Springer; New York: 2013. Family and community resilience relative to the experience of mass trauma: Connectedness to family and culture of origin as the core components of healing; pp. 459–480.
  • Landolt MA, Vollrath M, Timm K, Gnehm HE, Sennhauser FH. Predicting posttraumatic stress symptoms in children after road traffic accidents. Journal of the American Academy of Child & Adolescent Psychiatry. 2005;44:1276–1283. [PubMed]
  • Lejuez CW, Aklin W, Daughters S, Zvolensky M, Kahler C, Gwadz M. Reliability and validity of the youth version of the Balloon Analogue Risk Task (BART-Y) in the assessment of risk-taking behavior among inner-city adolescents. Journal of Clinical Child and Adolescent Psychology. 2007;36:106–111. [PubMed]
  • Lejuez CW, Daughters SB, Danielson CK, Ruggiero KJ. Unpublished manual. 2006. The Behavioral Indicator of Resiliency to Distress (BIRD)
  • Lejuez CW, Kahler CW, Brown RA. A modified computer version of the Paced Auditory Serial Addition Task (PASAT) as a laboratory-based stressor. The Behavior Therapist. 2003;26:290–293.
  • Lejuez CW, Read JP, Kahler CW, Richards JB, Ramsey SE, Stuart GL, Brown RA. Evaluation of a behavioral measure of risk taking: The Balloon Analogue Risk Task (BART) Journal of Experimental Psychology: Applied. 2002;8:75–84. [PubMed]
  • McLaughlin KA, Koenen KC, Hill ED, Petukhova M, Sampson NA, Zaslavsky AM, Kessler RC. Trauma exposure and posttraumatic stress disorder in a national sample of adolescents. Journal of the American Academy of Child & Adolescent Psychiatry. 2013;52:815–830. [PMC free article] [PubMed]
  • Meehl PE. Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press; Minneapolis, MN: 1954.
  • Meiser-Stedman RA, Yule W, Dalgleish T, Smith P, Glucksman E. The role of the family in child and adolescent posttraumatic stress following attendance at an emergency department. Journal of Pediatric Psychology. 2005;31:1–6. [PubMed]
  • Morgan L, Scourfield J, Williams D, Jasper A, Lewis G. The Aberfan disaster: 33-year follow-up of survivors. British Journal of Psychiatry. 2003;182:532–536. [PubMed]
  • NOAA NCDC Storm Events Database. 2011 Retrieved February 11, 2015, from http://www.crh.noaa.gov/sgf.
  • Norris FH, Friedman MJ, Watson PJ, Byrne CM, Diaz E, Kaniasty K. 60,000 disaster victims speak: Part I. An empirical review of the empirical literature, 1981–2001. Psychiatry. 2002;65:207–239. [PubMed]
  • Norris FH, Stevens SP. Community resilience and the principles of mass trauma intervention. Psychiatry. 2007;70:320–328.
  • O'Donnell ML, Creamer MC, Parslow R, Elliott P, Holmes AC, Ellen S, Bryant RA. A predictive screening index for posttraumatic stress disorder and depression following traumatic injury. Journal of Consulting and Clinical Psychology. 2008;76:923–932. [PubMed]
  • Parson C, Fussell E, Rhodes J, Waters M. Five years later: Recovery from post traumatic stress and psychological distress among low-income mothers affected by Hurricane Katrina. Social Science & Medicine. 2012;74:150–157. [PMC free article] [PubMed]
  • Pelcovitz D, Kaplan SJ, DeRosa RR, Mandel FS, Salzinger S. Psychiatric disorders in adolescents exposed to domestic violence and physical abuse. The American Journal of Orthopsychiatry. 2000;70:360–369. [PubMed]
  • Pfefferbaum B, Sweeton JL, Newman E, Varma V, Nitiéma P, Shaw JA, Noffsinger MA. Child disaster mental health interventions: Part I. Techniques, outcomes, and Methodological considerations. Disaster Health. 2014;2:46–57. [PMC free article] [PubMed]
  • Pfefferbaum B, Sweeton JL, Newman E, Varma V, Noffsinger MA, Shaw JA, Nitiéma P. Child disaster mental health interventions: Part II. Timing of implementation, delivery settings and providers, and therapeutic approaches. Disaster Health. 2014;2:58–67. [PMC free article] [PubMed]
  • Prinz RJ, Foster S, Kent RN, O'Leary KD. Multivariate assessment of conflict in distressed and nondistressed mother-adolescent dyads. Journal of Applied Behavioral Analysis. 1979;12:691–700. [PMC free article] [PubMed]
  • Resnick H, Kilpatrick D, Dansky B, Saunders B, Best C. Prevalence of civilian trauma and posttraumatic stress disorder in a representative national sample of women. Journal of Consulting and Clinical Psychology. 1993;61:984–991. [PubMed]
  • Rettew DC, Lynch AD, Achenbach TM, Dumenci L, Ivanova MY. Meta-analyses of agreement between diagnoses made from clinical evaluations and standardized diagnostic interviews. International Journal of Methods in Psychiatric Research. 2009;18:169–184. [PubMed]
  • Robin A, Foster S. Negotiating parent-adolescent conflict. Guilford; New York: 1989.
  • Roussos A, Goenjian AK, Steinberg AM, Sotiropoulou C, Kakaki M, Kabakos C, Manouras V. Posttraumatic stress and depressive reactions among children and adolescents after the 1999 earthquake in Ano Liosia, Greece. The American Journal of Psychiatry. 2005;162:530–537. [PubMed]
  • Ruggiero KJ, Davidson TM, McCauley J, Gros KS, Welsh K, Price M, Amstadter AB. Bounce Back Now! Protocol of a population-based randomized controlled trial to examine the efficacy of a web-based intervention with disaster-affected families. Contemporary Clinical Trials. 2015;40:138–149. [PMC free article] [PubMed]
  • Seidman E, Allen L, Aber JL, Mitchell C, Feinman J, Yoshikawa H, Roper GC. Development and validation of adolescent-perceived microsystem scales: Social support, daily hassles, and involvement. American Journal of Community Psychology. 1995;23:355–388. [PubMed]
  • Stallard P, Smith E. Appraisals and cognitive coping styles associated with chronic post-traumatic symptoms in child road traffic accident survivors. Journal of Child Psychology and Psychiatry. 2007;48:194–201. [PubMed]
  • Straus S, Glasziou P, Richardson W, Haynes R. Evidence-based medicine: How to practice and teach EBM. 4th ed. Churchill Livingstone; New York, NY: 2011.
  • Straus SE, Tetroe JM, Graham ID. Knowledge translation is the use of knowledge in healthcare decision making. Journal of Clinical Epidemiology. 2011;64:6–10. [PubMed]
  • Suliman S, Kaminer D, Seedat S, Stein D. Assessing post-traumatic stress disorder in South African adolescents: Using the child and adolescent trauma survey (CATS) as a screening tool. Annals of General Psychiatry. 2005;4:10. [PMC free article] [PubMed]
  • Swets J. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285–1293. [PubMed]
  • Swets JA, Dawes RM, Monahan J. Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest. 2000;1:1–26. [PubMed]
  • Trickey D, Siddaway AP, Meiser-Stedman R, Serpell L, Field AP. A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clinical Psychology Review. 2012;32:122–138. [PubMed]
  • Udwin O, Boyle S, Yule W, Bolton D, O'Ryan D. Risk factors for long-term psychological effects of a disaster experienced in adolescence: Predictors of post traumatic stress disorder. Journal of Child Psychology and Psychiatry. 2000;41:969–979. [PubMed]
  • van den Berg B, Wong A, van der Velden PG, Boshuizen HC, Grievink L. Disaster exposure as a risk factor for mental health problems, eighteen months, four and ten years post-disaster: A longitudinal study. BMC Psychiatry. 2012;12:147. [PMC free article] [PubMed]
  • Van Meter A, Youngstrom E, Youngstrom JK, Ollendick T, Demeter C, Findling RL. Clinical Decision Making About Child and Adolescent Anxiety Disorders Using the Achenbach System of Empirically Based Assessment. Journal of Clinical Child & Adolescent Psychology. 2014;43:552–565. [PMC free article] [PubMed]
  • Vernberg EM, La Greca AM, Silverman WK, Prinstein MJ. Prediction of posttraumatic stress symptoms in children after Hurricane Andrew. Journal of Abnormal Psychology. 1996;105:237–248. [PubMed]
  • Vujanovic A, Bonn-Miller M, Potter C, Marshall E, Zvolensky M. An evaluation of the relation between distress tolerance and posttraumatic stress within a trauma-exposed sample. Journal of Psychopathology and Behavioral Assessment. 2011;33:129–135. [PMC free article] [PubMed]
  • Weems CF, Overstreet S. Child and adolescent mental health research in the context of Hurricane Katrina: An ecological needs-based perspective and introduction to the special section. Journal of Clinical Child & Adolescent Psychology. 2008;37:487–494. [PubMed]
  • Wickrama K, Kaspar V. Family context of mental health risk in tsunami-exposed adolescents: Findings from a pilot study in Sri Lanka. Social Science & Medicine. 2007;64:713–723. [PubMed]
  • You SD, Youngstrom EA, Feeny NC, Youngstrom JK, Findling RL. Comparing the diagnostic accuracy of five instruments for detecting posttraumatic stress disorder in youth. Journal of Clinical Child & Adolescent Psychology. in press. [PMC free article] [PubMed]
  • Youngstrom EA. A primer on Receiver Operating Characteristic analysis and diagnostic efficiency statistics for pediatric psychology: We are ready to ROC. Journal of Pediatric Psychology. 2014;39:204–221. [PMC free article] [PubMed]
  • Youngstrom EA, Choukas-Bradley S, Calhoun CD, Jensen-Doss A. Clinical guide to the Evidence-Based Assessment approach to diagnosis and treatment. Cognitive and Behavioral Practice. 2014;22:20–35.
  • Youngstrom EA, de Los Reyes A. Commentary: Moving toward cost-effectiveness in using psychophysiological measures in clinical assessment: validity, decision making, and adding value. Journal of Clinical Child and Adolescent Psychology. 2015;44:352–361. [PubMed]
  • Youngstrom EA, Frazier TW. Evidence-based strategies for the assessment of children and adolescents: Measuring prediction, prescription, and process. In: Miklowitz DJ, Craighead WE, Craighead L, editors. Developmental psychopathology. 2nd ed. Wiley; New York: 2013. pp. 36–79.
  • Youngstrom EA, Freeman AJ, Jenkins MM. The assessment of children and adolescents with bipolar disorder. Child and Adolescent Psychiatric Clinics of North America. 2009;18:353–390. [PMC free article] [PubMed]
  • Yule W, Bolton D, Udwin O, Boyle S, O'Ryan D, Nurrish J. The long-term psychological effects of a disaster experienced in adolescence: I: The incidence and course of PTSD. Journal of Child Psychology and Psychiatry. 2000;41:503–511. [PubMed]
  • Zatzick D, Rivara F, Jurkovich G, Russo J, Trusz S, Wang J, Katon W. Enhancing the population impact of collaborative care interventions: Mixed method development and implementation of stepped care targeting posttraumatic stress disorder and related comorbidities after acute trauma. General Hospital Psychiatry. 2011;33:123–134. [PMC free article] [PubMed]
  • Zhang Z, Shi Z, Wang L, Liu M. One year later: Mental health problems among survivors in hard-hit areas of the Wenchuan earthquake. Public Health. 2011;125:293–300. [PubMed]
  • Zimmerman M, Mattia JI. Is posttraumatic stress disorder underdiagnosed in routine clinical settings? The Journal of Nervous and Mental Disease. 1999;187:420–428. [PubMed]