A brief screening tool for assessing psychological trauma in clinical practice: development and validation of the New York PTSD Risk Score,

doi:10.1016/j.genhosppsych.2011.06.001

Gen Hosp Psychiatry. Author manuscript; available in PMC 2013 January 28.

Published in final edited form as:

Gen Hosp Psychiatry. 2011 Sep-Oct; 33(5): 489–500.

Published online 2011 July 20. doi: 10.1016/j.genhosppsych.2011.06.001

PMCID: PMC3557518

NIHMSID: NIHMS435347

A brief screening tool for assessing psychological trauma in clinical practice: development and validation of the New York PTSD Risk Score^,

Joseph A. Boscarino, Ph.D., M.P.H.,^a^b^c^* H. Lester Kirchner, Ph.D.,^a Stuart N. Hoffman, D.O.,^d Jennifer Sartorius, M.S.,^a Richard E. Adams, Ph.D.,^e and Charles R. Figley, Ph.D.^f

^aCenter for Health Research, Geisinger Clinic, Danville, PA 17822, USA

^bDepartments of Medicine and Pediatrics, Mount Sinai School of Medicine, New York, NY 10029, USA

^cDepartment of Psychiatry, Temple University School of Medicine, Philadelphia, PA 19140, USA

^dDepartment of Neurology, Geisinger Clinic, Danville, PA 17822, USA

^eDepartment of Sociology, Kent State University, Kent, OH 44242, USA

^fGraduate School of Social Work, Tulane University, New Orleans, LA 70118, USA

^*Corresponding author. Center for Health Research, Geisinger Clinic, Danville, PA 17822-4400, USA. Tel.: +1 570 214 9622; fax: +1 570 214 9451. Email: jaboscarino/at/geisinger.edu (J.A. Boscarino)

The publisher's final edited version of this article is available at Gen Hosp Psychiatry

Abstract

Objective

The objective was to develop a brief posttraumatic stress disorder (PTSD) screening instrument that is useful in clinical practice, similar to the Framingham Risk Score used in cardiovascular medicine.

Methods

We used data collected in New York City after the World Trade Center disaster (WTCD) and other trauma data to develop a new PTSD prediction tool — the New York PTSD Risk Score. We used diagnostic test methods to examine different clinical domains, including PTSD symptoms, trauma exposures, sleep disturbances, suicidal thoughts, depression symptoms, demographic factors and other measures to assess different PTSD prediction models.

Results

Using receiver operating curve (ROC) and bootstrap methods, five prediction domains, including core PTSD symptoms, sleep disturbance, access to care status, depression symptoms and trauma history, and five demographic variables, including gender, age, education, race and ethnicity, were identified. For the best prediction model, the area under the ROC curve (AUC) was 0.880 for the Primary Care PTSD Screen alone (specificity=82.2%, sensitivity=93.7%). Adding care status, sleep disturbance, depression and trauma exposure increased the AUC to 0.943 (specificity=85.7%, sensitivity=93.1%), a significant ROC improvement (P < .0001). Adding demographic variables increased the AUC to 0.945, which was not significant (P=.250). To externally validate these models, we applied the WTCD results to 705 pain patients treated at a multispecialty group practice and to 225 trauma patients treated at a Level I Trauma Center. These results validated those from the original WTCD development and validation samples.

Conclusion

The New York PTSD Risk Score is a multifactor prediction tool that includes the Primary Care PTSD Screen, depression symptoms, access to care, sleep disturbance, trauma history and demographic variables and appears to be effective in predicting PTSD among patients seen in healthcare settings. This prediction tool is simple to administer and appears to outperform other screening measures.

Keywords: Posttraumatic stress disorder, Psychological Trauma, Diagnostic testing, Patient screening, Area under receiver operating characteristic (ROC) curve

1. Introduction

The goal of this study is to identify risk assessment instruments for posttraumatic stress disorder (PTSD) that are useful and effective in clinical practice. To develop screening interventions that can become part of routine care, there is a need for decision tools to be brief and useful for both clinicians and patients [1]. Strategies for disease screening are dictated by evidence that the testing is practical and effective. Most screening tests in medicine are readily available, easy to administer and inexpensive (e.g., tests for cholesterol, blood pressure assessments for hypertension, etc.) [1]. New screening tests need to be subjected to evaluation of sensitivity, specificity, impact on disease and cost-effectiveness. Clinicians are continuously introduced to new screening tests, oftentimes before evaluation is complete. For example, the use of genetic testing has been advocated as a screener for many disorders. However, due to the costs and risk of false-positive results, there is little evidence to justify this approach in many clinical areas [2].

To meet our study objective, we used a prospective study of adults exposed to the World Trade Center disaster (WTCD) in New York City (NYC) [3–14]. This study collected extensive clinical data, including mental health, physical health, trauma exposure, and psychosocial and demographic data spanning both the pre- and postdisaster periods [15]. The WTCD data set included a large sample of community-dwelling adults randomly selected from the five boroughs of NYC. In addition, as we note below, PTSD was assessed in this study based on the full Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria [16]. The WTCD data were used together with other trauma data from other studies, including a study of chronic pain patients and a study of trauma patients discharged from a Level I Trauma Center.

To date, a number of brief PTSD screening tools are available. These screeners include the Primary Care PTSD Screen (PCPS), the Short Screening Scale for PTSD (SSSP), the abbreviated PTSD Checklist, the Short PTSD Rating Interview and the Screening Tool for Early Prediction of PTSD (STEPP), among others [17–25]. With the exception of the STEPP scale, which was designed for children and parents, these screening scales were developed for adult populations. They are all relatively short, appear to have acceptable specificity and sensitivity, and are generally focused on only PTSD symptoms. Presently, the PCPS is being used not only among Department of Defense and Department of Veterans Affairs personnel but also among civilian populations [18,20,26]. This usage would make the PCPS the most common PTSD screener in current use in the United States.

The goal of the current research was to develop a new screening instrument that was multidimensional, was effective and could be used in clinical settings. To guide our approach, we used the Framingham Risk Score (FRS) as a clinical model [27,28]. The FRS uses clinical data to estimate a patient’s risk for future coronary heart disease. Although there are different Framingham scores in use [29,30], the risk factors used in the FRS model typically include age, total cholesterol, high-density lipoprotein cholesterol, systolic blood pressure, treatment for hypertension and cigarette smoking. The objective of the FRS is to identify the common factors in different clinical domains that contribute to disease onset. Consistent with the FRS, we chose to examine a range of risk factors that would extend beyond PTSD screeners in current use [17,22]. Our plan was to evaluate a range of core symptoms, psychosocial risk measures and demographic factors to develop robust prediction models that use the minimum number of data elements.

Past research regarding the health consequences of traumatic events guided our current study [31–34]. Although some investigators have contended that persons recover quickly from traumatic experiences [35], reviews of studies have concluded that traumatic event exposures can result in significant health consequences for some persons [36–38]. In terms of PTSD, it has been suggested that this condition was associated with impairment comparable to other serious mental disorders [39]. Similarly, national studies of veterans have reported that many with PTSD had significant impairments [40,41]. Recent research also suggests that PTSD is associated not only with neuroendocrine and immune system alterations [42–45] but also with the onset of chronic medical conditions [46–48]. Furthermore, research on community disasters has suggested that survivors of these events have increased psychological problems, health concerns and chronic problems in living and typically experience significant psychosocial resource losses [12,38,49]. In summary, past research related to traumatic stress exposures and PTSD suggests that psychological, social and physical health consequences are associated with these experiences [50]. Based on this body of research, the focus of the current study was to develop valid and reliable PTSD diagnostic tools that were multifactorial, but that could still be used in routine clinical practice.

2. Methods

2.1. Conceptual approach

Although level of exposure and trauma-related loss are often associated with the impact of traumatic events [36,38], there are other mediating variables. Research suggests that increased PTSD vulnerability often occurs among those with a history of mental health disorders, child adversity and previous traumas [42,51,52]. Socioeconomic and racial/ethnic factors are also reported to influence these experiences [10,53]. In addition, research has identified the role of social support among those exposed to traumatic events [12,49]. Thus, the degree of exposure, social/cultural variables and other factors are often involved in determining the impact of traumatic stressors [3]. In addition, the psychobiological bases of these stress syndromes have also become apparent [54,55]. Consequently, it was expected that a number of behavioral health problems would emerge among traumatized persons, including sleep disturbances, substance misuse, alterations in functional health and mental health status, and the onset of other health problems and concerns [47,56].

Currently, PTSD is thought to be associated with outcomes along several causal pathways that encompass psychological, behavioral and biological domains [57,58]. For example, PTSD can directly result in changes in health status through alterations in physiologic functions associated with stress exposure or, indirectly, through altered health behaviors (e.g., alcohol abuse, increased cigarette smoking, etc.) resulting from altered psychological states (e.g., increased anxiety) brought on by PTSD psychopathology (e.g., intrusive thoughts). The onset of PTSD symptoms can also result in treatment or social support seeking, which may affect future mental health status. Personality and genetic factors are also thought to mediate these symptoms [59]. Consequently, we used a multifactorial conceptual approach to guide model building combined with agnostic (i.e., atheoretical) examination of statistical results. It is also noted that earlier studies assessing the relationship between trauma exposure and psychosocial outcomes have been often confined to smaller, institutional or nonrepresentative community samples [33,34,60]. As described below, the WTCD cohort represents one of the major longitudinal studies that examined mental health outcomes following a major traumatic event among a random population sample of adults [3–14]. These data, combined with the other validation data described below, enabled the testing of specific models that were both conceptually and empirically grounded and included sample sizes adequate for data analysis. Table 1 presents the main measures used in this study related to the WTCD development sample.

Table 1

Main measurements in WTCD study and measurement timeframes

2.2. Statistical approach

We used a process of moving variables in and out of the prediction models to allow for the manipulation of specificity and sensitivity for different combination of variables [61]. This calibration permitted establishment of risk score cut-points that would be the most useful for clinicians. This approach was sensitive to a balance between statistical and clinical significance in developing models that could be useful in clinical decision-making. We used methods developed for diagnostic test assessments, including sensitivity, specificity, receiver operating characteristic (ROC) curves and bootstrap techniques [61]. First, an initial model (i.e., the base model) was developed using traditional variables thought to be related to PTSD. This model was then extended to include the unique collection of measurements of interest available from the WTCD cohort study. These variables included, but were not limited to, mental health status, substance misuse, stress exposures, social/community resources and functional status measures, among others (Table 1). The goal of this model building was to estimate the area under the ROC curve (AUC), while using the fewest number of parameters and the briefest measurement scales. The AUC was estimated at each step to quantify the prediction accuracy of the diagnostic models [62]. That is, it provided a quantitative measure of the discrimination ability of a model to correctly classify patients with and without PTSD. The sequential addition of variables to the base model was evaluated in terms of increasing the AUC irrespective of the change in the likelihood function in logistic regressions [61].

A nonparametric method was used to hierarchically compare the added effects of other variables above the contribution of the base model [63]. This was achieved by comparing the areas under two correlated ROC curves using a nonparametric approach described by DeLong et al. and implemented in Stata 11.2 and SAS 9.2 statistical packages [63–65]. Specifically, this comparison is achieved by generating ROC curves using each possible outcome of the diagnostic test as a classification cut-point and computing the corresponding sensitivity and 1–specificity. These points are then connected by straight lines, and the resulting AUC is compared using a trapezoidal rule [63]. The results of the final models identified were then used to construct risk scores for PTSD onset following trauma exposure by subgroups of key predictor variables. The properties of these risk scores were examined in terms of sensitivity, specificity and AUC and by use of a nomogram, which is a graphical tool used to pictorially represent the model [66]. In addition, a common problem in estimating measures of diagnostic ability is that of overestimation. This is because the same data set is used for both model identification and prediction [67]. This problem was overcome by estimating a bias-corrected version using a bootstrap procedure that pulled 1000 random WTCD samples to provide an estimate of the AUC [68]. This procedure has been shown to be superior to the method of cross-validation using a training and validation data set [68]. Additionally, bootstrap 95% confidence intervals (CIs) for the bias-corrected AUC were also calculated for the ROC curves reported for the WTCD development sample. In addition to estimating the AUC, we also used Youden’s Index [61,69]. Youden’s Index is a summary measure of the ROC curve, as it provides a criterion for choosing a specific cutoff value for which both sensitivity and specificity are maximized [63,70].

It is expected that the NY PTSD model may not perform well for all participants. Therefore, we used these techniques to model sensitivity and specificity as functions of participant demographics including age, gender and ethnicity [71]. A single model was developed to create a risk score from resultant logistic regression analyses for each final prediction model identified. Using Youden’s Index, we obtained an optimal cutoff value of the risk score to classify the participants as PTSD cases or noncases. Leisenring et al. described how logistic regression may be used to model sensitivity and specificity as a function of covariates [71]. We utilized this approach to identify subgroups of interest in which the risk assessment model performs better than other subgroups. For our analyses, we used SAS version 9.2 and Stata version 11.2 [63–65]. In addition, Pepi software (version 4.0) was also used to estimate Youden’s Index and positive predictive and negative predictive values for each final model identified [69].

2.3. WTCD development and validation study

The WTCD was a mass-casualty event and represented the largest trauma-related loss of life and destruction in the United States since the American Civil War [3,72]. To study the impact of this event, baseline diagnostic interviews, using random-digit dialing, were conducted among NYC adults (18 years and older) by telephone in English and Spanish 1 year after the attack. A follow-up interview was conducted 1 year later. For the baseline survey, 2368 residents completed the interview. For this study, PTSD was diagnosed based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria (DSM-IV) [16]. This PTSD measure was developed for telephone administration and used in previous mental health surveys [72–74]. To meet criteria for PTSD in this study, the person had to meet the full diagnostic criteria (A–F) for PTSD [15]. Cronbach’s alpha for the symptoms used in this scale was 0.90 [3], and the concordance of this PTSD scale compared to a clinical diagnostic interview was reported to be good [75]. Versions of this scale have been used in mental health surveys involving over 15,000 telephone interviews, including several WTCD surveys [3,72–74,76]. An important aspect of the WTCD survey is that it used the full PTSD diagnostic criteria, which is relatively rare in trauma studies [15]. Additional information on the WTCD study has been published elsewhere [3–14].

2.4. Chronic pain validation study

For this validation study, adult subjects (18+ years old) were selected from among noncancer chronic pain patients, defined as those prescribed four or more opioid pain prescriptions over a 12-month period. These individuals were selected from primary and specialty care patients seen in the Geisinger Clinic, a large integrated healthcare system that serves 2.5 million residents within 40 central and northeastern Pennsylvania counties. The majority of these pain patients had a history of physical injury, a condition often associated with PTSD [77,78]. Patients were eligible for this study if they received care from one of nine community practice clinics or from the three specialty clinics, including a pain management and an orthopedics clinic, and were prescribed opioid medications for non-malignant pain. Altogether, 12 clinics participated in the study, providing a sample pool of 2459 chronic pain patients. Diagnostic telephone interviews were completed from August 2007 through November 2008. Following a patient notification letter, telephone recruitment was initiated. Interviewers administered structured diagnostic interviews by telephone that included assessment of PTSD, depression, general anxiety and psychological trauma using the diagnostic interview adopted from the WTCD Study discussed above. This survey was administered using a computer-assisted telephone interviewing (CATI) system (WinCati, version 4.2; Sawtooth Technologies, Northbrook, IL, USA). Altogether, 705 patients were interviewed in this study. The survey cooperation rate (i.e., percentage interviewed after patient contact) was 51% (705/1390) [79]. This study was approved by the Geisinger Institutional Review Board (IRB). Additional information related to this study has been published elsewhere [80].

2.5. Trauma center validation study

For this validation study, adult subjects (18+ years old) were selected from trauma patients discharged from Geisinger Clinic’s Level I Trauma Center. As noted above, Geisinger serves 2.5 million residents located within 40 central and northeastern Pennsylvania counties. Patients were eligible for this trauma study if they were discharged alive from the Trauma Center and were not institutionalized at the time of the survey follow-up. Study interviews took place from June 2008 through December 2008. Patients for this study were contacted approximately 6 months after discharge from the Trauma Center. Following a patient notification letter, telephone recruitment was initiated. Following consent, interviewers also administered structured diagnostic interviews adopted from the WTCD Study that included assessment of PTSD, depression, anxiety and psychological trauma using a diagnostic interview designed for this purpose [15]. As in the pain study described, the survey was administered using a CATI system (WinCati, version 4.2; Sawtooth Technologies, Northbrook, IL, USA). Altogether, 225 patients were interviewed in the trauma study. Our survey cooperation rate for this study was 50% (225/451) [79]. This trauma study was approved by the Geisinger IRB.

2.6. Inclusion of existing PTSD screener instruments

As part of our study effort, we reviewed existing PTSD screening instruments currently in clinical use [17,21,22]. In our study, we included two instruments that are in common use, including the SSSP and the PCPS [19,26,81,82]. This inclusion was made possible because, although there were some minor variations in question wording, the PTSD symptom items used in these screeners were also included in the PTSD scales deployed in all three trauma studies.

3. Results

Table 1 presents the core predictor variables identified in the WTCD developmental data set. As seen, these variables span the pre- and the postdisaster periods and bridge different clinical domains, including the social, environmental and psychological factors. The general population profiles for three studies used in our analyses are shown in Table 2. Clearly, the WTCD developmental sample appears to differ from the pain and trauma studies in terms of race/ethnicity and level of education. These study samples, however, appear more comparable with respect to lifetime trauma exposure and the prevalence of PTSD (Table 2). Nevertheless, χ² tests indicated that differences between study samples shown were all statistically significant (P < .05).

Table 2

Study profiles of populations used in New York PTSD risk score study

Table 3 summarizes the results for each study by different groups of predictor variables. As suggested, the SSSP and the PCPS were selected because they were in general use and also included in the PTSD symptoms scales used in the studies evaluated. As can be seen, these screeners alone are good predictors of PTSD. For example, in the WTCD bootstrapped evaluation sample, the SSSP had a specificity of 88.8% and a sensitivity of 92.5% (AUC=0.907); the PCPS had a specificity of 82.2% and a sensitivity of 93.7% (AUC=0.880). Among the pain patients, the results for the SSSP were a specificity of 95.6% and a sensitivity of 94.3% (AUC=0.949); the PCPS also had a specificity of 95.6% and a sensitivity of 94.3% (AUC=0.949). When we used these screeners in the trauma study, the results for the SSSP was a specificity of 85.9% and a sensitivity of 65.4% (AUC=0.757). However, the PCPS performed better among trauma patients, with a specificity of 80.9% and a sensitivity of 96.2% (AUC=0.885).

Table 3

Prediction results: WTCD, pain and trauma studies using different prediction models

In the WTCD development sample, adding the psychosocial predictors to the model with the SSSP resulted in a significant improvement (AUC=945, P < .0001). This is also true for the PCPS (AUC=0.943, P < .0001). This level of improvement was not observed in the pain study, either for the SSSP (AUC=0.965, P=.0725) or for the PCPS (AUC=0.970, P < .0516), after psychosocial variables were added to the model. By comparison, in the trauma study, the results were significant when psychosocial variables were added both for the SSSP (AUC=0.874, P < .0001) and the PCPS (AUC=0.929, P=.0063). However, adding demographic variables did not improve the predictions in the WTCD study (AUC=0.949, P=.0862 for the SSSP model; AUC=0.945, P=.2497 for the PCPS model). The results were similar for the pain study (AUC=0.966, P=.7779 for the SSSP model; AUC=0.972, P=.3673 for the PCPS model) and for the trauma study (AUC=0.883, P=.2337 for the SSSP model; AUC=0.928, P=.8929 for the PCPS model).

Because the prevalence of PTSD in our study samples was relatively low (7.3%–11.6%), the predictive value of a positive test (PV+) was generally less than 50%, while the predictive value of a negative test (PV−) was typically 99% (Table 3). However, we note that given our prediction models, if our study populations had a PTSD prevalence of ~20%, statistical simulations using Pepi software suggested that the positive predictive value of a positive test would be about 80% to 90%, a substantial improvement.

Following the assessments using a combination of predictors, we also examined the predictive value of the psychosocial risk factors alone (Table 4). As can be seen, the use of the psychosocial predictors alone (i.e., sleep disturbance, depression symptoms, trauma history, not having a regular source of care) also worked well in predicting PTSD. In Table 4, the AUC results for the WTCD, pain and trauma studies range from 0.873 to 0.946, suggesting good predictive results using only these variables. However, adding demographic factors to the latter prediction models only improved the prediction results for the WTCD study, with AUC=0.906 and P=.004, but not for the pain or the trauma studies (Table 4).

Table 4

Prediction results: WTCD, pain and trauma studies using psychosocial risk factor and demographic models only

Table 5 presents PTSD scoring results (i.e., the final derived weights) for the PCPS used to generate the classification results shown in Table 3 for these PCPS models. We present the PCPS results because these appeared to produce the best overall predictions and the PCPS has fewer scale items than the SSSP (four items vs. seven items). As seen, a positive score on the PCPS (i.e., three or more positive items) is given a base score of 100, and the psychosocial and demographic items are given weights (or scores) relative to this score, This scoring is based on the logistic regression analyses, whereby the b coefficients in these logistic regression models are converted to standard weights with the aid of a nomogram, as discussed (Fig. 1). The last row of Table 5 shows the cutoff score for a PTSD classification based on these weights: 100 for the PCPS used alone, 158 for the PCPS+psychosocial predictors and 194 for PCPS+psychosocial+demographic factors. Cutoff scores are also shown for psychosocial predictors alone (score=152) and for psychosocial+demographic factors (score=159). Thus, to be classified as a PTSD case using the full NY PTSD Risk Score model simply requires a positive score on the PCPS (score=100), exposure to two lifetime traumatic events (score=26), current sleep disturbance (score=46) and having no regular source of medical care (score=23), which sum to a total score of 195.

Table 5

New York PTSD risk scores for PCPS, psychosocial risk factors and demographic factors

Fig. 1

Nomogram for PCPS with psychosocial predictors included. Fig. 1 shows scale metrics used in score development with the aid of a nomogram, whereby final regression coefficients are converted to additive risk scores relative the PCPS using a linear predictor (more ...)

Finally, we assessed these results for men and women separately and found that while the overall results were generally the same, the PTSD screeners tended to contribute more to the prediction models for the women, while the depression symptoms contributed more to the prediction model for the men. We plan to analyze these findings further by gender to determine if separate prediction models are justified in a future study.

4. Discussion

We examined different clinical domains, including stressor exposures, psychosocial measures, sleep disturbances, suicidal thoughts, PTSD symptoms, depression symptoms and demographic factors, to evaluate different PTSD prediction models. As stated, our goal was to develop a prediction tool that was brief and consistent with the FRS concept. As shown, five prediction domains, including two PTSD symptom screeners (i.e., the Brief and PC-PTSD screeners), sleep disturbance, access to care, depression symptoms and past trauma exposure, and five demographic variables, including gender, age, education, race and Hispanic ethnicity, were identified. For the best overall model, the AUC was 0.880 for the PCPS alone (specificity=82.2%, sensitivity=93.7%). Adding physician status, sleep disturbance, depression symptoms and trauma exposure to this model increased the AUC to 0.943 (specificity=85.7%, sensitivity=93.1%), a significant ROC improvement (P < .0001). Adding demographic variables increased the AUC to 0.945, which was not significant (P=.2497). As discussed, to validate these findings, we also applied the prediction results to 705 pain patients treated at a large, multispecialty group practice and 225 patients discharged from a Level I Trauma Center.

In summary, these results were validated within the WTCD cohort development sample, as well as for the pain and trauma samples, for both the Brief PTSD and the PC-PTSD screeners. The results for both screeners were generally similar, with and without the other predictor variables. The one exception was that, in the trauma study, the Brief PTSD screener tended to have lower sensitivity (65.4%), while the specificity for this was reasonably good among these trauma patients (85.9%). In addition, using the psychosocial risk factors alone tended to also predict PTSD onset relatively well in the WTCD study. This was also true in the pain and trauma data sets (Table 4), which was not expected.

The use of PTSD screeners has increased recently with the growing interest in the impact of traumatic stressors in primary care. As indicated, the Department of Veterans Affairs and the Department of Defense are routinely using the PC-PTSD screener in clinical practice to assess veterans and active duty personnel [18]. As seen in the current study, the PC-PTSD screener appears to work well with non-veterans and nonmilitary personnel, as has been reported in other studies [26]. The addition of psychosocial predictors increases the predictive ability of the PC-PTSD screener, but adding demographic factors did not. However, even when used alone, the predictive utility of the PC-PTSD screener is impressive, with AUCs=0.880–0.949. The PC-PTSD screener consists of four PTSD symptom questions, which would require only a few minutes to administer in most cases. If the psychosocial questions are added, which include two depression questions, a trauma question, a sleep question and a healthcare status question, this would likely require less than 5 min to administer. Thus, the objective of the current study to develop a brief PTSD screener with high specificity and sensitivity appears to have been achieved. Our estimate is that the core instrument (i.e., PCPS+risk factors, without the demographics) would consist of nine questions and would require less than 5–6 min to administer in most cases.

The current study has several strengths and limitations. A major strength was that our study involved a large-scale random survey among a multiethnic urban population and three validation studies. The latter included the WTCD bootstrapped validation, the pain study validation and the trauma study validation. Thus, this study included a total combined sample size of 3298 adults, which included 270 PTSD cases diagnosed using the full DSM-IV criteria. We also assessed a broad range of psychological and interpersonal risk factors using standardized instruments and medical test development methods. Potential study limitations include the omission of individuals without a telephone and those who were institutionalized or homeless. Also, the pain and trauma studies were conducted by telephone and excluded those too ill to be interviewed or who were institutionalized. In addition, nonresponse bias also could have affected all our survey result — our survey response rates could have been higher in our surveys. The lower response rates could have biased our results. Additionally, the item wording for the PTSD symptoms included in the two screeners we used (i.e., the SSSP and PCPS) was slightly different than what is actually used in these screeners. This may have affected our results. Finally, we did not use the predictor identified to predict PTSD beyond the first year after trauma exposure. As part of this investigation, we are planning to study additional predictor variables to forecast PTSD 2 years after exposure. As noted elsewhere [7], longer-term PTSD onset, especially what has been termed “delayed” PTSD, is more difficult to predict.

Despite these limitations, our study suggests that simple, multifactor screening tools, such as the NY PTSD Risk Score, can be highly effective in PTSD screening. Using the Framingham Study as a conceptual model, we developed a multidimensional PTSD risk score based on brief PTSD screeners, depression symptoms, sleep disturbance, trauma history and health care/physician status that was highly effective in discriminating PTSD cases from noncases in three different study populations. This classification system can be used based on the available patient or provider time, including the PCPS alone or in combination with psychosocial risk factor predictors and demographic factors.

As suggested, the goal of this effort was to develop risk assessment tools that are sensitive to statistical and clinical significance in order to develop data useful for clinical decision-making. Our plan was to develop PTSD prediction models to facilitate early intervention by making it possible to identify high-risk groups from among all persons exposed to trauma. Judging by the results of our study, our study goal appears to have been met. Use of the New York PTSD Risk Score, and use of other measures, would make it possible to target resources for those at highest risk of developing PTSD while allowing lower-risk patients to be managed more conservatively [8]. For example, it has been recently reported that brief mental health interventions made available shortly after a traumatic exposure may be more effective than traditional psychotherapy sessions delivered at a later point in time [8]. If this intervention finding can be confirmed, then it may make sense clinically to offer this type of treatment after traumatic exposures among those who screen positive following such events. Further research is recommended to further verify our findings and to make contextual modifications to improve our results.

Appendix

Primary Care PTSD Screen (three positive symptoms out of four, past 12 months)

You had repeated bad dreams or nightmares or had disturbing or unpleasant memories, thoughts, or images that kept coming into your mind whether you wanted to think of them or not.
You deliberately tried hard not to think about something that happened to you or went out of your way to avoid certain places or activities that might remind you of something that happened in the past.
You felt you had to stay on guard much of the time or unexpected noises startled you more than usual.
You felt cut off from other people, found it difficult to feel close to other people, or you could not feel things anymore or you had much less emotion than you used to have.

Depression Symptoms (lifetime)

Have you ever had a period of 2 weeks or longer when you were feeling depressed or down most of the day or nearly everyday?
Have you ever had a period of 2 weeks or longer when you were uninterested in most things or unable to enjoy things you used to do?

Trauma Exposure (lifetime)

How many traumatic events do you think you have ever experienced? These are events outside of everyday experiences and include being in combat or a war zone, being assaulted or sexually attacked, being in a major disaster, fire, or accident, experiencing the sudden and unexpected death of a loved one, and things like these.
Would you say you never experienced events like these, experienced events like these only once, experienced these events 2–3 times, or experienced these event 4 times or more in your lifetime?

Sleep Disturbance (past 12 months)

In the past 12 months, have you had difficulty falling asleep or staying asleep?

Regular Doctor/Source of Healthcare (current)

Do you have a regular doctor or a usual source of care that you can go to for medical care?

Demographics

What is the highest level of education or schooling you completed (record as college graduate vs. not college graduate)?
How old are you (record in years)?
Are you of Spanish or Hispanic origin?
How would you describe your racial background: White, Black/African American, Asian, or something else (record as White vs. or not White)?
Patient/person’s gender (record by observation): female or male?

Study PTSD Criteria

The PTSD measure used was based on the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition [16]. To have PTSD, a person had to meet the full PTSD criteria (A–F). In the current study, this measure was based on the past 12 months. For each of the 17 DSM-IV PTSD symptoms, the respondent was asked if there was ever been a period of 2 weeks or more during which he/she experienced that symptom. If the respondent answered “yes,” he/she was asked if that symptom was related to up to three traumatic events. Following this, the person was asked when was the last time he/she experienced this symptom, how long he/she had experienced it and if this symptom resulted in significant impairment/distress. If these A–F criteria were met, the person was defined as a PTSD case [7,8].

Footnotes

Previous presentation: Preliminary results from this study were presented at the 31st Annual Meeting of the Anxiety Disorders Association of America, New Orleans, LA, March 26, 2011.

Source of funding: Supported in part by grants from the National Institute of Mental Health (Grants R01-MH-66403 and R21-MH-086317), Pennsylvania Department of Health (Contract 4100042573) and the Geisinger Clinic Endowment (Grants SRC-041 and TRA-015); Boscarino P.I.

References

1. Haynes RB, Sackett DL, Guyatt GH, Tugwell P. Clinical epidemiology: how to do clinical practice research. 3. New York: Lippincott Williams & Wilkins; 2006. pp. 273–322.

2. Fauci AS, Braunwald E, Kasper DL, Hauser SL, editors. Harrison’s principles of internal medicine. 17. New York: McGraw–Hill; 2008. pp. 26–8.

3. Boscarino JA, Adams RE, Figley CR. Mental health service use 1-year after the World Trade Center disaster: implications for mental health care. Gen Hosp Psychiatry. 2004;26(5):346–58. [PMC free article] [PubMed]

4. Boscarino JA, Adams RE, Figley CR. Worker productivity and outpatient service use after the September 11th attacks: results from the New York City Terrorism Outcome Study. Am J Ind Med. 2006;49 (8):670–82. [PMC free article] [PubMed]

5. Boscarino JA, Adams RE, Galea S. Alcohol use in New York after the terrorist attacks: a study of the effects of psychological trauma on drinking behavior. Addict Behav. 2006;31(4):606–21. [PMC free article] [PubMed]

6. Boscarino JA, Adams RE, Stuber J, Galea S. Disparities in mental health treatment following the World Trade Center disaster: implications for mental health care and health services research. J Trauma Stress. 2005;18(4):287–97. [PMC free article] [PubMed]

7. Boscarino JA, Adams RE. PTSD onset and course following the World Trade Center disaster: findings and implications for future research. Soc Psychiatry Psychiatr Epidemiol. 2009;44(10):887–98. [PMC free article] [PubMed]

8. Boscarino JA, Adams RE, Figley CR. Mental health service use after the World Trade Center disaster: utilization trends and comparative effectiveness. J Nerv Ment Dis. 2011;199(2):91–9. [PMC free article] [PubMed]

9. Adams RE, Boscarino JA. Predictors of PTSD and delayed PTSD after disaster: the impact of exposure and psychosocial resources. J Nerv Ment Dis. 2006;194(7):485–93. [PMC free article] [PubMed]

10. Adams RE, Boscarino JA. Differences in mental health outcomes among Whites, African Americans, and Hispanics following a community disaster. Psychiatry. 2005;68(3):250–65. [PMC free article] [PubMed]

11. Adams RE, Boscarino JA, Galea S. Social and psychological resources and health outcomes after the World Trade Center disaster. Soc Sci Med. 2006;62(1):176–88. [PMC free article] [PubMed]

12. Adams RE, Boscarino JA. A structural equation model of perievent panic and posttraumatic stress disorder after a community disaster. J Trauma Stress. 2011;24(1):61–9. [PMC free article] [PubMed]

13. Adams RE, Boscarino JA. Stress and well-being in the aftermath of the World Trade Center attack: the continuing effects of a communitywide disaster. J Community Psychol. 2005;33(2):175–90. [PMC free article] [PubMed]

14. Adams RE, Boscarino JA, Galea S. Alcohol use, mental health status and psychological well-being 2 years after the World Trade Center attacks in New York City. Am J Drug Alcohol Abuse. 2006;32 (2):203–24. [PMC free article] [PubMed]

15. Boscarino JA, Adams RE. Overview of findings from the World Trade Center disaster outcome study: recommendations for future research after exposure to psychological trauma. Int J Emerg Ment Health. 2008;10(4):275–90. [PMC free article] [PubMed]

16. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 4. Arlington (VA): American Psychiatric Publishing; 1994.

17. Brewin CR. Systematic review of screening instruments for adults at risk of PTSD. J Trauma Stress. 2005;18(1):53–62. [PubMed]

18. Calhoun PS, McDonald SD, Guerra VS, Eggleston AM, Beckham JC, Straits-Troster K, et al. Clinical utility of the Primary Care-PTSD Screen among U.S. veterans who served since September 11, 2001. Psychiatry Res. 2010;178(2):330–5. [PubMed]

19. Breslau N, Peterson EL, Kessler RC, Schultz LR. Short Screening Scale for DSM-IV posttraumatic stress disorder. Am J Psychiatry. 1999;156(6):908–11. [PubMed]

20. Bliese PD, Wright KM, Adler AB, Cabrera O, Castro CA, Hoge CW. Validating the Primary Care Posttraumatic Stress Disorder Screen and the Posttraumatic Stress Disorder Checklist with soldiers returning from combat. J Consult Clin Psychol. 2008;76(2):272–81. [PubMed]

21. Wilson JP, Keane TM. Assessing psychological trauma and PTSD. 2. New York: The Guilford Press; 2004.

22. Tanielian T, Jaycox LH, editors. Invisible wounds of war. Santa Monica (CA): RAND Corporation; 2008. p. 453.

23. Lang AJ, Stein MB. An abbreviated PTSD Checklist for use as a screening instrument in primary care. Behav Res Ther. 2005;43 (5):585–94. [PubMed]

24. Norris FH, Donahue SA, Felton CJ, Watson PJ, Hamblen JL, Marshall RD. A psychometric analysis of Project Liberty’s adult enhanced services referral tool. Psychiatr Serv. 2006;57(9):1328–34. [PubMed]

25. Winston FK, Kassam-Adams N, Garcia-Espana F, Ittenbach R, Cnaan A. Screening for risk of persistent posttraumatic stress in injured children and their parents. JAMA. 2003;290(5):643–9. [PubMed]

26. van Dam D, Ehring T, Vedel E, Emmelkamp PM. Validation of the Primary Care Posttraumatic Stress Disorder Screening questionnaire (PC-PTSD) in civilian substance use disorder patients. J Subst Abuse Treat. 2010;39(2):105–13. [PubMed]

27. Dawber TR, Meadors GF, Moore FE., Jr Epidemiological approaches to heart disease: the Framingham Study. Am J Public Health. 1951;41 (3):279–81. [PubMed]

28. Dawber TR, Kannel WB. The Framingham Study. An epidemiological approach to coronary heart disease. Circulation. 1966;34(4):553–5. [PubMed]

29. Lloyd-Jones DM, Wang TJ, Leip EP, Larson MG, Levy D, Vasan RS, et al. Lifetime risk for development of atrial fibrillation: the Framingham Heart Study. Circulation. 2004;110(9):1042–6. [PubMed]

30. Lloyd-Jones DM, Wilson PW, Larson MG, Beiser A, Leip EP, D’Agostino RB, et al. Framingham Risk Score and prediction of lifetime risk for coronary heart disease. Am J Cardiol. 2004;94 (1):20–4. [PubMed]

31. Freedy JR, Kilpatrick DG, Resnick HS. Natural disasters and mental health: theory, assessment, and intervention. Journal of Social Behavior and Personality. 1993;8:49–103.

32. Neria Y, Galea S, Norris FH, editors. Mental health and disasters. New York: Cambridge University Press; 2009. pp. 1–3.

33. Fullerton CS, Ursano RJ, editors. Posttraumatic stress disorder: acute and long-term responses to trauma and disaster. Washington (DC): American Psychiatric Press, Inc; 1997. pp. 1–20.

34. Gleser GC, Green BL, Winget CN. Prolonged psychosocial effects of disaster: a study of Buffalo Creek. New York: Academic Press; 1981.

35. McFarlane AC. The aetiology of post-traumatic morbidity: predisposing, precipitating and perpetuating factors. Br J Psychiatry. 1989;154:221–8. [PubMed]

36. Brewin CR, Andrews B, Valentine JD. Meta-analysis of risk factors for posttraumatic stress disorder in trauma-exposed adults. J Consult Clin Psychol. 2000;68(5):748–66. [PubMed]

37. Rubonis AV, Bickman L. Psychological impairment in the wake of disaster: the disaster–psychopathology relationship. Psychol Bull. 1991;109(3):384–99. [PubMed]

38. Norris FH, Friedman MJ, Watson PJ. 60,000 disaster victims speak: part II. summary and implications of the disaster mental health research. Psychiatry. 2002;65(3):240–60. [PubMed]

39. Kessler RC. Posttraumatic stress disorder: the burden to the individual and to society. J Clin Psychiatry. 2000;61(Suppl 5):4, 12. [discussion 13–4] [PubMed]

40. Kulka RA, Schlenger WE, Fairbank JA, Hough RL, Jordan BK, Marmar CR, et al. Trauma and the Vietnam war generation: report of findings from the National Vietnam Readjustment Study. New York: Brunner/Mazel; 1990.

41. Boscarino JA. Post-traumatic stress and associated disorders among Vietnam veterans: the significance of combat exposure and social support. J Trauma Stress. 1995;8(2):317–36. [PubMed]

42. Yehuda R, editor. Risk factors for posttraumatic stress disorder. Washington (DC): American Psychiatric Press, Inc; 1999. pp. 23–59.

43. Vasterling JJ, Brewin CR, editors. Neuropsychology of PTSD: biological, cognitive, and clinical perspectives. New York: Guilford Press; 2005. pp. 27–102.

44. Boscarino JA. Posttraumatic stress disorder, exposure to combat, and lower plasma cortisol among Vietnam veterans: findings and clinical implications. J Consult Clin Psychol. 1996;64(1):191–201. [PubMed]

45. Boscarino JA, Chang J. Higher abnormal leukocyte and lymphocyte counts 20 years after exposure to severe stress: research and clinical implications. Psychosom Med. 1999;61(3):378–86. [PubMed]

46. Boscarino JA. A prospective study of PTSD and early-age heart disease mortality among Vietnam veterans: implications for surveillance and prevention. Psychosom Med. 2008;70(6):668–76. [PMC free article] [PubMed]

47. Schnurr PP, Green BL, editors. Trauma and health: physical health consequences of exposure to extreme stress. Washington (DC): American Psychological Association; 2004. pp. 3–69.

48. Benyamini Y, Solomon Z. Combat stress reactions, posttraumatic stress disorder, cumulative life stress, and physical health among Israeli veterans twenty years after exposure to combat. Soc Sci Med. 2005;61 (6):1267–77. [PubMed]

49. Hobfoll SE, Palmieri PA, Johnson RJ, Canetti-Nisim D, Hall BJ, Galea S. Trajectories of resilience, resistance, and distress during ongoing terrorism: the case of Jews and Arabs in Israel. J Consult Clin Psychol. 2009;77(1):138–48. [PMC free article] [PubMed]

50. Boscarino JA. Psychobiologic predictors of disease mortality after psychological trauma: implications for research and clinical surveillance. J Nerv Ment Dis. 2008;196(2):100–7. [PubMed]

51. Breslau N, Chilcoat HD, Kessler RC, Davis GC. Previous exposure to trauma and PTSD effects of subsequent trauma: results from the Detroit area Survey of Trauma. Am J Psychiatry. 1999;156(6):902–7. [PubMed]

52. Kessler RC, Sonnega A, Bromet E, Hughes M, Nelson CB. Posttraumatic stress disorder in the National Comorbidity Survey. Arch Gen Psychiatry. 1995;52(12):1048–60. [PubMed]

53. Galea S, Tracy M, Norris F, Coffey SF. Financial and social circumstances and the incidence and course of PTSD in Mississippi during the first two years after Hurricane Katrina. J Trauma Stress. 2008;21(4):357–68. [PubMed]

54. van der Kolk BA. The body keeps the score: approaches to the psychobiology of posttraumatic stress disorder. In: van der Kolk BA, McFarlane AC, Weisaeth L, editors. Traumatic stress: the effects of overwhelming experience on mind, body, and society. New York: Guilford Press; 1996. pp. 214–41.

55. Ursano RJ, Goldenberg M, Zhang L, Carlton J, Fullerton CS, Li H, et al. Posttraumatic stress disorder and traumatic stress: from bench to bedside, from war to disaster. Ann N Y Acad Sci. 2010 Oct;1208:72–81. [PubMed]

56. Boscarino JA. Diseases among men 20 years after exposure to severe stress: implications for clinical research and medical care. Psychosom Med. 1997;59(6):605–14. [PubMed]

57. Boscarino JA. Posttraumatic stress disorder and physical illness: results from clinical and epidemiologic studies. Ann N Y Acad Sci. 2004;1032:141–53. [PubMed]

58. Schnurr PP, Jankowski MK. Physical health and post-traumatic stress disorder: review and synthesis. Semin Clin Neuropsychiatry. 1999;4 (4):295–304. [PubMed]

59. Boscarino JA, Erlich PM, Hoffman SN, Rukstalis M, Stewart WF. Association of FKBP5, COMT and CHRNA5 polymorphisms with PTSD among outpatients at risk for PTSD. Psychiatry Res. 2011;188 (1):173–4. [PMC free article] [PubMed]

60. Frey-Wouters E, Laufer RS. Legacy of war: the American soldier in Vietnam. Armonk (NY): Sharpe Pub, Inc; 1986.

61. Pepe MS. The statistical evaluation of medical tests for classification and prediction. New York: Oxford University Press; 2003.

62. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143 (1):29–36. [PubMed]

63. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–45. [PubMed]

64. SAS Institute, Inc. SAS version 9.2. Cary (NC): SAS Institute Inc; 2010.

65. Stata Corporation. Stata, Version 11.2. College Station, TX: 2011.

66. Kattan MW, Wheeler TM, Scardino PT. Postoperative nomogram for disease recurrence after radical prostatectomy for prostate cancer. J Clin Oncol. 1999;17(5):1499–507. [PubMed]

67. Efron B. How biased is the apparent error rate of a prediction rule? Journal of the American Statistical Association. 1986;81(394):461–70.

68. Harrell FE. Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. New York: Springer; 2001.

69. Abramson JH, Gahlinger PM. PEPI, Version 4.0. Salt Lake City (UT): Sagebrush Press; 2001. Computer programs for epidemiologists; p. 305.

70. Fluss R, Faraggi D, Reiser B. Estimation of the Youden index and its associated cutoff point. Biom J. 2005;47(4):458–72. [PubMed]

71. Leisenring W, Pepe MS, Longton G. A marginal regression modelling framework for evaluating medical diagnostic tests. Stat Med. 1997;16 (11):1263–81. [PubMed]

72. Galea S, Ahern J, Resnick H, Kilpatrick D, Bucuvalas M, Gold J, et al. Psychological sequelae of the September 11 terrorist attacks in New York City. N Engl J Med. 2002;346(13):982–7. [PubMed]

73. Resnick HS, Kilpatrick DG, Dansky BS, Saunders BE, Best CL. Prevalence of civilian trauma and posttraumatic stress disorder in a representative national sample of women. J Consult Clin Psychol. 1993;61(6):984–91. [PubMed]

74. Acierno R, Kilpatrick DG, Resnick H, Saunders B, De Arellano M, Best C, et al. family substance use, and depression as risk factors for cigarette use in youth: findings from the National Survey of Adolescents. J Trauma Stress. 2000;13(3):381–96. [PubMed]

75. Kilpatrick DG, Resnick HS, Freedy JR, Pelcovitz D, Resick P, Roth S, et al. The posttraumatic stress disorder field trial: evaluation of the PTSD construct — criteria A through E. In: Widiger T, Frances A, Pincus H, et al., editors. DSM-IV sourcebook. Washington (DC): American Psychiatric Association Press; 1998. pp. 803–44.

76. Boscarino JA, Galea S, Adams RE, Ahern J, Resnick H, Vlahov D. Mental health service and medication use in New York City after the September 11, 2001, terrorist attack. Psychiatr Serv. 2004;55 (3):274–83. [PubMed]

77. Shipherd JC, Keyes M, Jovanovic T, Ready DJ, Baltzell D, Worley V, et al. Veterans seeking treatment for posttraumatic stress disorder: what about comorbid chronic pain? J Rehabil Res Dev. 2007;44(2):153–66. [PubMed]

78. McFarlane AC. The long-term costs of traumatic stress: intertwined physical and psychological consequences. World Psychiatry. 2010;9 (1):3–10. [PubMed]

79. American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 5. Lenexa (KS): American Association for Public Opinion Research; 2008.

80. Boscarino JA, Rukstalis M, Hoffman SN, Han JJ, Erlich PM, Gerhard GS, et al. Risk factors for drug dependence among out-patients on opioid therapy in a large US health-care system. Addiction. 2010;105(10):1776–82. [PubMed]

81. Kimerling R, Ouimette P, Prins A, Nisco P, Lawler C, Cronkite R, et al. Brief report: utility of a short screening scale for DSM-IV PTSD in primary care. J Gen Intern Med. 2006;21(1):65–7. [PMC free article] [PubMed]

82. Ouimette P, Wade M, Prins A, Schohn M. Identifying PTSD in primary care: comparison of the Primary Care-PTSD Screen (PC-PTSD) and the General Health Questionnaire-12 (GHQ) J Anxiety Disord. 2008;22(2):337–43. [PubMed]