Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Mol Psychiatry. Author manuscript; available in PMC 2017 March 24.
Published in final edited form as:
PMCID: PMC5247428

Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)


The 2013 U.S. Veterans Administration/Department of Defense Clinical Practice Guidelines (VA/DoD CPG) require comprehensive suicide risk assessments for VA/DoD patients with mental disorders but provide minimal guidance on how to carry out these assessments. Given that clinician-based assessments are known not to be strong predictors of suicide, we investigated whether a precision medicine model using administrative data after outpatient mental health specialty visits could be developed to predict suicides among outpatients. We focused on male non-deployed Regular U.S. Army soldiers because they account for the vast majority of such suicides. Four machine learning classifiers (naïve Bayes, random forests, support vector regression, elastic net penalized regression) were explored. 41.5% of Army suicides in 2004-2009 occurred among the 12.0% of soldiers seen as outpatient by mental health specialists, with risk especially high within 26 weeks of visits. An elastic net classifier with 10-14 predictors optimized sensitivity (45.6% of suicide deaths occurring after the 15% of visits with highest predicted risk). Good model stability was found for a model using 2004-2007 data to predict 2008-2009 suicides, although stability decreased in a model using 2008-2009 data to predict 2010-2012 suicides. The 5% of visits with highest risk included only 0.1% of soldiers (1047.1 suicides/100,000 person-years in the 5 weeks after the visit). This is a high enough concentration of risk to have implications for targeting preventive interventions. An even better model might be developed in the future by including the enriched information on clinician-evaluated suicide risk mandated by the VA/DoD CPG to be recorded.

Keywords: Army, machine learning, military, predictive modeling, risk assessment, suicide


The historically low U.S. Army suicide rate climbed beginning in 20041 to exceed the civilian rate since 2009.2-3 Preventive interventions exist to reduce Army suicides,4 including a protocol for outpatients treated by mental health specialists based on the 2013 VA/DoD Clinical Practice Guidelines (CPG) on Assessment and Management of Patients at Risk for Suicide for comprehensive suicide risk assessments of all patients in treatment for mental disorders followed by interventions for high-risk patients.5 Although the CPG includes recommendations for risk assessment and stratification, no precision medicine prediction scheme was provided. This is an important gap, as previous research shows clinicians are not good at predicting suicide and that statistical risk models produce better predictions.6,7 The Army maintains electronic administrative systems that might be used to develop a risk model of this sort for soldier suicides. Two recent epidemiological studies demonstrated that such models can be developed. 8-9 The current report presents a similar precision medicine model to predict suicides among soldiers in outpatient treatment with mental health specialists.

Materials and Methods


Analysis was based on the Historical Administrative Data System (HADS) of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS),10 an integrated de-identified dataset of Army/Department of Defense administrative data systems (Appendix Table 1) for each month in service during the years 2004-2009 of all 975,057 Regular U.S. Army soldiers serving at any time during that time-period (32 million person-months), 569 of whom died by suicide. HADS construction and composition are discussed elsewhere.11 We focused initially on soldiers with any outpatient visit having a diagnosis of a mental disorders (ICD-9-CM Codes 290.0-319) or V code indicative of life difficulties often associated with mental disorders (V15.81; V61-62.9; V71.01-71.09), as risk of suicide death was substantially elevated in this segment of the force. Models were built to predict suicide deaths subsequent to these visits using a wide range of HADS predictors. Over 8,000 such visits occurred for each suicide death. As it would have been computationally intensive to include all these control visits in the analysis, we selected a probability sample of control visits equal to roughly 100 times the number of suicide deaths and compared values on predictors available at the times of those visits to the values of the same predictors available at the times of visits that occurred before suicide deaths. Control visits were weighted to adjust for their under-sampling so that the weighted sum of control visits equaled the population distribution (i.e., somewhat more than 8,000 times the number of visits followed by suicide deaths). This kind of subsampling and weighing of controls improves the efficiency of estimation without introducing bias into estimates compared to an analysis that included all control visits.12


Numerous epidemiological studies have examined predictors of suicide among outpatients13-18 and military personnel.1,19-26 HADS variables operationalized as many of these predictors as possible organized into six broad categories: socio-demographics, Army career (e.g., age-at-enlistment, occupation, deployment history), characteristics of the index visit, prior clinical factors (e.g., inpatient and outpatient mental and physical disorders, prescriptions, suicide attempts), crime codes (victimization and perpetration), and contextual factors (e.g., unit-level characteristics, registered weapons). We controlled year, season, and time until next visit to adjust for secular trends in the Army suicide rate and time-at-risk.

Given that the administrative data were collected for other purposes, we cast a wide net in extracting indicators of target constructs. For example, we examined 23 different categories of psychiatric diagnoses and 15 categories of NDC psychotropic medication codes based on the First Databank (FDB) Enhanced Therapeutic Classification System™,27 (Appendix Tables 2-3). Nearly 1,000 variables were constructed (Appendix Tables 4). Missing socio-demographic and Army career data were corrected when possible with nearest neighbor temporal imputations. Remaining missing values and inconsistencies were resolved using rational imputation. (e.g., a soldier classified female one month but male all other months was recoded male). Details about missing data patterns are available in Appendix Table 5.

Analysis methods

De-identified HADS analysis was approved by the Human Subjects Committees of the Uniformed Services University of the Health Sciences for the Henry M. Jackson Foundation (the primary grantee), the University of Michigan, and Harvard Medical School. Analysis began with cross-tabulations examining suicide risk in the 12 months after each outpatient visit, distinguishing visits in the general medical and mental health specialty sectors by prior psychiatric hospitalization, gender and deployment status. Model-building began by estimating univariate associations of predictor with suicide using discrete-time survival analysis of suicide death (coded 1) compared to all other outcomes (i.e., some other death, a subsequent mental health specialty visit, separation from service, end of the follow-up period, all coded 0). A logistic link function was used to estimate coefficients with proc logistic in SAS 9.3.28 Functional forms of significant non-dichotomous predictors were transformed to capture interpretable nonlinearities.

As multivariable associations were unstable, machine learning methods were used to generate stable estimates comparing four different classifiers: naïve Bayes29 using the R-package e1071 naiveBayes;30 random forest31 using the R-package RandomForest (RF);32 support vector regression33 using the R-package e1071 svm; and elastic net penalized regression34 using the R-package glmnet.35 Hyperparameters were selected to maximize cross-validated sensitivity (i.e., the proportion of observed suicide deaths among predicted positives) in the 5% of visits with highest predicted suicide risk. Selection of the optimal classifier was based on the same criterion.

Once the best classifier was selected, operating characteristics were examined by comparing predicted probability of suicide death for each sampled person-visit to observed suicide death in the entire sample by calculating area under the receiver operating characteristic curve (AUC) and graphing proportional suicide deaths after visits in each ventile (i.e., 5%) of visits grouped from highest to lowest predicted probabilities. We then calculated sensitivity (, as noted above, the proportion of observed suicides after visits predicted to have high suicide risk) and positive predictive value (suicide rate after visits predicted to have high risk expressed as number of suicides /100,000 person-years) in high-risk ventiles along with specificity (the proportion of visits not followed by observed suicides after visits predicted not to have high suicide risk), negative predictive value (the non-suicide rate/100,000 person-years after such visits), and AUC (which, in the case of dichotomous predictors, is the mean of sensitivity and specificity). Given the rarity of suicide deaths, we report 1-negative predictive value (i.e., suicides/100,000 person-years) rather than negative predictive value. Visit-level estimates were then projected to the person-level by aggregating results for selected contiguous 12-month time periods for a probability sample of 100,000 soldiers. Model predictive validity was evaluated by using coefficients estimated in earlier years to predict suicides in later years.


Outpatient visits and suicide by treatment sector, gender, deployment status and time

Sixty-eight (12%) of the 569 suicide deaths of Regular Army soldiers during 2004-2009 occurred among the 0.9% of soldiers with psychiatric hospitalizations in the prior 12 months (252.3/100,000 person-years). (Table 1) Another 240 (42.2%) suicides occurred among the 24.5% of soldiers without 12-month psychiatric hospitalization who were outpatients with target diagnoses or V codes (31.7/100,000 person-years). The remaining 261 (45.9%) suicides occurred among the other 74.6% of soldiers (11.3/100,000 person-years). Among the 0.9% with hospitalization, the suicide rate was highest among those seen outpatient after hospital discharge by both mental health and general medical treatment providers (0.65% of all soldiers; 312.2/100,000 person years), lowest among those seen only by mental health providers (0.11%; 85.7/100,000 person-years), and intermediate among those seen only by general medical providers (0.06%; 107.0/100,000 person-years). Among the 24.5% having outpatient visits without hospitalizations, the suicide rate was highest among those seen both by mental health and general medical providers (5.1% of all soldiers; 63.9/100,000 person years), intermediate among those seen only by mental health providers (6.0%; 36.1/100,000 person-years), and lowest among those seen only by general medical providers (13.4%; 17.4/100,000 person-years).

Table 1
The proportions of all soldiers and soldiers who died by suicide who had psychiatric hospitalization and outpatient treatment for mental disorders in the prior 12 months among Regular Army soldiers over the years 2004-2009 (n = 975,057)

Given the much higher suicide rate among outpatients seen by mental health providers than exclusively by general medical providers, we focused analysis on the former and distinguished between the 66 suicides with prior 12-month psychiatric hospitalization and the 168 suicides without such hospitalization. The population-at-risk consisted of 316,686 Regular Army soldiers making 2,950,967 outpatient mental health specialist visits in 2004-2009. 95.8% of these visits were made when patients were not deployed (173 suicides; 65.6/100,000 person-years) and the suicide rate after these visits was substantially higher among men than women (75.3/ versus 19.6/100,000 person-years), with 94.8% (164 of 173) of suicide deaths after these visits occurring among men. Based on these patterns, we focused analysis on non-deployed men. The majority (61.6%; 101/164) of suicide deaths in this group occurred within 5 weeks of mental health specialist outpatient visits (145.2, 96.3, 123.6, 116.5, and 115.1 suicides/100,000 person years, respectively, in those weeks), with a 57.4/100,000 person-years rate during the remainder of the first 6 months (28.7% [47/164] of suicide deaths over the 12 months after the index visit) and 31.3/100,000 person-year over the subsequent 6 months. Based on these results, we limited model-building to the 26 weeks after the index visit (148 suicides).

Selecting the optimal classifier

Roughly one-third of HADS variables for prior clinical characteristics (244/782 among soldiers with and 178/536 among soldiers without psychiatric hospitalizations) were significant univariate predictors of subsequent suicide. Much smaller proportions of variables characterizing the index outpatient visit (2/46), involvement in crime (2/67), and contextual factors (0/39) were significant. The significant univariate predictors plus 20 socio-demographic and 27 Army career variables were included in multivariable model-building. Based on many predictors about psychiatric hospitalization being significant, all analyses were carried out separately among soldiers who had (50 suicides) versus had not (97 suicides) psychiatric hospitalizations in the prior 12 months. The elastic net classifier out-performed the others in terms of higher cross-validated sensitivity in the weighted 5% of observations with highest predicted risk among both soldiers with and without prior 12-month psychiatric hospitalizations. Subsequent phases of analysis consequently focused on the elastic net models. Fourteen predictors were included in this model for soldiers with and 10 for soldiers without prior psychiatric hospitalizations.

Operating characteristics of model-based predictions

The model AUCs for the continuous distributions of predicted probabilities over 26 weeks were .72 among soldiers with prior psychiatric hospitalizations, .61 among soldiers without prior hospitalizations, and .66 among both combined. When the same models were applied to suicide deaths in the 5 weeks after the index visits, AUCs increased to .75 (prior hospitalization), .65 (no prior hospitalization), and .69 (both combined). Sensitivity was more than twice the expected value of 5% after visits in the three highest risk ventiles for both 26 weeks and 5 weeks (Figure 1) and either below or only slightly above their expected values in the remaining 17 ventiles, leading us to evaluate operating characteristics of two dichotomous classifications: between the top 1 and other 19 risk ventiles; and between and top 3 and other 17 risk ventiles.

Figure 1
Proportion of suicide deaths that occurred within 5 and 26 weeks of most recent specialty mental health outpatient visits within ventiles1 of visits ranked by predicted suicide risk based on the optimal elastic net penalized logistic regression model, ...

All calculations of operating characteristics combined soldiers with and without prior hospitalizations. (Table 2) Sensitivity in the top ventile was 22.4-24.0% (26-5 weeks after visits). Comparable sensitivities were 45.6-48.0% in the top 3 ventiles. Specificity was 94.9-94.9% in the lowest 19 ventiles and 84.0-84.0% in the 17 lowest ventiles. Positive predictive value was 1076.8-1047.6/100,000 person-years in the top ventile and 602.3-605.9/100,000 person-years in the top 3 ventiles compared to 52.9-71.5/100,000 person-years in the remaining 17 ventiles (i.e., 1–negative predictive value). AUC was .59-.66.

Table 2
Operating characteristics of dichotomous classifications distinguishing soldiers in the top 1 and 3 ventiles of predicted risk of suicide death after mental health specialty outpatient visits made by male non-deployed soldiers based on the optimal logistic ...

Person-level projections of visit-level results

As person-level inferences cannot be drawn from visit-level results, we drew a representative sample of 100,000 soldiers in service over the study period who did not die by suicide, combined them with all soldiers who died by suicide, and generated predicted suicide risk scores based on the coefficients in our best model for each mental health specialty outpatient visit of each soldier in this dataset. These visit-level scores were then aggregated to the person-level. The non-deployed men with 12-month mental health specialty outpatient visits had an average of 6.1 such visits. Extrapolating to an Army of 500,574 (the average number of non-deployed male soldiers on active duty in the Army over the study period), this would be 60,654 non-deployed men making 368,233 mental health specialty outpatient visits over a typical 12 months (17,629-55,286 visits in the 1-3 highest-risk ventiles). 4.2% of soldiers who made 12-month mental health specialty outpatient visits had visits in the top risk ventile, with a mean of 7.0 such visits and a mean of 10.3 weeks in the highest-risk time interval after such visits. This means that only 573 (i.e., .042 × .121 × 500,574 × 10.3/52) male non-deployed soldiers in an Army of 500,574 non-deployed men would be in the highest-risk group in a typical week. This number increased to 1,103 for patients in the highest-risk ventile over 26 weeks and to 3,657 for patients in the 3 highest-risk ventiles over 26 weeks.


Models were re-estimated in the 2004-2007 HADS data using the same predictors but allowing the coefficients to differ from the 2004-2009 model. Results were used to predict 2008-2009 suicides. AUC combining soldiers with and without prior psychiatric hospitalizations was .67-.72 predicting suicides within 26-5 weeks of most recent visit. Twenty-six week sensitivity was 26.7-41.3% for visits in the highest-1-3 risk ventiles. Five-week sensitivity was 29.8-47.4% for visits in the highest-1-3 risk ventiles. Replication of this validation exercise using coefficients estimated in 2008-2009 to predict suicides in 2010-2012 yielded much weaker results: sensitivities of 13.3-18.1% for 26-5 weeks in the 1 highest ventile and 36.1-27.4% in the 3 highest ventiles.

Model coefficients

The 14 predictors in the model for patients with prior hospitalization included 6 indicators of prior suicidality, 6 of prior inpatient-outpatient depression treatment, and 2 of non-affective psychosis and bipolar disorder treatment, all associated with elevated suicide risk. (Table 3, Model 1) Odds-ratios were all relatively modest (OR=1.01-1.32) due to elastic net penalties. Extreme coefficient instability (indicated by high variance inflation factors) occurred, in comparison, when a logistic regression model (Model 2) was estimated with the same predictors, although re-specification allowed this problem to be addressed in a less complex logistic model (Model 3) that retained essentially the same level of overall prediction accuracy (AUC=.72).

Table 3
Coefficients in the optimal elastic net and comparable conventional logistic regression models predicting suicide deaths within 26 weeks of mental health specialty outpatient visits made by non-deployed male soldiers with prior 12-month psychiatric hospitalizations ...

The 10 predictors in the model for patients with no prior hospitalization included one feature of the index visit -- whether with a psychiatrist (associated with elevated suicide risk) – along with 3 measures of treatment in the past month (frequency of visits for depression and ill-defined conditions; any inpatient treatment for a physical disorder), 3 measures of treatment in the past 3 months (any for either non-affective psychosis or personality disorder; number of anticonvulsant prescriptions), 2 measures of treatment in the past 12 months (frequency of outpatient visits for anxiety disorders; any prescription of an alcohol-narcotic abuse treatment agent), and a final measure for whether the soldier was an alleged perpetrator of multiple crimes in the 3 months before the index visit. (Table 4) The odds-ratios of these predictors were much more diverse than in the model for soldiers with prior hospitalizations (OR=1.2-8.8), reflecting the weaker associations among predictors (as indicated by the low variance inflation factors in the parallel logistic model).

Table 4
Coefficients in the optimal elastic net and conventional logistic regression models predicting suicide deaths within 26 weeks of mental health specialty outpatient visits made by non-deployed male soldiers without prior 12-month psychiatric hospitalizations ...


Despite the elevated suicide risk of soldiers with mental health specialty outpatient visits, which is consistent with civilian research,36 and the strong performance of our models, suicide was a rare outcome even among high-risk soldiers. This raises the question whether existing interventions are sufficiently powerful to make targeted preventive interventions cost-effective. There is controversy about this question.37-39 Empirical adjudication would require analyses beyond the scope of this report on competing needs, costs, and cost-effectiveness of intervention options.40, 41 Our aim was to address a prior question : whether a useful precision medicine model can be developed. We showed that it can. The 5% of visits with highest predicted risk include only 0.1% of soldiers with very high suicide risk (1047.1/100,000 person-years in the 5 weeks after the visit). This is a small enough proportion of individuals accounting for a large enough proportion of suicides to have intervention implications.

Interpretation of model predictors should only be undertaken with caution because machine learning methods maximize model performance at the expense of individual coefficient accuracy. Nonetheless, four observations are noteworthy. First, the vast majority of predictors measured mental disorders found to be important in prior studies of soldier suicides.19,21,23,42 The crime perpetration variable in the model for soldiers without prior hospitalization is consistent with evidence that a high proportion of soldiers who die by suicide had legal problems at the time of death.21

Second, we found that hospitalization for any physical health problem was an important predictor of soldier suicide. Although traumatic brain injury (TBI), a widely recognized suicide risk factor,43-44 was included as a potential predictor, the fact that this composite variable was selected over TBI in the predictor set underscores the need for future investigation to focus clinical attention on broader hospitalized physical conditions linked to suicide.

Third, despite previous research consistently finding suicide predicted by socio-demographic characteristics indicating disadvantaged social status (e.g., young age, non-married status) and Army career characteristics indicating low status (e.g., low rank, demotion) predicting soldier suicide,19,20,22,25,42,45 no such predictors emerged in our optimal models. No attempt was made to determine whether this was because the clinical variables in our models mediated the effects of socio-demographic and Army career variables, but future investigation of this possibility might provide insights into modifiable targets of preventive interventions.

Fourth, important differences were found between patients with versus without prior psychiatric hospitalization. AUC and concentration of risk were higher in models for those with (AUC=.72-.75 for 26-5 weeks, with 28-36% of suicides occurring among the 5% of patients with highest predicted risk) than without (AUC=.61-.65, with 22-24% of suicides occurring among the 5% of patients with highest predicted risk) prior hospitalizations. All but one predictor in the model for patients with hospitalization involved characteristics of outpatient visits prior to the hospitalization rather than of the hospitalization, with a focus on suicidality, depression, bipolar disorder, and non-affective psychosis. The model for patients without hospitalization, in comparison, included a much wider array of diagnoses, the one with the highest odds-ratio being alcohol/drug treatment. Recent inpatient treatment for a physical disorder was also a very powerful predictor in that model. These differences suggest that the causal processes underlying suicide are different for patients with and without psychiatric hospitalization. Further investigation of these differences might provide insights to help customize preventive interventions.

Our analysis was limited by considering a large number of predictors of a small number of suicides, introducing risk of over-fitting. We addressed this problem by using cross-validation to select the number of predictors in final models and using penalized regression to select predictors, but residual over-fitting might have occurred. We evaluated this by predicting 2008-2009 suicides based on 2004-2007 models and 2010-2012 suicides based on 2008-2009 models. Model stability was very good between 2004-2007 and 2008-2009 but much lower between 2008-2009 and 2010-2012, possibly reflecting changes in Army policies-practices for managing suicide risk as awareness of the rising Army suicide rate increased. The only way to guard against such a possibility going forward would be to update prediction models regularly (e.g., annually) and carry out sensitivity analyses of the extent to which predictors change depending on the number of years of prior data used in developing the models.

Another set of limitations involves the administrative data used in our models, which had more missing, inconsistent, and possibly erroneous values than in data collected for research purposes and lacked indicators of some suicide risk factors documented in the literature. These limitations presumably resulted to reduced model performance. Yet the models nonetheless had good prediction accuracy that would presumably be improved by increasing data quality (e.g., adding predictors based on the checklist the VA/DoD CPG now urges clinicians to use to evaluate suicide risk). A final noteworthy limitation is that we were unable to follow soldiers out of service to predict suicides that occurred after separation. This right censoring is an important limitation for long-term prediction given that soldiers with mental disorders are more likely than others to terminate service.45

It is unclear from the results reported here how much clinical judgment could be enhanced by having access to results of our models, as clinical assessments of suicide risk were not systematically recorded in Army medical records over the years we studied. As noted in the introduction, though, previous studies find that statistical models are much more accurate than clinical judgment of suicide risk,2,46-48 consistent with a larger literature showing statistical methods outperform expert judgment in many areas of prediction,6,7 suggesting that access to predictions based on our models could be of value to clinicians as one element in their evaluation of patient suicide risk.

Supplementary Material


Funding/Support: Army STARRS was sponsored by the Department of the Army and funded under cooperative agreement number U01MH087981 with the U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (NIH/NIMH). The contents are solely the responsibility of the authors and do not necessarily represent the views of the Department of Health and Human Services, NIMH, the Department of the Army, or the Department of Defense. Gilman's participation in this work was supported by the Intramural Research Program of the Eunice Kennedy Shriver National Institute of Child Health and Human Development. The authors would like to thank Kenneth L. Cox, MD for helpful comments on an earlier version of this paper.

Role of the Sponsors: Although a draft of this manuscript was submitted to the Army and NIMH for review and comment prior to submission, this was with the understanding that comments would be no more than advisory.

The authors would also like to thank John Mann, Maria Oquendo, Barbara Stanley, Kelly Posner, and John Keilp for their contributions to the early stages of Army STARRS development.


Additional Contributions: The Army STARRS Team consists of Co-Principal Investigators: Robert J. Ursano, MD (Uniformed Services University of the Health Sciences) and Murray B. Stein, MD, MPH (University of California San Diego and VA San Diego Healthcare System); Site Principal Investigators: Steven Heeringa, PhD (University of Michigan) and Ronald C. Kessler, PhD (Harvard Medical School); NIMH collaborating scientists: Lisa J. Colpe, PhD, MPH and Michael Schoenbaum, PhD; Army liaisons/consultants: COL Steven Cersovsky, MD, MPH (USAPHC) and Kenneth Cox, MD, MPH (USAPHC). Other team members: Pablo A. Aliaga, MA (Uniformed Services University of the Health Sciences); COL David M. Benedek, MD (Uniformed Services University of the Health Sciences); Susan Borja, PhD (National Institute of Mental Health); Gregory G. Brown, PhD (University of California San Diego); Laura Campbell-Sills, PhD (University of California San Diego); Catherine L. Dempsey, PhD, MPH (Uniformed Services University of the Health Sciences); Richard Frank, PhD (Harvard Medical School); Carol S. Fullerton, PhD (Uniformed Services University of the Health Sciences); Nancy Gebler, MA (University of Michigan); Robert K. Gifford, PhD (Uniformed Services University of the Health Sciences); Stephen E. Gilman, ScD (Eunice Kennedy Shriver National Institute of Child Health and Human Development, Harvard School of Public Health); Marjan G. Holloway, PhD (Uniformed Services University of the Health Sciences); Paul E. Hurwitz, MPH (Uniformed Services University of the Health Sciences); Sonia Jain, PhD (University of California San Diego); Tzu-Cheg Kao, PhD (Uniformed Services University of the Health Sciences); Karestan C. Koenen, PhD (Columbia University); Lisa Lewandowski-Romps, PhD (University of Michigan); Holly Herberman Mash, PhD (Uniformed Services University of the Health Sciences); James E. McCarroll, PhD, MPH (Uniformed Services University of the Health Sciences); Katie A. McLaughlin, PhD (Harvard Medical School); James A. Naifeh, PhD (Uniformed Services University of the Health Sciences); Matthew K. Nock, PhD (Harvard University); Rema Raman, PhD (University of California San Diego); Sherri Rose, Ph.D. (Harvard Medical School); Anthony Joseph Rosellini, Ph.D. (Harvard Medical School); Nancy A. Sampson, BA (Harvard Medical School); LCDR Patcho Santiago, MD, MPH (Uniformed Services University of the Health Sciences); Michaelle Scanlon, MBA (National Institute of Mental Health); Jordan Smoller, MD, ScD (Harvard Medical School); Michael L. Thomas, PhD (University of California San Diego); Patti L. Vegella, MS, MA (Uniformed Services University of the Health Sciences); Christina Wassel, Ph.D. (University of Pittsburgh); and Alan M. Zaslavsky, PhD (Harvard Medical School).

Financial Disclosure: In the past three years, Dr. Kessler has been a consultant for Hoffman-La Roche, Inc. and Johnson & Johnson Wellness and Prevention. Dr. Kessler has served on advisory boards for Mensante Corporation, Johnson & Johnson Services Inc. Lake Nona Life Project, and U.S. Preventive Medicine. Dr. Kessler is a co-owner of DataStat, Inc. Dr. Stein has in the last 3 years been a consultant for Healthcare Management Technologies and had research support for pharmacologic imaging studies from Janssen. The remaining authors report no competing interests. Drs. Petukhova, Bliese, Bossarte, Bromet, Fullerton, Gilman, Ivany, Lewandowski-Romps, Millikan Bell, Naifeh, Nock, Reis, Rosellini, Mrs. Sampson, Drs. Zaslavsky, and Ursano reported no biomedical financial interests or potential conflicts of interest.

Contributor Information

Ronald C. Kessler, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts.

Murray B. Stein, Departments of Psychiatry and Family and Preventive Medicine, University of California San Diego, La Jolla, California and VA San Diego Healthcare System, San Diego, California.

Maria V. Petukhova, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts.

Paul Bliese, Darla Moore School of Business, University of South Carolina, Columbia, South Carolina.

Robert M. Bossarte, Post Deployment Health Service, Department of Veterans Affairs, Washington DC.

Evelyn J. Bromet, Department of Psychiatry & Behavioral Science, Stony Brook School of Medicine, Stony Brook, New York.

Carol S. Fullerton, Center for the Study of Traumatic Stress, Department of Psychiatry, Uniformed Services University School of Medicine, Bethesda, Maryland.

Stephen E. Gilman, Health Behavior Branch, Division of Intramural Population Health Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, Maryland; Department of Epidemiology, Harvard T.H. Chan School of Public Health, and Department of Social and Behavioral Sciences, Harvard T.H. Chan School of Public Health, Boston, Massachusetts.

Christopher Ivany, U.S Army Medical Command, Behavioral Health Service Line, Bethesda, Maryland.

Lisa Lewandowski-Romps, Institute for Social Research, University of Michigan, Ann Arbor, Michigan.

Amy Millikan Bell, U.S. Army Public Health Command, Aberdeen Proving Ground, Maryland.

James A. Naifeh, Center for the Study of Traumatic Stress, Department of Psychiatry, Uniformed Services University School of Medicine, Bethesda, Maryland.

Matthew K. Nock, Department of Psychology, Harvard University, Cambridge, Massachusetts.

Benjamin Y. Reis, Children's Hospital Boston and Harvard Medical School, Boston, Massachusetts.

Anthony J. Rosellini, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts.

Nancy A. Sampson, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts.

Alan M. Zaslavsky, Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts.

Robert J. Ursano, Center for the Study of Traumatic Stress, Department of Psychiatry, Uniformed Services University School of Medicine, Bethesda, Maryland.


1. Armed Forces Health Surveillance Center. Deaths by suicide while on active duty, active and reserve components, U.S. Armed Forces, 1998-2011. Med Surveill Monthly Rep. 2012;19(6):7–10. [PubMed]
2. Nock MK, Deming CA, Fullerton CS, Gilman SE, Goldenberg M, Kessler RC, et al. Suicide among soldiers: a review of psychosocial risk and protective factors. Psychiatry. 2013;76(2):97–125. [PMC free article] [PubMed]
3. Smolenski DJ, Reger MA, Bush NE, Skopp NA, Zhang Y, Campise RL. Department of Defense Suicide Event Report: National Center for Telehealth & Technology. 2013
4. Zamorski MA. Suicide prevention in military organizations. Int Rev Psychiatry. 2011;23(2):173–180. [PubMed]
5. Veterans Affairs/Dept of Defense. Assessment and management of patients at risk for suicide (2013) 2013
6. Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science. 1989;243(4899):1668–1674. [PubMed]
7. Grove WM, Zald DH, Lebow BS, Snitz BE, Nelson C. Clinical versus mechanical prediction: a meta-analysis. Psychol Assess. 2000;12(1):19–30. [PubMed]
8. McCarthy JF, Bossarte RM, Katz IR, Thompson C, Kemp J, Hannemann CM, et al. Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health. 2015;105(9):1935–1942. [PubMed]
9. Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ, et al. Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS) JAMA Psychiatry. 2015;72(1):49–57. [PMC free article] [PubMed]
10. Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB, et al. The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) Psychiatry. 2014;77(2):107–119. [PMC free article] [PubMed]
11. Kessler RC, Colpe LJ, Fullerton CS, Gebler N, Naifeh JA, Nock MK, et al. Design of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) Int J Methods Psychiatr Res. 2013;22(4):267–275. [PMC free article] [PubMed]
12. Schlesselman JJ. Case-Control Studies: Design, Conduct, Analysis. 1st. Oxford University Press; New York: 1982.
13. Brown GK, Beck AT, Steer RA, Grisham JR. Risk factors for suicide in psychiatric outpatients: a 20-year prospective study. J Consult Clin Psychol. 2000;68(3):371–377. [PubMed]
14. Brugnoli R, Novick D, Haro JM, Rossi A, Bortolomasi M, Frediani S, et al. Risk factors for suicide behaviors in the observational schizophrenia outpatient health outcomes (SOHO) study. BMC Psychiatry. 2012;12:83. [PMC free article] [PubMed]
15. Leadholm AK, Rothschild AJ, Nielsen J, Bech P, Ostergaard SD. Risk factors for suicide among 34,671 patients with psychotic and non-psychotic severe depression. J Affect Disord. 2014;156:119–125. [PubMed]
16. Simon GE, Hunkeler E, Fireman B, Lee JY, Savarino J. Risk of suicide attempt and suicide death in patients treated for bipolar disorder. Int J Bipolar Disord. 2007;9(5):526–530. [PubMed]
17. Simon GE, Rutter CM, Peterson D, Oliver M, Whiteside U, Operskalski B, et al. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64(12):1195–1202. [PMC free article] [PubMed]
18. Simon GE, Savarino J, Operskalski B, Wang PS. Suicide risk during antidepressant treatment. Am J Psychiatry. 2006;163(1):41–47. [PubMed]
19. Bachynski KE, Canham-Chervak M, Black SA, Dada EO, Millikan AM, Jones BH. Mental health risk factors for suicides in the US Army, 2007--8. Inj Prev. 2012;18(6):405–412. [PubMed]
20. Bell NS, Harford TC, Amoroso PJ, Hollander IE, Kay AB. Prior health care utilization patterns and suicide among U.S. Army soldiers. Suicide Life Threat Behav. 2010;40(4):407–415. [PubMed]
21. Black SA, Gallaway MS, Bell MR, Ritchie EC. Prevalence and risk factors associated with suicides of Army soldiers 2001–2009. Mil Psychol. 2011;23(4):433–451.
22. Gilman SE, Bromet EJ, Cox KL, Colpe LJ, Fullerton CS, Gruber MJ, et al. Sociodemographic and career history predictors of suicide mortality in the United States Army 2004-2009. Psychol Med. 2014;44(12):2579–2592. [PMC free article] [PubMed]
23. Hyman J, Ireland R, Frost L, Cottrell L. Suicide incidence and risk factors in an active duty US military population. Am J Public Health. 2012;102 Suppl 1:S138–146. [PubMed]
24. Ireland RR, Kress AM, Frost LZ. Association between mental health conditions diagnosed during initial eligibility for military health care benefits and subsequent deployment, attrition, and death by suicide among active duty service members. Mil Med. 2012;177(10):1149–1156. [PubMed]
25. Schoenbaum M, Kessler RC, Gilman SE, Colpe LJ, Heeringa SG, Stein MB, et al. Predictors of suicide and accident death in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS): results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) JAMA Psychiatry. 2014;71(5):493–503. [PMC free article] [PubMed]
26. Street AE, Gilman SE, Rosellini AJ, Stein MB, Bromet EJ, Cox KL, et al. Understanding the elevated suicide risk of female soldiers during deployments. Psychol Med. 2015;45(4):717–726. [PMC free article] [PubMed]
27. FDB First Databank. 2015 Accessed Date Accessed 2015 Accessed.
28. SAS Institute Inc. SAS/STAT® Software 9.3 for Unix edn. SAS Institute Inc.; Cary, NC:
29. Rish I. An empirical study of the naive Bayes classifier. Yorktown Heights, NY: IBM Research Division; 2001.
30. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F, Chang CC, et al. Package ‘e1071’: Misc functions of the Department of Statistics, Probability Theory Group, TU Wien. 2015 1.5-7 edn.
31. Breiman L. Random Forests. Machine Learning. 2001;45:5–32.
32. Liaw A, Wiener M. Classification and regression by randomForest. R News. 2002;2:18–22.
33. Smola AJ, Scholkopf B. A tutorial on support vector regression. Stat Comput. 2004;14:199–222.
34. Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc B. 2005;67(2):301–320.
35. Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw. 2009;33(1):1–22. [PMC free article] [PubMed]
36. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, et al. Health care contacts in the year before suicide death. J Gen Intern Med. 2014;29(6):870–877. [PMC free article] [PubMed]
37. Large MM, Ryan CJ. Suicide risk categorisation of psychiatric inpatients: what it might mean and why it is of no use. Australas Psychiatry. 2014;22(4):390–392. [PubMed]
38. O'Connor E, Gaynes BN, Burda BU, Soh C, Whitlock EP. Screening for and treatment of suicide risk relevant to primary care: a systematic review for the U.S. Preventive Services Task Force. Ann Intern Med. 2013;158(10):741–754. [PubMed]
39. While D, Bickley H, Roscoe A, Windfuhr K, Rahman S, Shaw J, et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: a cross-sectional and before-and-after observational study. Lancet. 2012;379(9820):1005–1012. [PubMed]
40. Berrouiguet S, Gravey M, Le Galudec M, Alavi Z, Walter M. Post-acute crisis text messaging outreach for suicide prevention: a pilot study. Psychiatry Res. 2014;217(3):154–157. [PubMed]
41. Valenstein M, Kim HM, Ganoczy D, McCarthy JF, Zivin K, Austin KL, et al. Higher-risk periods for suicide among VA patients receiving depression treatment: prioritizing suicide prevention efforts. J Affect Disord. 2009;112(1-3):50–58. [PMC free article] [PubMed]
42. LeardMann CA, Powell TM, Smith TC, Bell MR, Smith B, Boyko EJ, et al. Risk factors associated with suicide in current and former US military personnel. JAMA. 2013;310(5):496–506. [PubMed]
43. Brenner LA, Ignacio RV, Blow FC. Suicide and traumatic brain injury among individuals seeking Veterans Health Administration services. J Head Trauma Rehabil. 2011;26(4):257–264. [PubMed]
44. Nowrangi MA, Kortte KB, Rao VA. A perspectives approach to suicide after traumatic brain injury: case and review. Psychosomatics. 2014;55(5):430–437. [PubMed]
45. Reger MA, Smolenski DJ, Skopp NA, Metzger-Abamukang MJ, Kang HK, Bullman TA, et al. Risk of Suicide Among US Military Service Members Following Operation Enduring Freedom or Operation Iraqi Freedom Deployment and Separation From the US Military. JAMA Psychiatry. 2015;72(6):561–569. [PubMed]
46. Erdman HP, Greist JH, Gustafson DH, Taves JE, Klein MH. Suicide risk prediction by computer interview: a prospective study. J Clin Psychiatry. 1987;48(12):464–467. [PubMed]
47. Gustafson DH, Greist JH, Stauss FF, Erdman H, Laughren T. A probabilistic system for identifying suicide attemptors. Comput Biomed Res. 1977;10(2):83–89. [PubMed]
48. Gustafson DH, Tianen B, Greist JH. A computer-based system for identifying suicide attemptors. Comput Biomed Res. 1981;14(2):144–157. [PubMed]