PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of scirepAboutEditorial BoardFor AuthorsScientific Reports
 
Sci Rep. 2017; 7: 46474.
Published online 2017 April 12. doi:  10.1038/srep46474
PMCID: PMC5388874

Development and validation of a prognostic model for predicting 30-day mortality risk in medical patients in emergency department (ED)

Abstract

The primary aim of this prospective study is to develop and validate a new prognostic model for predicting the risk of mortality in Emergency Department (ED) patients. The study involved 1765 patients in the development cohort and 1728 in the validation cohort. The main outcome was mortality up to 30 days after admission. Potential risk factors included clinical characteristics, vital signs, and routine haematological and biochemistry tests. The Bayesian Model Averaging method within the Cox’s regression model was used to identify independent risk factors for mortality. In the development cohort, the incidence of 30-day mortality was 9.8%, and the following factors were associated with a greater risk of mortality: male gender, increased respiratory rate and serum urea, decreased peripheral oxygen saturation and serum albumin, lower Glasgow Coma Score, and admission to intensive care unit. The area under the receiver operating characteristic curve for the model with the listed factors was 0.871 (95% CI, 0.844–0.898) in the development cohort and 0.783 (95% CI, 0.743–0.823) in the validation cohort. Calibration analysis found a close agreement between predicted and observed mortality risk. We conclude that the risk of mortality among ED patients could be accurately predicted by using common clinical signs and biochemical tests.

Medical patients admitted to Emergency Department (ED) are highly heterogeneous in terms of disease spectrum and severity. Mortality is the most important outcome of ED care, and the rate of mortality can be used as a means for better prioritization of care and resource allocation. The rate of mortality among ED patients is high1. In ED, the death of a patient is commonly preceded by a cumulative deterioration of vital signs and clinical abnormalities2,3. Therefore, several prognostic models, including the Rapid Emergency Medicine Score4, Rapid Acute Physiology Score5 and Worthing Physiological Scoring system6, have been developed to make use of the clinical signs and abnormalities for predicting the risk of death in ED patients.

In a previous study7, we demonstrated that the above algorithms had good prognostic performance in the prediction of 30-day mortality in a tertiary hospital in Vietnam. However, the degree of discrimination, as measured by the area under the receiver operating characteristic curve (AUC), was between 0.70 and 0.807, suggesting that there is room for further improvement of the existing prognostic models. In developing countries, EDs are normally overcrowded, and staff constantly struggle with an overwhelming number of patients from unplanned admissions. Moreover, the condition of care and disease severity among patients in developing countries are different from those in industrialized countries, and the difference calls for new prognostic models that can be applicable to ED patients in developing countries.

The present study sought to develop and validate a clinical predictive model for predicting 30-day mortality risk in ED patients by using routinely collected clinical, physiological and vital signs. We demonstrated that there exists a series of models that have comparable predictive accuracy, and that these models can be used to identify high-risk patients in ED. The models reported here can empower medical care providers to individualise ED care and optimise ED utilisation.

Results

The study was carried out in two cohorts: development and validation. The two cohorts were recruited from two independent hospitals. The development cohort was used to derive predictive models, and the validation cohort was used for testing the predictive models.

Between 13 March 2013 and 1 June 2013, we had enrolled 2175 medical patients in ED for development cohort. However, after excluding patients who did not meet the inclusion criteria and patients who withdrew from the study, 1765 patients remained for the model development. During the follow-up period, the 30-day incidence of mortality in the development cohort was 9.8% (n = 173). Baseline characteristics of patients in the development cohort stratified by mortality status are shown in Table 1. The average age of participants was 65.8 years (range: 16–105 years). There was no statistically significant difference in age between survivors and deceased.

Table 1
Clinical characteristics of 1765 patients in the development cohort and 1728 patients in the validation cohort classified by mortality status.

Bivariate analysis (Table 2) showed that the following risk factors were associated with an increased risk of mortality: male gender, advancing age, increased pulse, increased body temperature, lower systolic blood pressure and diastolic blood pressure, increased respiratory rate, reduced peripheral oxygen saturation, increased duration of illness, and lower Glasgow Coma Score (GCS). Moreover, cardiopulmonary resuscitation, mechanical ventilation, and admission to intensive care unit (ICU) were each associated with increased risk of mortality. Among comorbidities, cancer, chronic renal failure, cirrhosis, chronic respiratory failure, heart failure, and diabetes mellitus were also associated with a greater risk of mortality in ED patients.

Table 2
Risk factors for 30-day mortality from the development cohort: bivariate analysis.

Model development

The Bayesian Model Average (BMA) algorithm identified 3 most parsimonious models for predicting the risk of 30-day mortality. The three models included the following factors: gender, respiratory rate, peripheral oxygen saturation, duration of illness, GCS, ICU admission, serum urea, glycaemia, serum albumin, alanine aminotransferase (ALT), and high-sensitivity C-reactive protein (hsCRP).

Model I included 10 factors and had a posterior probability of 5.5%; model II included 8 factors with 4% posterior probability; and model III included 11 factors with 3.8% posterior probability. Several risk factors were present in different models. The common factors in all three models were: gender, respiratory rate, peripheral oxygen saturation, duration of illness, GCS, ICU admission, serum urea, and serum albumin. The hazard ratio (HR) and 95% CI associated with each factor and each model are shown in Table 3. Among the risk factors, ICU admission was associated with the greatest risk or mortality, with the average HR ranging from 5.6 (Model I) to 6.6 (Model II).

Table 3
Association between risk factors and 30-day mortality risk: results of multivariable analyses.

The AUC for the three models (Fig. 1) were comparable. Model I yielded the AUC value of 0.873 (95% CI, 0.848–0.90), not significantly different from that of Model II (AUC = 0.871; 95% CI, 0.844–0.898) or Model III (AUC = 0.873; 95% CI, 0.845–0.90). From these results, it appeared that Model II was the most optimal model, because it has the least number of risk factors but yielded good discriminatory power.

Figure 1
Area under the receiver operating characteristic curve (left panel) and calibration plot of three parsimonious models (middle and right panels).

Model validation

Between 19 October 2013 and 31 March 2014, 2060 patients had been recruited for the validation cohort. After excluding patients who did not meet the inclusion/exclusion criteria, 1728 patients were available for analysis (Fig. 2). The incidence of 30-day mortality was 7.8% (n = 135), and this incidence rate was slightly lower than that in the development cohort (P = 0.04, Fisher’s exact test). When the models derived from the development cohort were applied to predict the risk of mortality in the validation cohort, we found that there was a good agreement between observed and predicted risk of death. The maximum calibration error in predicting probability of mortality was about 4% for Model I and Model III, and 3.4% for Model II. In general, the AUC values obtained from the validation cohort was slightly lower than those obtained from the development cohort, but all AUC values were close to 0.8, a level deemed to be good discrimination. For example, the AUC for Model I, II and III was 0.788, 0.783, and 0.790, respectively (Fig. 1). Based on parameter estimates of Model II, a nomogram was built for individualising the risk of 30-day mortality (Fig. 3).

Figure 2
CONSORT diagram: Description of recruitment of study participants for the development and validation cohort.
Figure 3
Nomogram for predicting the 30-day mortality risk based on Model II.

Discussion

Emergency Departments worldwide are typically overcrowded and understaffed, and these problems have recently become a topic of discussion in emergency medicine. In developing countries, the problem is even more severe as EDs are often under-resourced. Under these circumstances, a risk stratification-based prioritization of care is a sensible approach. However, predicting the risk of mortality in EDs is a challenging task, because patients are not only highly heterogeneous but also medically complicated. In this study, by using a Bayesian approach to combine common clinical and biochemistry tests, we have developed and validated a series of predictive models for individualising the risk of 30-day mortality in ED patients. These models have proven good discrimination and calibration in external validation.

It is commonly thought that clinician’s assessment or clinical intuition can accurately predict the risk of mortality in EDs. However, when dealing with multiple risk factors, clinician’s assessment can be problematic, because they are unable to weigh information in a rational and objective manner. The lack of objectivity can result in inconsistent risk assessment between clinicians8. In some cases, considerable discrepancies between clinical intuition and actual mortality9,10. Therefore, in the presence of multiple risk factors, some of which may be potentially important, statistical thinking offers clinicians some useful guidance for dealing with uncertainty of risk assessment.

One of the most important aspects of statistical model building is to find relevant risk factors that have good predictive values. Typically, researchers develop a hypothesis about the relationship between risk factors and outcome, and then apply an algorithm, mostly stepwise regression, to identify a set of possible relevant risk factors. However, this classical approach suffers from two shortcomings that the stepwise method tends to identify redundant variables11, and that it produces only one “final” model. In reality, there are more than one competing model that can explain the relationship. A better approach is Bayesian Model Averaging which was introduced to scientific research about 20 years ago12, but has not been widely used in medical research. The Bayesian Model Averaging method has been shown to have superior and more robust performance than the stepwise method in the identification of relevant risk factors11,13. In this study, by using the Bayesian Model Averaging approach we have identify three sets of risk factors that can help clinicians estimate the risk of mortality in EDs patients.

Some risk factors for mortality identified in the present study have also been identified in previous studies6,14,15,16,17. It is well known that in ED men have a greater risk of mortality than women18, and this was also observed in our study. In clinical setting, about 45% of patients who died had antecedent abnormal vital signs, including respiratory distress, from 8 hours to 48 hours19. Moreover, among patients with normal vital signs at admission, 30% will be deteriorated within 24 hours20.

Tachypnea was a predictor of mortality in this study after adjusting for other predictors, and this finding is consistent with previous studies4,5,6,16. Peripheral oxygen saturation, another index for monitoring respiration, is also a known risk factor for mortality4,6,16,17. Low oxygen saturation (<90%) increases the odds of mortality within 24 hours by five-fold17, and in our study, decrease in peripheral oxygen saturation had a less pronounced effect, with a hazard ratio of 1.17 after adjusting for other covariates in the model.

In the present study, we found that decreased GCS, increased serum urea, increased glycaemia, decreased serum albumin, increased ALT, and increased hsCRP were independently associated with increased mortality risk. These associations have also been reported by various studies15,21,22,23,24,25. Interestingly, we found that a decision of ICU admission was strongly associated with mortality. Indeed, the risk of mortality among patients with this indication was increased by 6.6-fold compared with those without the indication. It could be argued that decision of ICU admission reflects a “clinical impression” which is known to be a good predictor of mortality status18,26. This finding suggests that clinical intuition can be an important factor, in addition to other measured factors, in the assessment of mortality in ED patients.

A number of multivariable models for predicting mortality in ED patients have been developed, and in validation cohorts, these models have good discrimination. For instance, the AUC for the Early Warning Score (7 factors), Simple Clinical Score (14 factors), Rapid Emergency Medicine Score (7 factors), and Worthing Physiological Scoring system (5 factors) for predicting mortality was 0.7327, 0.82628, 0.7127, and 0.7977, respectively. Among the models, only the Rapid Emergency Medicine Score (REMS) and Worthing Physiological System (WPS) were externally validated in developing countries. However, none of the available prognostic models appears to be suitable for individualised prognosis in ED patients. Our models have comparable AUC to those existing models. In previous studies, combining physician’s prediction with an objective model did not substantially improve the discrimination of mortality26; however, in our study, the integration of physician’s decision of ICU admission with objective models markedly increased the discrimination.

Our results should be interpreted within context of strengths and potential limitations. The study was designed as a prospective investigation with large sample size which allows us to define modest associations between risk factors and mortality. We have rigorously validated our models in a totally independent cohort which avoided the problem of over-fitting commonly found in previous studies. Moreover, we used sophisticated statistical approaches to derive at a series of predictive models, not just a single model that was reported in previous studies. In our view, there exists more than one model that can help predict mortality status, and the reliance on a single model is an underestimate of the uncertainty in the data. However, since patients’ information were collected at one time point, which might not reflect a full spectrum of a patient’s dynamics. The decision of ICU admission is, of course, not objective, and could potentially be a bias.

In summary, we have developed and validated new prognostic models to predict the risk of 30-day mortality in medical ED patients. The models will allow accurate risk assessment to identify high risk patients, and help optimize resources and patient management in ED.

Methods

Setting and patients

The study was conducted at the Can Tho National Hospital, Vietnam (for development cohort) and the Can Tho General Hospital, Vietnam (for validation cohort). The Can Tho National Hospital is a tertiary teaching hospital that serves 17 million residents in the Mekong Delta region; its ED on average admits 75 non-surgical patients per day. The Can Tho General Hospital has 400 beds, providing care to the residents of Can Tho City; on average, the ED admits 50 non-surgical patients per day.

We enrolled medical patients (non-trauma and non-surgical) from the two emergency departments between 13 March 2013 and 31 March 2014. The inclusion criteria were: all patients aged 16 years and older and who could give informed consent. Patients were excluded from the study if they had one of the following conditions: acute coronary syndrome, burns, cardiac arrest before admitting to the hospital or which occurred in the ED with failure of cardiopulmonary resuscitation, snakebite, insect bite or sting, poisoning (drugs, alcohol, intoxication, paraquat, insecticides, rodenticides, corrosive substances). We also excluded patients with burns, cardiac arrest with failure of cardiopulmonary resuscitation. Women in labor and dead-on-arrival patients were also excluded from the study. The study protocol and procedure were approved by the Can Tho National Hospital ethics committee. All patients gave written informed consent. Patients could withdraw from the study at any time without giving reasons. All methods were performed in accordance with the relevant guidelines and regulations.

Study procedure

All patients who met inclusion and exclusion criteria were invited to participate in the study. Upon giving the written informed consent, data collection was conducted by a trained research worker by using a structured questionnaire. The questionnaire collected data concerning demographic characteristics, medical history, physiological data, haematological and biochemistry tests (see Supplementary Table S1). After 30 days of admission, the research worker would contact patients or relatives or guardians to obtain information on survival status. Patients who had stayed in hospital for more than 30 days were considered “censored”.

Outcome measure

The primary outcome of the study was mortality occurred within 30 days of hospital admission. Mortality was defined as (1) death in hospital from any cause; (2) family-initiated discharge and death either on the way home or within 24 hours after discharge; (3) doctor-initiated discharge and death at home. It should be noted that in Vietnamese culture, when a patient was in the end stage of disease, the patient or family often requests for discharge from hospital because they prefer to pass away at home.

Risk factors

Risk factors considered in this study included vital signs, peripheral oxygen saturation, duration of illness, GCS (see Supplementary Table S2), cardiopulmonary resuscitation, mechanical ventilation, ICU admission, functional status, comorbidity, haematological and biochemical test (see Supplementary Table S1). Blood pressure and pulse rate were measured electronically (OMRON HEALTHCARE Co, Vietnam) and rechecked manually where blood pressure was either too low or too high. Peripheral oxygen saturation was measured by an electronic device (NONIN Co, USA). Decision of ICU admission was made by a senior attending physician after considering laboratory tests and primary diagnosis.

Data management and analysis

Data were entered into a designed database twice. The first data entry was made within a day after admission. The second data entry was undertaken at the termination of the study. Data from the two entries were used to check for potential inconsistencies, and any inconsistency was adjudicated with the original patient record.

Data analysis was performed according to a statistical analysis plan prior to the collection of data. First, we used descriptive statistics (i.e., mean, standard deviation, median with interquartile range, proportion) for each clinical and laboratory variable with stratification by survival status. Difference between groups in categorical variables was tested by the Fisher’s exact test. The normal distribution of continuous variables was tested by the Shapiro-Wilk normality test. The association between continuous variables and 30-day mortality was tested by the Student’s t test for normally distributed variables or Mann-Whitney test for non-normally distributed variables.

Second, we used the Cox’s proportional hazards model29 to assess the association between potential risk factors and 30-day mortality (i.e. hazard ratio) from the development cohort. Since there were several potential risk factors and the number of “potential models” for predicting 30-day mortality can be very large, the Bayesian Model Averaging (BMA)30 was used to search for the most parsimonious models from the development cohort. This approach has been shown to have superior performance compared to “traditional” approaches such as stepwise regression11,13. In the BMA approach, the regression analysis was performed for 2M (where M is the number of risk factors) competing models. The BMA averaged point estimates for regression coefficient over the all possible models. BMA produces a posterior probability of each possible model and posterior probability for regression coefficient associated with each genetic variant. The posterior probability is a function of a prior probability and the likelihood of a model. In this study, given the large number of genetic variant and there is little information available for eliciting prior distributions, we used the “uninformative” prior distributions, that a priori, make all models and parameters equally likely are appealing. Thus, in BMA, we consider a set of competing models, not just a single model, to account for the variation in the data. The analysis was done with the R statistical environment31 and the BMA package32.

The discrimination of the most parsimonious models was estimated by the AUC33 for the development and validation cohort. The difference between two AUCs was tested by the Delong method34. Agreement between observed mortality and predicted mortality was evaluated using the bootstrap technique. In this technique, 1000 sub-samples, each with 500 subjects, of the entire sample were repeatedly sampled (with replacement) and analyzed35. Based on the parameters selected from the most parsimonious models, a nomogram using the “rms” library35,36 was constructed for predicting 30-day mortality. In a further validation, the predicted probability of 30-day mortality was compared with the observed probability on the development cohort and validation cohort. All statistical analyses were performed with the R Statistical Environment version 3.1.0 on Windows platform37.

Additional Information

How to cite this article: Ha, D. T. et al. Development and validation of a prognostic model for predicting 30-day mortality risk in medical patients in emergency department (ED). Sci. Rep. 7, 46474; doi: 10.1038/srep46474 (2017).

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Material

Supplementary Information:

Acknowledgments

We acknowledge the assistance of the nursing and medical staff of the Emergency Department, Can Tho National Hospital and Can Tho General Hospital in the data collection. We also thank medical students of the Can Tho University of Medicine for helping in the collection of data.

Footnotes

The authors declare no competing financial interests.

Author Contributions Study conception and study design: D.T.H., T.V.N. Patient recruitment and data collection: D.T.H., N.V.T., T.N.T.P., T.Q.D. Data analysis: D.T.H., N.D.N., T.V.N. Drafting and revising manuscript: D.T.H., T.V.N. Approving final version of manuscript: D.T.H., T.V.N. All authors have read and approved the final version of the manuscript.

References

  • Berthelot S., Lang E. S., Quan H. & Stelfox H. T. Development of a Hospital Standardized Mortality Ratio for Emergency Department Care. Annals of emergency medicine 67, 517–524 e526, doi: (2016).10.1016/j.annemergmed.2015.08.005 [PubMed] [Cross Ref]
  • Buist M., Bernard S., Nguyen T. V., Moore G. & Anderson J. Association between clinically abnormal observations and subsequent in-hospital mortality: a prospective study. Resuscitation 62, 137–141, doi: (2004).10.1016/j.resuscitation.2004.03.005 [PubMed] [Cross Ref]
  • Kause J. et al. . A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom–the ACADEMIA study. Resuscitation 62, 275–282, doi: (2004).10.1016/j.resuscitation.2004.05.016 [PubMed] [Cross Ref]
  • Olsson T., Terent A. & Lind L. Rapid Emergency Medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. Journal of internal medicine 255, 579–587, doi: (2004).10.1111/j.1365-2796.2004.01321.x [PubMed] [Cross Ref]
  • Rhee K. J., Fisher C. J. Jr. & Willitis N. H. The Rapid Acute Physiology Score. The American journal of emergency medicine 5, 278–282 (1987). [PubMed]
  • Duckitt R. W. et al. . Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study. British journal of anaesthesia 98, 769–774, doi: (2007).10.1093/bja/aem097 [PubMed] [Cross Ref]
  • Ha D. T. et al. . Prognostic performance of the Rapid Emergency Medicine Score (REMS) and Worthing Physiological Scoring system (WPS) in emergency department. International journal of emergency medicine 8, 18, doi: (2015).10.1186/s12245-015-0066-3 [PMC free article] [PubMed] [Cross Ref]
  • Chew D. P. et al. . An examination of clinical intuition in risk assessment among acute coronary syndromes patients: observations from a prospective multi-center international observational registry. International journal of cardiology 171, 209–216, doi: (2014).10.1016/j.ijcard.2013.12.010 [PubMed] [Cross Ref]
  • Ivanov J. et al. . Predictive accuracy study: comparing a statistical model to clinicians’ estimates of outcomes after coronary bypass surgery. The Annals of thoracic surgery 70, 162–168 (2000). [PubMed]
  • Cornwell L. D. et al. . Predicting mortality in high-risk coronary artery bypass: surgeon versus risk model. The Journal of surgical research 174, 185–191, doi: (2012).10.1016/j.jss.2011.09.011 [PubMed] [Cross Ref]
  • Genell A., Nemes S., Steineck G. & Dickman P. W. Model selection in medical research: a simulation study comparing Bayesian model averaging and stepwise regression. BMC medical research methodology 10, 108, doi: (2010).10.1186/1471-2288-10-108 [PMC free article] [PubMed] [Cross Ref]
  • Raftery A. E., Madigan D. & Hoeting J. Bayesian model averaging for linear regression models. J Amer Statist Assoc 92, 179–191 (1997).
  • Wang D., Zhang W. & Bakhai A. Comparison of Bayesian model averaging and stepwise methods for model selection in logistic regression. Statistics in medicine 23, 3451–3467, doi: (2004).10.1002/sim.1930 [PubMed] [Cross Ref]
  • Froom P. & Shimoni Z. Prediction of hospital mortality rates by admission laboratory tests. Clinical chemistry 52, 325–328, doi: (2006).10.1373/clinchem.2005.059030 [PubMed] [Cross Ref]
  • Lyons O., Whelan B., Bennett K., O’Riordan D. & Silke B. Serum albumin as an outcome predictor in hospital emergency medical admissions. European journal of internal medicine 21, 17–20, doi: (2010).10.1016/j.ejim.2009.10.010 [PubMed] [Cross Ref]
  • Kellett J. & Deane B. The Simple Clinical Score predicts mortality for 30 days after admission to an acute medical unit. QJM: monthly journal of the Association of Physicians 99, 771–781, doi: (2006).10.1093/qjmed/hcl112 [PubMed] [Cross Ref]
  • Kellett J., Deane B. & Gleeson M. Derivation and validation of a score based on Hypotension, Oxygen saturation, low Temperature, ECG changes and Loss of independence (HOTEL) that predicts early mortality between 15 min and 24 h after admission to an acute medical unit. Resuscitation 78, 52–58, doi: (2008).10.1016/j.resuscitation.2008.02.011 [PubMed] [Cross Ref]
  • Beglinger B. et al. . Physician’s first clinical impression of emergency department patients with nonspecific complaints is associated with morbidity and mortality. Medicine 94, e374, doi: (2015).10.1097/MD.0000000000000374 [PMC free article] [PubMed] [Cross Ref]
  • Hillman K. M. et al. . Antecedents to hospital deaths. Internal medicine journal 31, 343–348 (2001). [PubMed]
  • Henriksen D. P., Brabrand M. & Lassen A. T. Prognosis and risk factors for deterioration in patients admitted to a medical emergency department. PloS one 9, e94649, doi: (2014).10.1371/journal.pone.0094649 [PMC free article] [PubMed] [Cross Ref]
  • Francia E. et al. . Ability of physiological parameters versus clinical categories to predict mortality on admission to an internal medicine ward. European journal of internal medicine 20, 636–639, doi: (2009).10.1016/j.ejim.2009.06.008 [PubMed] [Cross Ref]
  • Mohammed M. A. et al. . Index blood tests and national early warning scores within 24 hours of emergency admission can predict the risk of in-hospital mortality: a model development and validation study. PloS one 8, e64340, doi: (2013).10.1371/journal.pone.0064340 [PMC free article] [PubMed] [Cross Ref]
  • Glynn N., Owens L., Bennett K., Healy M. L. & Silke B. Glucose as a risk predictor in acute medical emergency admissions. Diabetes research and clinical practice 103, 119–126, doi: (2014).10.1016/j.diabres.2013.10.015 [PubMed] [Cross Ref]
  • Kim W. R., Flamm S. L., Di Bisceglie A. M. & Bodenheimer H. C. & Public Policy Committee of the American Association for the Study of Liver, D. Serum activity of alanine aminotransferase (ALT) as an indicator of health and disease. Hepatology 47, 1363–1370, doi: (2008).10.1002/hep.22109 [PubMed] [Cross Ref]
  • Rehman S. U., Martinez-Rumayor A., Mueller T. & Januzzi J. L. Jr. Independent and incremental prognostic value of multimarker testing in acute dyspnea: results from the ProBNP Investigation of Dyspnea in the Emergency Department (PRIDE) study. Clinica chimica acta; international journal of clinical chemistry 392, 41–45, doi: (2008).10.1016/j.cca.2008.03.002 [PubMed] [Cross Ref]
  • Brabrand M., Hallas J. & Knudsen T. Nurses and physicians in a medical admission unit can accurately predict mortality of acutely admitted patients: a prospective cohort study. PloS one 9, e101739, doi: (2014).10.1371/journal.pone.0101739 [PMC free article] [PubMed] [Cross Ref]
  • Corfield A. R. et al. . Utility of a single early warning score in patients with sepsis in the emergency department. Emergency medicine journal : EMJ 31, 482–487, doi: (2014).10.1136/emermed-2012-202186 [PubMed] [Cross Ref]
  • Straede M. & Brabrand M. External validation of the simple clinical score and the HOTEL score, two scores for predicting short-term mortality after admission to an acute medical unit. PloS one 9, e105695, doi: (2014).10.1371/journal.pone.0105695 [PMC free article] [PubMed] [Cross Ref]
  • Cox D. R. Regression Models and Life-Tables. Journal of the Royal Statistical Society. Series B (Methodological) 34, 187–220 (1972).
  • Volinsky C. T., Madigan D., Raftery A. E. & Kronmal R. A. Bayesian Model Averaging in Proportional Hazard Models: Assessing the Risk of a Stroke. J R Stat Soc Ser C Appl Stat 433–448. (1997).
  • Saiman L. et al. . Identification and antimicrobial susceptibility of Alcaligenes xylosoxidans isolated from patients with cystic fibrosis. Journal of clinical microbiology 39, 3942–3945, doi: (2001).10.1128/JCM.39.11.3942-3945.2001 [PMC free article] [PubMed] [Cross Ref]
  • Raftery A., Hoeting J., Volinsky C., Painter I. & Yeung K. Y. BMA: Bayesian model averaging. R package version 3.17.1. (2014).
  • Hanley J. A. & McNeil B. J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 29–36 (1982). [PubMed]
  • DeLong E. R., DeLong D. M. & Clarke-Pearson D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837–845 (1988). [PubMed]
  • Harrell F. E. Jr., Lee K. L. & Mark D. B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine 15, 361–387, doi: (1996).10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4 [PubMed] [Cross Ref]
  • Harrell F. E Jr. et al. . Development of a clinical prediction model for an ordinal outcome: the World Health Organization Multicentre Study of Clinical Signs and Etiological agents of Pneumonia, Sepsis and Meningitis in Young Infants. WHO/ARI Young Infant Multicentre Study Group. Statistics in medicine 17, 909–944 (1998). [PubMed]
  • Team R. C. R. A language and environment for statistical computing http://www.R-project.org/ (2014).

Articles from Scientific Reports are provided here courtesy of Nature Publishing Group