PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of bmcmrmBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Medical Research Methodology
 
BMC Med Res Methodol. 2012; 12: 99.
Published online Jul 20, 2012. doi:  10.1186/1471-2288-12-99
PMCID: PMC3441433
Validating and updating a risk model for pneumonia – a case study
Ulrike Held,corresponding author1 Daniel Sabanes Bové,2 Johann Steurer,1 and Leonhard Held2
1Horten Centre for Patient Oriented Research and Knowledge Transfer, University Hospital Zurich, Zurich, Switzerland
2Institute for Social and Preventive Medicine, Division of Biostatistics, University of Zurich, Zurich, Switzerland
corresponding authorCorresponding author.
Ulrike Held: ulrike.held/at/usz.ch; Daniel Sabanes Bové: daniel.sabanesbove/at/ifspm.uzh.ch; Johann Steurer: johann.steurer/at/usz.ch; Leonhard Held: leonhard.held/at/ifspm.uzh.ch
Received December 19, 2011; Accepted July 20, 2012.
Abstract
Background
The development of risk prediction models is of increasing importance in medical research - their use in practice, however, is rare. Among other reasons this might be due to the fact that thorough validation is often lacking. This study focuses on two Bayesian approaches of how to validate a prediction rule for the diagnosis of pneumonia, and compares them with established validation methods.
Methods
Expert knowledge was used to derive a risk prediction model for pneumonia. Data on more than 600 patients presenting with cough and fever at a general practitioner’s practice in Switzerland were collected in order to validate the expert model and to examine the predictive performance of it. Additionally, four modifications of the original model including shrinkage of the regression coefficients, and two Bayesian approaches with the expert model used as prior mean and different weights for the prior covariance matrix were fitted. We quantify the predictive performance of the different methods with respect to calibration and discrimination, using cross-validation.
Results
The predictive performance of the unshrinked regression coefficients was poor when applied to the Swiss cohort. Shrinkage improved the results, but a Bayesian model formulation with unspecified weight of the informative prior lead to large AUC and small Brier score, naïve and after cross-validation. The advantage of this approach is the flexibility in case of a prior-data conflict.
Conclusions
Published risk prediction rules in clinical research need to be validated externally before they can be used in new settings. We propose to use a Bayesian model formulation with the original risk prediction rule as prior. The posterior means of the coefficients, given the validation data showed best predictive performance with respect to cross-validated calibration and discriminative ability.
Keywords: Validation, Predictive performance, Bayesian model, g-factor, Pneumonia
Articles from BMC Medical Research Methodology are provided here courtesy of
BioMed Central