Search tips
Search criteria

Results 1-25 (95)

Clipboard (0)

Select a Filter Below

more »
Year of Publication
more »
1.  Improving the Transparency of Prognosis Research: The Role of Reporting, Data Sharing, Registration, and Protocols 
PLoS Medicine  2014;11(7):e1001671.
George Peat and colleagues review and discuss current approaches to transparency and published debates and concerns about efforts to standardize prognosis research practice, and make five recommendations.
Please see later in the article for the Editors' Summary
PMCID: PMC4086727  PMID: 25003600
2.  Evaluation of markers and risk prediction models: Overview of relationships between NRI and decision-analytic measures 
For the evaluation and comparison of markers and risk prediction models, various novel measures have recently been introduced as alternatives to the commonly used difference in the area under the ROC curve (ΔAUC). The Net Reclassification Improvement (NRI) is increasingly popular to compare predictions with one or more risk thresholds, but decision-analytic approaches have also been proposed.
We aimed to identify the mathematical relationships between novel performance measures for the situation that a single risk threshold T is used to classify patients as having the outcome or not.
We considered the NRI and three utility-based measures that take misclassification costs into account: difference in Net Benefit (ΔNB), difference in Relative Utility (ΔRU), and weighted NRI (wNRI). We illustrate the behavior of these measures in 1938 women suspect of ovarian cancer (prevalence 28%).
The three utility-based measures appear transformations of each other, and hence always lead to consistent conclusions. On the other hand, conclusions may differ when using the standard NRI, depending on the adopted risk threshold T, prevalence P and the obtained differences in sensitivity and specificity of the two models that are compared. In the case study, adding the CA-125 tumor marker to a baseline set of covariates yielded a negative NRI yet a positive value for the utility-based measures.
The decision-analytic measures are each appropriate to indicate the clinical usefulness of an added marker or compare prediction models, since these measures each reflect misclassification costs. This is of practical importance as these measures may thus adjust conclusions based on purely statistical measures. A range of risk thresholds should be considered in applying these measures.
PMCID: PMC4066820  PMID: 23313931
3.  Prediction of Survival with Alternative Modeling Techniques Using Pseudo Values 
PLoS ONE  2014;9(6):e100234.
The use of alternative modeling techniques for predicting patient survival is complicated by the fact that some alternative techniques cannot readily deal with censoring, which is essential for analyzing survival data. In the current study, we aimed to demonstrate that pseudo values enable statistically appropriate analyses of survival outcomes when used in seven alternative modeling techniques.
In this case study, we analyzed survival of 1282 Dutch patients with newly diagnosed Head and Neck Squamous Cell Carcinoma (HNSCC) with conventional Kaplan-Meier and Cox regression analysis. We subsequently calculated pseudo values to reflect the individual survival patterns. We used these pseudo values to compare recursive partitioning (RPART), neural nets (NNET), logistic regression (LR) general linear models (GLM) and three variants of support vector machines (SVM) with respect to dichotomous 60-month survival, and continuous pseudo values at 60 months or estimated survival time. We used the area under the ROC curve (AUC) and the root of the mean squared error (RMSE) to compare the performance of these models using bootstrap validation.
Of a total of 1282 patients, 986 patients died during a median follow-up of 66 months (60-month survival: 52% [95% CI: 50%−55%]). The LR model had the highest optimism corrected AUC (0.791) to predict 60-month survival, followed by the SVM model with a linear kernel (AUC 0.787). The GLM model had the smallest optimism corrected RMSE when continuous pseudo values were considered for 60-month survival or the estimated survival time followed by SVM models with a linear kernel. The estimated importance of predictors varied substantially by the specific aspect of survival studied and modeling technique used.
The use of pseudo values makes it readily possible to apply alternative modeling techniques to survival problems, to compare their performance and to search further for promising alternative modeling techniques to analyze survival time.
PMCID: PMC4065009  PMID: 24950066
4.  [No title available] 
Heart  2007;93(10):1293.
PMCID: PMC2000926
elderly; clopidogrel; glycoprotein IIb/IIIa blockers
5.  Effects of platelet glycoprotein IIb/IIIa receptor blockers in non‐ST segment elevation acute coronary syndromes: benefit and harm in different age subgroups 
Heart  2006;93(4):450-455.
To investigate whether the beneficial and harmful effects of platelet glycoprotein IIb/IIIa receptor blockers in non‐ST elevation acute coronary syndromes (NSTE‐ACS) depend on age.
A meta‐analysis of six trials of platelet glycoprotein IIb/IIIa receptor blockers in patients with NSTE‐ACS (PRISM, PRISM‐PLUS, PARAGON‐A, PURSUIT, PARAGON‐B, GUSTO IV‐ACS; n = 31 402) was performed. We applied multivariable logistic regression analyses to evaluate the drug effects on death or non‐fatal myocardial infarction at 30 days, and on major bleeding, by age subgroups (<60, 60–69, 70–79, ⩾80 years). We quantified the reduction of death or myocardial infarction as the number needed to treat (NNT), and the increase of major bleeding as the number needed to harm (NNH).
Subgroups had 11 155 (35%), 9727 (31%), 8468 (27%) and 2049 (7%) patients, respectively. The relative benefit of platelet glycoprotein IIb/IIIa receptor blockers did not differ significantly (p = 0.5) between age subgroups (OR (95% CI) for death or myocardial infarction: 0.86 (0.74 to 0.99), 0.90 (0.80 to 1.02), 0.97 (0.86 to 1.10), 0.90 (0.73 to 1.16); overall 0.91 (0.86 to 0.99). ORs for major bleeding were 1.9 (1.3 to 2.8), 1.9 (1.4 to 2.7), 1.6 (1.2 to 2.1) and 2.5 (1.5–4.1). Overall NNT was 105, and overall NNH was 90. The oldest patients had larger absolute increases in major bleeding, but also had the largest absolute reductions of death or myocardial infarction. Patients ⩾80 years had half of the NNT and a third of the NNH of patients <60 years.
In patients with NSTE‐ACS, the relative reduction of death or non‐fatal myocardial infarction with platelet glycoprotein IIb/IIIa receptor blockers was independent of patient age. Larger absolute outcome reductions were seen in older patients, but with a higher risk of major bleeding. Close monitoring of these patients is warranted.
PMCID: PMC1861476  PMID: 17065179
6.  Risk Prediction Scores for Recurrence and Progression of Non-Muscle Invasive Bladder Cancer: An International Validation in Primary Tumours 
PLoS ONE  2014;9(6):e96849.
We aimed to determine the validity of two risk scores for patients with non-muscle invasive bladder cancer in different European settings, in patients with primary tumours.
We included 1,892 patients with primary stage Ta or T1 non-muscle invasive bladder cancer who underwent a transurethral resection in Spain (n = 973), the Netherlands (n = 639), or Denmark (n = 280). We evaluated recurrence-free survival and progression-free survival according to the European Organisation for Research and Treatment of Cancer (EORTC) and the Spanish Urological Club for Oncological Treatment (CUETO) risk scores for each patient and used the concordance index (c-index) to indicate discriminative ability.
The 3 cohorts were comparable according to age and sex, but patients from Denmark had a larger proportion of patients with the high stage and grade at diagnosis (p<0.01). At least one recurrence occurred in 839 (44%) patients and 258 (14%) patients had a progression during a median follow-up of 74 months. Patients from Denmark had the highest 10-year recurrence and progression rates (75% and 24%, respectively), whereas patients from Spain had the lowest rates (34% and 10%, respectively). The EORTC and CUETO risk scores both predicted progression better than recurrence with c-indices ranging from 0.72 to 0.82 while for recurrence, those ranged from 0.55 to 0.61.
The EORTC and CUETO risk scores can reasonably predict progression, while prediction of recurrence is more difficult. New prognostic markers are needed to better predict recurrence of tumours in primary non-muscle invasive bladder cancer patients.
PMCID: PMC4048166  PMID: 24905984
7.  Predictive Value of Updating Framingham Risk Scores with Novel Risk Markers in the U.S. General Population 
PLoS ONE  2014;9(2):e88312.
According to population-based cohort studies CT coronary calcium score (CTCS), carotid intima-media thickness (cIMT), high-sensitivity C- reactive protein (CRP), and ankle-brachial index (ABI) are promising novel risk markers for improving cardiovascular risk assessment. Their impact in the U.S. general population is however uncertain. Our aim was to estimate the predictive value of four novel cardiovascular risk markers for the U.S. general population.
Methods and Findings
Risk profiles, CRP and ABI data of 3,736 asymptomatic subjects aged 40 or older from the National Health and Nutrition Examination Survey (NHANES) 2003–2004 exam were used along with predicted CTCS and cIMT values. For each subject, we calculated 10-year cardiovascular risks with and without each risk marker. Event rates adjusted for competing risks were obtained by microsimulation. We assessed the impact of updated 10-year risk scores by reclassification and C-statistics. In the study population (mean age 56±11 years, 48% male), 70% (80%) were at low (<10%), 19% (14%) at intermediate (≥10–<20%), and 11% (6%) at high (≥20%) 10-year CVD (CHD) risk. Net reclassification improvement was highest after updating 10-year CVD risk with CTCS: 0.10 (95%CI 0.02–0.19). The C-statistic for 10-year CVD risk increased from 0.82 by 0.02 (95%CI 0.01–0.03) with CTCS. Reclassification occurred most often in those at intermediate risk: with CTCS, 36% (38%) moved to low and 22% (30%) to high CVD (CHD) risk. Improvements with other novel risk markers were limited.
Only CTCS appeared to have significant incremental predictive value in the U.S. general population, especially in those at intermediate risk. In future research, cost-effectiveness analyses should be considered for evaluating novel cardiovascular risk assessment strategies.
PMCID: PMC3928195  PMID: 24558385
8.  Improving the Manchester Triage System for Pediatric Emergency Care: An International Multicenter Study 
PLoS ONE  2014;9(1):e83267.
This multicenter study examines the performance of the Manchester Triage System (MTS) after changing discriminators, and with the addition use of abnormal vital sign in patients presenting to pediatric emergency departments (EDs).
International multicenter study
EDs of two hospitals in The Netherlands (2006–2009), one in Portugal (November–December 2010), and one in UK (June–November 2010).
Children (<16years) triaged with the MTS who presented at the ED.
Changes to discriminators (MTS 1) and the value of including abnormal vital signs (MTS 2) were studied to test if this would decrease the number of incorrect assignment. Admission to hospital using the new MTS was compared with those in the original MTS. Likelihood ratios, diagnostic odds ratios (DORs), and c-statistics were calculated as measures for performance and compared with the original MTS. To calculate likelihood ratios and DORs, the MTS had to be dichotomized in low urgent and high urgent.
60,375 patients were included, of whom 13% were admitted. When MTS 1 was used, admission to hospital increased from 25% to 29% for MTS ‘very urgent’ patients and remained similar in lower MTS urgency levels. The diagnostic odds ratio improved from 4.8 (95%CI 4.5–5.1) to 6.2 (95%CI 5.9–6.6) and the c-statistic remained 0.74. MTS 2 did not improve the performance of the MTS.
MTS 1 performed slightly better than the original MTS. The use of vital signs (MTS 2) did not improve the MTS performance.
PMCID: PMC3893080  PMID: 24454699
9.  Assessing discriminative ability of risk models in clustered data 
The discriminative ability of a risk model is often measured by Harrell’s concordance-index (c-index). The c-index estimates for two randomly chosen subjects the probability that the model predicts a higher risk for the subject with poorer outcome (concordance probability). When data are clustered, as in multicenter data, two types of concordance are distinguished: concordance in subjects from the same cluster (within-cluster concordance probability) and concordance in subjects from different clusters (between-cluster concordance probability). We argue that the within-cluster concordance probability is most relevant when a risk model supports decisions within clusters (e.g. who should be treated in a particular center). We aimed to explore different approaches to estimate the within-cluster concordance probability in clustered data.
We used data of the CRASH trial (2,081 patients clustered in 35 centers) to develop a risk model for mortality after traumatic brain injury. To assess the discriminative ability of the risk model within centers we first calculated cluster-specific c-indexes. We then pooled the cluster-specific c-indexes into a summary estimate with different meta-analytical techniques. We considered fixed effect meta-analysis with different weights (equal; inverse variance; number of subjects, events or pairs) and random effects meta-analysis. We reflected on pooling the estimates on the log-odds scale rather than the probability scale.
The cluster-specific c-index varied substantially across centers (IQR = 0.70-0.81; I 2 = 0.76 with 95% confidence interval 0.66 to 0.82). Summary estimates resulting from fixed effect meta-analysis ranged from 0.75 (equal weights) to 0.84 (inverse variance weights). With random effects meta-analysis – accounting for the observed heterogeneity in c-indexes across clusters – we estimated a mean of 0.77, a between-cluster variance of 0.0072 and a 95% prediction interval of 0.60 to 0.95. The normality assumptions for derivation of a prediction interval were better met on the probability than on the log-odds scale.
When assessing the discriminative ability of risk models used to support decisions at cluster level we recommend meta-analysis of cluster-specific c-indexes. Particularly, random effects meta-analysis should be considered.
PMCID: PMC3897966  PMID: 24423445
Clustered data; Concordance; Discrimination; Meta-analysis; Prediction; Risk model
10.  Situational awareness, relational coordination and integrated care delivery to hospitalized elderly in The Netherlands: a comparison between hospitals 
BMC Geriatrics  2014;14:3.
It is known that interprofessional collaboration is crucial for integrated care delivery, yet we are still unclear about the underlying mechanisms explaining effectiveness of integrated care delivery to older patients. In addition, we lack research comparing integrated care delivery between hospitals. Therefore, this study aims to (i) provide insight into the underlying components ‘relational coordination’ and ‘situational awareness’ of integrated care delivery and the role of team and organizational context in integrated care delivery; and (ii) compare situational awareness, relational coordination, and integrated care delivery of different hospitals in the Netherlands.
This cross-sectional study took place in 2012 among professionals from three different hospitals involved in the delivery of care to older patients. A total of 215 professionals filled in the questionnaire (42% response rate).Descriptive statistics and paired-sample t-tests were used to investigate the level of situational awareness, relational coordination, and integrated care delivery in the three different hospitals. Correlation and multilevel analyses were used to investigate the relationship between background characteristics, team context, organizational context, situational awareness, relational coordination and integrated care delivery.
No differences in background characteristics, team context, organizational context, situational awareness, relational coordination and integrated care delivery were found among the three hospitals. Correlational analysis revealed that situational awareness (r = 0.30; p < 0.01), relational coordination (r = 0.17; p < 0.05), team climate (r = 0.29; p < 0.01), formal internal communication (r = 0.46; p < 0.01), and informal internal communication (r = 0.36; p < 0.01) were positively associated with integrated care delivery. Stepwise multilevel analyses showed that formal internal communication (p < 0.001) and situational awareness (p < 0.01) were associated with integrated care delivery. Team climate was not significantly associated with integrated care delivery when situational awareness and relational coordination were included in the equation. Thus situational awareness acted as mediator between team climate and integrated care delivery among professionals delivering care to older hospitalized patients.
The results of this study show the importance of formal internal communication and situational awareness for quality of care delivery to hospitalized older patients.
PMCID: PMC3890569  PMID: 24410889
11.  The Development of the Older Persons and Informal Caregivers Survey Minimum DataSet (TOPICS-MDS): A Large-Scale Data Sharing Initiative 
PLoS ONE  2013;8(12):e81673.
In 2008, the Ministry of Health, Welfare and Sport commissioned the National Care for the Elderly Programme. While numerous research projects in older persons’ health care were to be conducted under this national agenda, the Programme further advocated the development of The Older Persons and Informal Caregivers Survey Minimum DataSet (TOPICS-MDS) which would be integrated into all funded research protocols. In this context, we describe TOPICS data sharing initiative (
Materials and Methods
A working group drafted TOPICS-MDS prototype, which was subsequently approved by a multidisciplinary panel. Using instruments validated for older populations, information was collected on demographics, morbidity, quality of life, functional limitations, mental health, social functioning and health service utilisation. For informal caregivers, information was collected on demographics, hours of informal care and quality of life (including subjective care-related burden).
Between 2010 and 2013, a total of 41 research projects contributed data to TOPICS-MDS, resulting in preliminary data available for 32,310 older persons and 3,940 informal caregivers. The majority of studies sampled were from primary care settings and inclusion criteria differed across studies.
TOPICS-MDS is a public data repository which contains essential data to better understand health challenges experienced by older persons and informal caregivers. Such findings are relevant for countries where increasing health-related expenditure has necessitated the evaluation of contemporary health care delivery. Although open sharing of data can be difficult to achieve in practice, proactively addressing issues of data protection, conflicting data analysis requests and funding limitations during TOPICS-MDS developmental phase has fostered a data sharing culture. To date, TOPICS-MDS has been successfully incorporated into 41 research projects, thus supporting the feasibility of constructing a large (>30,000 observations), standardised dataset pooled from various study protocols with different sampling frameworks. This unique implementation strategy improves efficiency and facilitates individual-level data meta-analysis.
PMCID: PMC3852259  PMID: 24324716
12.  Prevalence and Phenotypes of APC and MUTYH Mutations in Patients with Multiple Colorectal Adenomas 
Patients with multiple colorectal adenomas may carry germline mutations in the APC or MUTYH genes.
To determine the prevalence of pathogenic APC and MUTYH mutations in patients who had undergone genetic testing and compare the prevalence and clinical characteristics of APC and MUTYH mutation carriers.
Design, Setting and Participants
This cross-sectional study consisted of 8676 unrelated individuals who had undergone full gene sequencing and large rearrangement analysis of the APC gene and targeted sequence analysis for the two most common MUTYH mutations (Y179C and G396D) between 2004 and 2011. Individuals with either mutation underwent full MUTYH gene sequencing. We evaluated APC and MUTYH mutation prevalence by polyp burden and the clinical characteristics associated with a pathogenic mutation using logistic regression analyses.
Main Outcome Measure
Deleterious mutations in APC and MUTYH genes.
Colorectal adenomas were reported in 7225 individuals; 1457 with classic polyposis (≥ 100 adenomas) and 3253 with attenuated polyposis (20-99 adenomas). The prevalence of APC and biallelic MUTYH mutations was 95/119 (80%, 95%CI 71-87%) and 2/119 (2%, 95%CI 0.2-6%) among individuals with ≥ 1000 adenomas, 756/1338 (56%, 95%CI 54-59%) and 94/1338 (7%, 95%CI 6-8%) among individuals with 100-999 adenomas, 326/3253 (10%, 95%CI (9-11%) and 233/3253 (7%, 95%CI 6-8%) among individuals with 20-99 adenomas, and 50/970 (5%, 95%CI 4-7%) and 37/970 (4%, 95%CI 3-5%) among those with 10-19 adenomas.
Among patients with multiple colorectal adenomas, APC and MUTYH mutation prevalence varied considerably by adenoma count including within those with a classic polyposis phenotype. APC mutations predominate in patients with classic polyposis, whereas prevalence of APC and MYH mutations is similar in attenuated polyposis. These findings require external validation.
PMCID: PMC3770297  PMID: 22851115
13.  Individual participant data meta-analyses should not ignore clustering 
Journal of Clinical Epidemiology  2013;66(8):865-873.e4.
Individual participant data (IPD) meta-analyses often analyze their IPD as if coming from a single study. We compare this approach with analyses that rather account for clustering of patients within studies.
Study Design and Setting
Comparison of effect estimates from logistic regression models in real and simulated examples.
The estimated prognostic effect of age in patients with traumatic brain injury is similar, regardless of whether clustering is accounted for. However, a family history of thrombophilia is found to be a diagnostic marker of deep vein thrombosis [odds ratio, 1.30; 95% confidence interval (CI): 1.00, 1.70; P = 0.05] when clustering is accounted for but not when it is ignored (odds ratio, 1.06; 95% CI: 0.83, 1.37; P = 0.64). Similarly, the treatment effect of nicotine gum on smoking cessation is severely attenuated when clustering is ignored (odds ratio, 1.40; 95% CI: 1.02, 1.92) rather than accounted for (odds ratio, 1.80; 95% CI: 1.29, 2.52). Simulations show models accounting for clustering perform consistently well, but downwardly biased effect estimates and low coverage can occur when ignoring clustering.
Researchers must routinely account for clustering in IPD meta-analyses; otherwise, misleading effect estimates and conclusions may arise.
PMCID: PMC3717206  PMID: 23651765
Individual participant data meta-analysis; Individual patient data; Evidence synthesis; Cluster; Simulation; Binary outcome; Pooled analysis
14.  Comparison of Scoring Methods for ACE-27: Simpler Is Better 
Journal of geriatric oncology  2012;3(3):238-245.
To examine the prognostic value of different comorbidity coding schemes for predicting survival of newly diagnosed elderly cancer patients.
Materials and Methods
We analyzed data from 8,867 patients aged 65 years of age or older, newly diagnosed with cancer. Comorbidities present at the time of diagnosis were collected using the Adult Comorbidity Evaluation-27 index (ACE-27). We examined multiple scoring schemes based on the individual comorbidity ailments, and their severity rating. Harrell’s c index and Akaike Information Criterion (AIC) were used to evaluate the performance of the different comorbidity models.
Comorbidity led to an increase in c index from 0.771 for the base model to 0.782 for a model that included indicator variables for every ailment. The prognostic value was however much higher for prostate and breast cancer patients. A simple model which considered linear scores from 0 to 3 per ailment, controlling for cancer type, was optimal according to AIC.
The presence of comorbidity impacts on the survival of elderly cancer patients, especially for less lethal cancers, such as prostate and breast cancers. Different ailments have different impacts on survival, necessitating the use of different weights per ailment in a simple summary score of the ACE-27.
PMCID: PMC3375822  PMID: 22712031
Comorbidity; comorbid ailment; elderly; cancer patients; prognostic; survival
15.  Influences of hospital information systems, indicator data collection and computation on reported Dutch hospital performance indicator scores 
For health care performance indicators (PIs) to be reliable, data underlying the PIs are required to be complete, accurate, consistent and reproducible. Given the lack of regulation of the data-systems used in the Netherlands, and the self-report based indicator scores, one would expect heterogeneity with respect to the data collection and the ways indicators are computed. This might affect the reliability and plausibility of the nationally reported scores.
We aimed to investigate the extent to which local hospital data collection and indicator computation strategies differ and how this affects the plausibility of self-reported indicator scores, using survey results of 42 hospitals and data of the Dutch national quality database.
The data collection and indicator computation strategies of the hospitals were substantially heterogenic. Moreover, the Hip and Knee replacement PI scores can be regarded as largely implausible, which was, to a great extent, related to a limited (computerized) data registry. In contrast, Breast Cancer PI scores were more plausible, despite the incomplete data registry and limited data access. This might be explained by the role of the regional cancer centers that collect most of the indicator data for the national cancer registry, in a standardized manner. Hospitals can use cancer registry indicator scores to report to the government, instead of their own locally collected indicator scores.
Indicator developers, users and the scientific field need to focus more on the underlying (heterogenic) ways of data collection and conditional data infrastructures. Countries that have a liberal software market and are aiming to implement a self-report based performance indicator system to obtain health care transparency, should secure the accuracy and precision of the heath care data from which the PIs are calculated. Moreover, ongoing research and development of PIs and profound insight in the clinical practice of data registration is warranted.
PMCID: PMC3698115  PMID: 23758921
Performance indicators; Health care quality; Reliability; Hospital information system
16.  Predicting 14-Day Mortality after Severe Traumatic Brain Injury: Application of the IMPACT Models in the Brain Trauma Foundation TBI-trac® New York State Database 
Journal of Neurotrauma  2012;29(7):1306-1312.
Prognostic models for outcome prediction in patients with traumatic brain injury (TBI) are important instruments in both clinical practice and research. To remain current a continuous process of model validation is necessary. We aimed to investigate the performance of the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) prognostic models in predicting mortality in a contemporary New York State TBI registry developed and maintained by the Brain Trauma Foundation. The Brain Trauma Foundation (BTF) TBI-trac® database contains data on 3125 patients who sustained severe TBI (Glasgow Coma Scale [GCS] score ≤8) in New York State between 2000 and 2009. The outcome measure was 14-day mortality. To predict 14-day mortality with admission data, we adapted the IMPACT Core and Extended models. Performance of the models was assessed by determining calibration (agreement between observed and predicted outcomes), and discrimination (separation of those patients who die from those who survive). Calibration was explored graphically with calibration plots. Discrimination was expressed by the area under the receiver operating characteristic (ROC) curve (AUC). A total of 2513 out of 3125 patients in the BTF database met the inclusion criteria. The 14-day mortality rate was 23%. The models showed excellent calibration. Mean predicted probabilities were 20% for the Core model and 24% for the Extended model. Both models showed good discrimination with AUCs of 0.79 (Core) and 0.83 (Extended). We conclude that the IMPACT models validly predict 14-day mortality in the BTF database, confirming generalizability of these models for outcome prediction in TBI patients.
PMCID: PMC3335134  PMID: 22150207
external validation; outcome; prediction models; traumatic brain injury
17.  Prediction of Outcome after Moderate and Severe Traumatic Brain Injury: External Validation of the IMPACT and CRASH Prognostic Models 
Critical Care Medicine  2012;40(5):1609-1617.
The International Mission on Prognosis and Analysis of Clinical Trials (IMPACT) and Corticoid Randomisation After Significant Head injury (CRASH) prognostic models predict outcome after traumatic brain injury (TBI) but have not been compared in large datasets. The objective of this is study is to validate externally and compare the IMPACT and CRASH prognostic models for prediction of outcome after moderate or severe TBI.
External validation study.
We considered 5 new datasets with a total of 9036 patients, comprising three randomized trials and two observational series, containing prospectively collected individual TBI patient data.
Outcomes were mortality and unfavourable outcome, based on the Glasgow Outcome Score (GOS) at six months after injury. To assess performance, we studied the discrimination of the models (by AUCs), and calibration (by comparison of the mean observed to predicted outcomes and calibration slopes).
Main Results
The highest discrimination was found in the TARN trauma registry (AUCs between 0.83 and 0.87), and the lowest discrimination in the Pharmos trial (AUCs between 0.65 and 0.71). Although differences in predictor effects between development and validation populations were found (calibration slopes varying between 0.58 and 1.53), the differences in discrimination were largely explained by differences in case-mix in the validation studies. Calibration was good, the fraction of observed outcomes generally agreed well with the mean predicted outcome. No meaningful differences were noted in performance between the IMPACT and CRASH models. More complex models discriminated slightly better than simpler variants.
Since both the IMPACT and the CRASH prognostic models show good generalizability to more recent data, they are valid instruments to quantify prognosis in TBI.
PMCID: PMC3335746  PMID: 22511138
external validation; outcome; prediction models; traumatic brain injury
18.  Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury 
Journal of clinical epidemiology  2011;65(5):474-481.
We aimed to determine to what extent covariate adjustment could affect power in a randomized controlled trial (RCT) of a heterogeneous population with traumatic brain injury (TBI).
Study Design and Setting
We analyzed 14-day mortality in 9497 participants in the Corticosteroid Randomisation After Significant Head Injury (CRASH) RCT of corticosteroid vs. placebo. Adjustment was made using logistic regression for baseline covariates of two validated risk models derived from external data (IMPACT) and from the CRASH data. The relative sample size (RESS) measure, defined as the ratio of the sample size required by an adjusted analysis to attain the same power as the unadjusted reference analysis, was used to assess the impact of adjustment.
Corticosteroid was associated with higher mortality compared to placebo (OR=1.25, 95% CI: 1.13, 1.39). RESS of 0.79 and 0.73 were obtained by adjustment using the IMPACT and CRASH models, respectively, which for example implies an increase from 80% to 88% and 91% power, respectively.
Moderate gains in power may be obtained using covariate adjustment from logistic regression in heterogeneous conditions such as TBI. Although analyses of RCTs might consider covariate adjustment to improve power, we caution against this approach in the planning of RCTs.
PMCID: PMC3589911  PMID: 22169080
covariate adjustment; prognostic targeting; strict selection; relative sample size; power in clinical trials; traumatic brain injury
19.  Assessing the incremental value of diagnostic and prognostic markers: a review and illustration 
New markers may improve prediction of diagnostic and prognostic outcomes. We review various measures to quantify the incremental value of markers over standard, readily available characteristics. Widely used traditional measures include the improvement in model fit or in the area under the receiver operating characteristic (ROC) curve (AUC). New measures include the net reclassification index (NRI) and decision–analytic measures, such as the fraction of true positive classifications penalized for false positive classifications (‘net benefit’, NB).
For illustration we discuss a case study on the presence of residual tumor versus benign tissue in 544 patients with testicular cancer. We assessed 3 tumor markers (AFP, HCG, and LDH) for their incremental value over currently standard clinical predictors. AUC and R2 values suggested adding continuous LDH and AFP whereas NB only favored HCG as a potentially promising marker at a clinically defendable decision threshold of 20% risk. Results based on the NRI fell in the middle, suggesting reclassification potential of all three markers.
We conclude that improvement in standard discrimination measures, which focus on finding variables that might be promising across all decision thresholds, may not detect the most informative markers at a specific threshold of particular clinical relevance. When a marker is intended to support decision making, calculation of the improvement in a decision–analytic measure, such as NB, is preferable over an overall judgment as obtained from the AUC in ROC analysis.
PMCID: PMC3587963  PMID: 21726217
prediction; logistic regression model; performance measures; incremental value
20.  Assessing the performance of prediction models: a framework for some traditional and novel measures 
Epidemiology (Cambridge, Mass.)  2010;21(1):128-138.
The performance of prediction models can be assessed using a variety of different methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic (ROC) curve), and goodness-of-fit statistics for calibration.
Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.
We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n=544 for model development, n=273 for external validation).
We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for making clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.
PMCID: PMC3575184  PMID: 20010215
21.  Laparoscopic versus Open Peritoneal Dialysis Catheter Insertion: A Meta-Analysis 
PLoS ONE  2013;8(2):e56351.
Peritoneal dialysis is an effective treatment for end-stage renal disease. Key to successful peritoneal dialysis is a well-functioning catheter. The different insertion techniques may be of great importance. Mostly, the standard operative approach is the open technique; however, laparoscopic insertion is increasingly popular. Catheter malfunction is reported up to 35% for the open technique and up to 13% for the laparoscopic technique. However, evidence is lacking to definitely conclude that the laparoscopic approach is to be preferred. This review and meta-analysis was carried out to investigate if one of the techniques is superior to the other.
Comprehensive searches were conducted in MEDLINE, Embase and CENTRAL (the Cochrane Library 2012, issue 10). Reference lists were searched manually. The methodology was in accordance with the Cochrane Handbook for interventional systematic reviews, and written based on the PRISMA-statement.
Three randomized controlled trials and eight cohort studies were identified. Nine postoperative outcome measures were meta-analyzed; of these, seven were not different between operation techniques. Based on the meta-analysis, the proportion of migrating catheters was lower (odds ratio (OR) 0.21, confidence interval (CI) 0.07 to 0.63; P = 0.006), and the one-year catheter survival was higher in the laparoscopic group (OR 3.93, CI 1.80 to 8.57; P = 0.0006).
Based on these results there is some evidence in favour of the laparoscopic insertion technique for having a higher one-year catheter survival and less migration, which would be clinically relevant.
PMCID: PMC3574153  PMID: 23457554
22.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research 
PLoS Medicine  2013;10(2):e1001381.
In this article, the third in the PROGRESS series on prognostic factor research, Sara Schroter and colleagues review how prognostic models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.
PMCID: PMC3564751  PMID: 23393430
23.  Prognosis Research Strategy (PROGRESS) 2: Prognostic Factor Research 
PLoS Medicine  2013;10(2):e1001380.
In the second article in the PROGRESS series on prognostic factor research, Sara Schroter and colleagues discuss the role of prognostic factors in current clinical practice, randomised trials, and developing new interventions, and explain why and how prognostic factor research should be improved.
PMCID: PMC3564757  PMID: 23393429
24.  Comparison of the clinical prediction model PREMM1,2,6 and molecular testing for the systematic identification of Lynch syndrome in colorectal cancer 
Gut  2012;62(2):272-279.
Lynch syndrome is caused by germline mismatch repair (MMR) gene mutations. The PREMM1,2,6 model predicts the likelihood of a MMR gene mutation based on personal and family cancer history.
To compare strategies using PREMM1,2,6 and tumour testing (microsatellite instability (MSI) and/or immunohistochemistry (IHC) staining) to identify mutation carriers.
Data from population-based or clinic-based patients with colorectal cancers enrolled through the Colon Cancer Family Registry were analysed. Evaluation included MSI, IHC and germline mutation analysis for MLH1, MSH2, MSH6 and PMS2. Personal and family cancer histories were used to calculate PREMM1,2,6 predictions. Discriminative ability to identify carriers from non-carriers using the area under the receiver operating characteristic curve (AUC) was assessed. Predictions were based on logistic regression models for (1) cancer assessment using PREMM1,2,6, (2) MSI, (3) IHC for loss of any MMR protein expression, (4) MSI + IHC, (5) PREMM1,2,6 + MSI, (6) PREMM1,2,6 + IHC, (7) PREMM1,2,6 + IHC + MSI.
Among 1651 subjects, 239 (14%) had mutations (90 MLH1, 125 MSH2, 24 MSH6). PREMM1,2,6 discriminated well with AUC 0.90 (95% CI 0.88 to 0.92). MSI alone, IHC alone, or MSI + IHC each had lower AUCs: 0.77, 0.82 and 0.82, respectively. The added value of IHC + PREMM1,2,6 was slightly greater than PREMM1,2,6 + MSI (AUC 0.94 vs 0.93). Adding MSI to PREMM1,2,6 + IHC did not improve discrimination.
PREMM1,2,6 and IHC showed excellent performance in distinguishing mutation carriers from non-carriers and performed best when combined. MSI may have a greater role in distinguishing Lynch syndrome from other familial colorectal cancer subtypes among cases with high PREMM1,2,6 scores where genetic evaluation does not disclose a MMR mutation.
PMCID: PMC3470824  PMID: 22345660
25.  The Prevention and Reactivation Care Program: intervention fidelity matters 
The Prevention and Reactivation Care Program (PReCaP) entails an innovative multidisciplinary, integrated and goal oriented approach aimed at reducing hospital related functional decline among elderly patients. Despite calls for process evaluation as an essential component of clinical trials in the geriatric care field, studies assessing fidelity lag behind the number of effect studies. The threefold purpose of this study was (1) to systematically assess intervention fidelity of the hospital phase of the PReCaP in the first year of the intervention delivery; (2) to improve our understanding of the moderating factors and modifications affecting intervention fidelity; and (3) to explore the feasibility of the PReCaP fidelity assessment in view of the modifications.
Based on the PReCaP description we developed a fidelity instrument incorporating nineteen (n=19) intervention components. A combination of data collection methods was utilized, i.e. data collection from patient records and individual Goal Attainment Scaling care plans, in-depth interviews with stakeholders, and non-participant observations. Descriptive analysis was performed to obtain levels of fidelity of each of the nineteen PReCaP components. Moderating factors were identified by using the Conceptual Framework for Implementation Fidelity.
Ten of the nineteen intervention components were always or often delivered to the group of twenty elderly patients. Moderating factors, such as facilitating strategies and context were useful in explaining the non- or low-adherence of particular intervention components.
Fidelity assessment was carried out to evaluate the adherence to the PReCaP in the Vlietland Ziekenhuis in the Netherlands. Given that the fidelity was assessed in the first year of PReCaP implementation it was commendable that ten of the nineteen intervention components were performed always or often. The adequate delivery of the intervention components strongly depended on various moderating factors. Since the intervention is still developing and undergoing continuous modifications, it has been concluded that the fidelity criteria should evolve with the modified intervention. Furthermore, repeated intervention fidelity assessments will be necessary to ensure a valid and reliable fidelity assessment of the PReCaP.
Trial registration
The Netherlands National Trial Register: NTR2317
PMCID: PMC3566920  PMID: 23351355
Geriatric care intervention; Intervention fidelity; Moderating factors

Results 1-25 (95)