Search tips
Search criteria

Results 1-25 (1138997)

Clipboard (0)

Related Articles

1.  Evaluating a New Risk Marker’s Predictive Contribution in Survival Models 
Although the area under the receiver operating characteristic (ROC) curve (AUC) is the most popular measure of the performance of prediction models, it has limitations, especially when it is used to evaluate the added discrimination of a new risk marker in an existing risk model. Pencina et al. (2008) proposed two indices, the net reclassification improvement (NRI) and integrated discrimination improvement (IDI), to supplement the improvement in the AUC (IAUC). Their NRI and IDI are based on binary outcomes in case-control settings, which do not involve time-to-event outcome. However, many disease outcomes are time-dependent and the onset time can be censored. Measuring discrimination potential of a prognostic marker without considering time to event can lead to biased estimates. In this paper, we extended the NRI and IDI to time-to-event settings and derived the corresponding sample estimators and asymptotic tests. Simulation studies showed that the time-dependent NRI and IDI have better performance than Pencina’s NRI and IDI for measuring the improved discriminatory power of a new risk marker in prognostic survival models.
PMCID: PMC3439820  PMID: 22984361
Improved discrimination; Prognostic survival models; Time-dependent NRI; Time-dependent IDI
2.  Biomarkers in Peripheral Arterial Disease Patients and Near and Longer Term Mortality 
Journal of vascular surgery  2010;52(1):85-90.
To determine in patients with peripheral arterial disease (PAD) whether novel biomarkers improve prediction of cardiovascular disease (CVD) mortality and total mortality.
Whether novel biomarkers improve risk prediction of mortality beyond standard CVD risk markers in PAD patients, and whether any such prediction differs with length of follow-up, remains controversial.
A cohort of 397 patients were referred to a vascular lab had PAD diagnosed by non-invasive testing. 58% also had coronary or cerebrovascular disease at baseline. Predictors of total, CVD, and non-CVD mortality were assessed with Cox proportional hazards models, and the incremental value of predictors were evaluated with both the C-statistic and the integrated discrimination improvement (IDI) index.
Total mortality was 11 % at 2 years of follow-up and 65 % at an average of 7 years of follow-up (maximum 11.4 years). At 2 years, hs-CRP was a strong and significant predictor of mortality, with a hazard ratio (HR) of 1.56 per standard deviation, p=.006. However, at full follow-up standard CVD risk markers were significant (age, sex, ankle-brachial index [ABI], other CVD, and hypertension), but hs-CRP no longer showed a significant relationship HR = 1.12, p = .11. None of the other biomarkers studied showed a significant independent association with mortality. Hs-CRP improved the C-statistic and the IDI beyond standard risk markers at 2 years, but not at full follow-up.
Hs-CRP was a strong predictor of short-term mortality in this cohort of PAD patients, while standard risk markers were better at predicting longer-term mortality.
PMCID: PMC4077155  PMID: 20471776
biomarkers; peripheral arterial disease; cardiovascular diseases; risk prediction; mortality
3.  Performance of Reclassification Statistics in Comparing Risk Prediction Models 
Concerns have been raised about the use of traditional measures of model fit in evaluating risk prediction models for clinical use, and reclassification tables have been suggested as an alternative means of assessing the clinical utility of a model. Several measures based on the table have been proposed, including the reclassification calibration (RC) statistic, the net reclassification improvement (NRI), and the integrated discrimination improvement (IDI), but the performance of these in practical settings has not been fully examined. We used simulations to estimate the type I error and power for these statistics in a number of scenarios, as well as the impact of the number and type of categories, when adding a new marker to an established or reference model. The type I error was found to be reasonable in most settings, and power was highest for the IDI, which was similar to the test of association. The relative power of the RC statistic, a test of calibration, and the NRI, a test of discrimination, varied depending on the model assumptions. These tools provide unique but complementary information.
PMCID: PMC3395053  PMID: 21294152
Calibration; Discrimination; Model accuracy; Prediction; Reclassification
4.  Differences in biological features of gastric dysplasia, indefinite dysplasia, reactive hyperplasia and discriminant analysis of these lesions 
AIM: To investigate the differences in biological features of gastric dysplasia (Dys), indefinite dysplasia (IDys) and reactive hyperplasia (RH) by studying the biomarker alterations in cell proliferation, cell differentiation, cell cycle control and the expression of house-keeping genes, and further to search for markers which could be used in guiding the pathological diagnosis of three lesions.
METHODS: Expressions of MUC5AC, MUC6, adenomatous polyposis coli (APC), p53, Ki-67, proliferation cell nuclear antigen (PCNA) and EGFR were studied by immunohistochemistry with a standard Envision technique in formalin-fixed and paraffin-embedded specimens from 43 RH, 35 IDys, 35 Dys and 36 intestinal type gastric carcinomas (IGC). In addition, Bayes discriminant analysis was used to investigate the value of markers studied in differential diagnosis of RH, IDys, Dys and IGC.
RESULTS: The MUC5AC and MUC6 antigen expressions in RH, IDys, Dys and IGC decreased gradually (MUC5AC: 86.04%, 77.14%, 28.57%, 6.67%; MUC6: 65.15%, 54.29%, 20.00%, 25.00%, respectively). The expressions of the two markers had no significant difference between RH and IDys, but were all significantly higher than those of the other two lesions (MUC5AC: χ2 = 27.607, 38.027 and 17.33, 26.092; MUC6: χ2 = 16.54, 12.665 and 9.282, 6.737, P<0.01). There was no significant difference between RH and IDys, Dys and IGC in MUC6 expression. The APC gene expression in the four lesions had a similar decreasing tendency (RH 69.76%, IDys 68.57%, Dys 39.39%, IGC 22.86%), and it was significantly higher in the first two lesions than in the last two (χ2 = 7.011, 16.995 and 14.737, 19.817, P<0.05). The p53 expression in RH, IDys, Dys and IGC was 6.98%, 20%, 57.14% and 50%, respectively. There was no significant difference between RH and IDys or Dys and IGC, but the p53 expression in RH and IDys was significantly lower than that in Dys and IGC (χ2 = 7.011, 16.995 and 14.737, 19.817, P<0.01). The Ki-67 label index was significantly different among four lesions (RH: 0.298±8.92%, IDys: 0.358±9.25%, Dys: 0.498±9.03%, IGC: 0.620±10.8%, P<0.001). Positive immunostaining of PCNA was though observed in all specimens, significant differences were detected among four lesions (F = 95.318, P<0.01). In addition, we used Bayes discriminant analysis to investigate molecular pathological classification of the lesions, and obtained the best result with the combination of MUC5AC, Ki-67 and PCNA. The overall rate of correct classification was 67.4% (RH), 68.6% (IDys), 70.6% (Dys) and 84.8% (IGC), respectively.
CONCLUSION: Dys has neoplastic biological characteristics, while RH and IDys display hyperplastic characteristics. MUC5AC and proliferation-related biomarkers (Ki-67, PCNA) are more specific in distinguishing Dys from RH and IDys.
PMCID: PMC4315969  PMID: 15962383
Gastric dysplasia; Indefinite dysplasia; Reactive hyperplasia
5.  Incremental value of T-SPOT.TB for diagnosis of active pulmonary tuberculosis in children in a high-burden setting: a multivariable analysis 
Thorax  2013;68(9):860-866.
Interferon γ release assays (IGRAs) are increasingly used for tuberculosis (TB) infection, but their incremental value beyond patient demographics, clinical signs and conventional tests for active disease has not been evaluated in children.
The incremental value of T-SPOT.TB was assessed in 491 smear-negative children from two hospitals in Cape Town, South Africa. Bayesian model averaging was used to select the optimal set of patient demographics and clinical signs for predicting culture-confirmed TB. The added value of T-SPOT.TB over and above patient characteristics and conventional tests was measured using statistics such as the difference in the area under the receiver operating characteristic curve (AUC), the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI).
Cough longer than 2 weeks, fever longer than 2 weeks, night sweats, malaise, history of household contact and HIV status were the most important predictors of culture-confirmed TB. Binary T-SPOT.TB results did not have incremental value when added to the baseline model with clinical predictors, chest radiography and the tuberculin skin test. The AUC difference was 3% (95% CI 0% to 7%). Using risk cut-offs of <10%, 10–30% and >30%, the NRI was 7% (95% CI −8% to 31%) but the CI included the null value. The IDI was 3% (95% CI 0% to 11%), meaning that the average predicted probability across all possible cut-offs improved marginally by 3%.
In a high-burden setting, the T-SPOT.TB did not have added value beyond clinical data and conventional tests for diagnosis of TB disease in smear-negative children.
PMCID: PMC3862980  PMID: 23674550
Clinical Epidemiology; Tuberculosis
6.  Cholesteryl Esters Associated with ACAT2 Predict Coronary Artery Disease in Patients with Symptoms of Acute Coronary Syndrome 
Identifying the likelihood of a patient having coronary artery disease (CAD) at the time of emergency department (ED) presentation with chest pain could reduce the need for stress testing or coronary imaging after myocardial infarction (MI) has been excluded. The authors aimed to determine if a novel cardiac biomarker consisting of plasma cholesteryl ester (CE) levels typically derived from the activity of the enzyme acyl-CoA:cholesterol acyltransferase (ACAT2) are predictive of CAD in a clinical model.
A single-center prospective cohort design enrolled participants with symptoms of acute coronary syndrome undergoing coronary computed tomography angiography (CCTA) or invasive angiography. Plasma samples were analyzed for CE composition with mass spectrometry. The primary endpoint was any CAD determined at angiography. Multivariable logistic regression analyses were used to estimate the relationship between the sum of the plasma concentrations from cholesteryl palmitoleate (16:1) and cholesteryl oleate (18:1) (defined as ACAT2-CE) and the presence of CAD. The added value of ACAT2-CE to the model was analyzed comparing the C statistics and integrated discrimination improvement (IDI).
The study cohort was comprised of 113 participants with a mean (± standard deviation [SD]) age 49 (SD ± 11.7) years, 59% had CAD at angiography, and 23% had an MI within 30 days. The median (interquartile range [IQR]) plasma concentration of ACAT2-CE was 938 μM (IQR, 758 to 1,099 μM) in patients with CAD and 824 μM (IQR 683 to 998 μM) in patients without CAD (p = 0.03). When considered with age, sex, and the number of conventional CAD risk factors, ACAT2-CE levels were associated with a 6.5% increased odds of having CAD per 10 μM increase in concentration. The addition of ACAT2-CE significantly improved the C statistic (0.89 vs 0.95, p = 0.0035), and IDI (0.15, p < 0.001) compared to the reduced model. In the subgroup of low-risk observation unit patients, the CE model had superior discrimination compared to the Diamond Forrester classification (IDI 0.403, p < 0.001).
Plasma levels of ACAT2-CE have strong potential to predict a patient’s likelihood of having CAD when considered in a clinical model but not when used alone. In turn, a clinical model containing ACAT2-CE could reduce the need for cardiac imaging after the exclusion of myocardial infarction.
PMCID: PMC3566778  PMID: 22687182
7.  Multicategory reclassification statistics for assessing improvements in diagnostic accuracy 
Biostatistics (Oxford, England)  2012;14(2):382-394.
In this paper, we extend the definitions of the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) in the context of multicategory classification. Both measures were proposed in Pencina and others (2008. Evaluating the added predictive ability of a new marker: from area under the receiver operating characteristic (ROC) curve to reclassification and beyond. Statistics in Medicine 27, 157–172) as numeric characterizations of accuracy improvement for binary diagnostic tests and were shown to have certain advantage over analyses based on ROC curves or other regression approaches. Estimation and inference procedures for the multiclass NRI and IDI are provided in this paper along with necessary asymptotic distributional results. Simulations are conducted to study the finite-sample properties of the proposed estimators. Two medical examples are considered to illustrate our methodology.
PMCID: PMC3695653  PMID: 23197381
Area under the ROC curve; Integrated discrimination improvement; Multicategory classification; Multinomial logistic regression; Net reclassification improvement
8.  Incremental Value of Biochemical and Echocardiographic Measures in Prediction of Ischemic Stroke: The Strong Heart Study 
Background and Purpose
American Indians suffer high rates of stroke. Improved risk stratification could enhance prevention, but the ability of biochemical and echocardiographic markers of preclinical disease to improve stroke prediction is not well defined.
We evaluated such markers as predictors of ischemic stroke in a community-based cohort of American Indians without prevalent cardiovascular or renal disease. Laboratory markers included C-reactive protein (CRP), fibrinogen, urine albumin-creatinine ratio (UACR), and glycohemoglobin (HbA1c), while echocardiographic parameters comprised left atrial (LA) diameter, left ventricular mass, mitral annular calcification (MAC), and mitral E/A ratio. Predictive performance was judged by indices of discrimination, reclassification and calibration.
After adjustment for standard risk factors, only HbA1c, albuminuria, and LA diameter were significantly associated with first ischemic stroke. Addition of HbA1c, though not UACR, to a basic clinical model significantly improved the C-statistic (0.714 vs. 0.695, p=0.044), whereas LA diameter modestly enhanced integrated discrimination improvement (IDI=0.90%, p=0.004), but not the C-statistic (0.701, p=0.528). When combined with HbA1c, LA diameter further increased IDI (1.81%, p<0.001), though not the C-statistic (0.716). No marker achieved significant net reclassification improvement (NRI).
In this cohort at high cardiometabolic risk, HbA1c emerged as the foremost predictor of ischemic stroke when added to traditional risk factors, affording substantially improved discrimination, with a more modest contribution for LA diameter. These findings bolster the role of HbA1c in cardiovascular risk assessment among persons with glycometabolic disorders, and provide impetus for further study of the incremental value of echocardiography in high-risk populations.
PMCID: PMC3288714  PMID: 22207511
Stroke; Echocardiography; Biomarkers
9.  Usefulness of N-terminal pro-brain natriuretic peptide and C-reactive protein to predict ICU mortality in unselected medical ICU patients: a prospective, observational study 
Critical Care  2011;15(1):R42.
The performance of N-terminal pro-brain natriuretic peptide (NT-proBNP) and C-reactive protein (CRP) to predict clinical outcomes in ICU patients is unimpressive. We aimed to assess the prognostic value of NT-proBNP, CRP or the combination of both in unselected medical ICU patients.
A total of 576 consecutive patients were screened for eligibility and followed up during the ICU stay. We collected each patient's baseline characteristics including the Acute Physiology and Chronic Health Evaluation II (APACHE-II) score, NT-proBNP and CRP levels. The primary outcome was ICU mortality. Potential predictors were analyzed for possible association with outcomes. We also evaluated the ability of NT-proBNP and CRP additive to APACHE-II score to predict ICU mortality by calculation of C-index, net reclassification improvement (NRI) and integrated discrimination improvement (IDI) indices.
Multiple regression revealed that CRP, NT-proBNP, APACHE-II score and fasting plasma glucose independently predicted ICU mortality (all P < 0.01). The C-index with respect to prediction of ICU mortality of APACHE II score (0.82 ± 0.02; P < 0.01) was greater than that of NT-proBNP (0.71 ± 0.03; P < 0.01) or CRP (0.65 ± 0.03; P < 0.01) (all P < 0.01). As compared with APACHE-II score (0.82 ± 0.02; P < 0.01), combination of CRP (0.83 ± 0.02; P < 0.01) or NT-proBNP (0.83 ± 0.02; P < 0.01) or both (0.84 ± 0.02; P < 0.01) with APACHE-II score did not significantly increase C-index for predicting ICU mortality (all P > 0.05). However, addition of NT-proBNP to APACHE-II score gave IDI of 6.6% (P = 0.003) and NRI of 16.6% (P = 0.007), addition of CRP to APACHE-II score provided IDI of 5.6% (P = 0.026) and NRI of 12.1% (P = 0.023), and addition of both markers to APACHE-II score yielded IDI of 7.5% (P = 0.002) and NRI of 17.9% (P = 0.002). In the cardiac subgroup (N = 213), NT-proBNP but not CRP independently predicted ICU mortality and addition of NT-proBNP to APACHE-II score obviously increased predictive ability (IDI = 10.2%, P = 0.018; NRI = 18.5%, P = 0.028). In the non-cardiac group (N = 363), CRP rather than NT-proBNP was an independent predictor of ICU mortality.
In unselected medical ICU patients, NT-proBNP and CRP can serve as independent predictors of ICU mortality and addition of NT-proBNP or CRP or both to APACHE-II score significantly improves the ability to predict ICU mortality. NT-proBNP appears to be useful for predicting ICU outcomes in cardiac patients.
PMCID: PMC3221971  PMID: 21272380
10.  An integrated biochemical prediction model of all-cause mortality in patients undergoing lower extremity bypass surgery for advanced peripheral artery disease 
Patients with advanced peripheral artery disease (PAD) have a high prevalence of cardiovascular (CV) risk factors and shortened life expectancy. However, CV risk factors poorly predict midterm (<5 years) mortality in this population. This study was designed to test the hypothesis that baseline biochemical parameters would add clinically meaningful predictive information in patients undergoing lower extremity bypass.
This was a prospective cohort study of subjects with clinically advanced PAD undergoing lower extremity bypass surgery. The Cox proportional hazard was used to assess the main outcome of all-cause mortality. A clinical model was constructed with known cardiovascular risk factors and the incremental value of the addition of clinical chemistry, lipid, and a panel of 11 inflammatory parameters were investigated using c-statistic, the integrated discrimination improvement (IDI) index and Akaike information criterion (AIC).
225 subjects were followed for a median 893 days; IQR 539–1315 days). In this study 50 (22.22%) subjects died during the follow-up period. By life table analysis (expressed as percent surviving ± standard error), survival at 1, 2, 3, 4, and 5 years respectively was 90.5 ± 1.9%, 83.4 ± 2.5%, 77.5 ± 3.1%, 71.0 ± 3.8%, and 65.3 ± 6.5%. Compared with survivors, decedents were older, diabetic, had extant CAD, and were more likely to present with CLI as their indication for bypass surgery, P<.05. After adjustment for the above, clinical chemistry and inflammatory parameters significant for all cause mortality were albumin, HR .43 (95% CI .26–.71); P=.001, estimated glomerular filtration rate (eGFR), HR .98 (95% CI .97–.99), P=.023, high sensitivity C-reactive protein (hsCRP), HR 3.21 (95% CI 1.21–8.55), P=.019, and soluble vascular cell adhesion molecule (sVCAM), HR 1.74 (1.04–2.91), P=.034. Of all inflammatory molecules investigated, hsCRP proved most robust and representative of the integrated inflammatory response. Albumin, eGFR, and hsCRP improved the c-statistic and IDI beyond that of the clinical model and produced a final c-statistic of .82.
A risk prediction model including traditional risk factors and parameters of inflammation, renal function and nutrition had excellent discriminatory ability in predicting all cause mortality in patients with clinically advanced PAD undergoing bypass surgery.
PMCID: PMC3413780  PMID: 22554422
11.  Prognostic Value of Biomarkers in Heart Failure: Application of Novel Methods in the Community 
Circulation. Heart failure  2009;2(5):393-400.
Mortality among patients with heart failure (HF) is high. Though individual biomarkers have been investigated to determine their value in mortality risk prediction, the role of a multimarker strategy requires further evaluation.
Methods and Results
Olmsted County residents presenting with HF from July 2004 to September 2007 were recruited to undergo biomarker measurement. We investigated whether addition of C-reactive protein (CRP), B-type natriuretic peptide (BNP), and troponin T (TnT) to a model including established risk indicators improved 1-year mortality risk prediction using the c statistic, integrated discrimination improvement (IDI), and net reclassification improvement (NRI). Among 593 participants, the mean age was 76.4 years and 48% were men. After 1 year follow-up, 122 (20.6%) participants had died. Patients with CRP (<11.8mg/L), BNP (<350pg/mL), and TnT (≤0.01ng/mL) below the median had low 1-year mortality (3.3%), while those with two or three biomarkers above the median had markedly increased mortality (30.8% and 35.5%, respectively). The addition of two or more biomarkers to the model offered greater improvement in 1-year mortality risk prediction than use of a single biomarker. The combination of CRP and BNP resulted in an increase in the c statistic from 0.757 to 0.810 (p<0.001), an IDI gain of 7.1% (p<0.001), and a NRI of 22.1% (p<0.001). Use of all three biomarkers offered no incremental gain (IDI gain 0.7% vs. CRP+BNP, p=0.065).
Biomarkers improved 1-year mortality risk prediction beyond established indicators. The use of a two-biomarker combination was superior to a single biomarker in risk prediction, though addition of a third biomarker conferred no added benefit.
PMCID: PMC2774116  PMID: 19808368
epidemiology; heart failure; prognosis; inflammation; community
12.  Adjustment of the GRACE score by growth differentiation factor 15 enables a more accurate appreciation of risk in non-ST-elevation acute coronary syndrome 
European Heart Journal  2011;33(9):1095-1104.
The aim of the study was to evaluate whether knowledge of the circulating concentration of growth differentiation factor 15 (GDF-15) adds predictive information to the Global Registry of Acute Coronary Events (GRACE) score, a validated scoring system for risk assessment in non-ST-elevation acute coronary syndrome (NSTE-ACS). We also evaluated whether GDF-15 adds predictive information to a model containing the GRACE score and N-terminal pro-B-type natriuretic peptide (NT-proBNP), a prognostic biomarker already in clinical use.
Methods and results
The GRACE score, GDF-15, and NT-proBNP levels were determined on admission in 1122 contemporary patients with NSTE-ACS. Six-month all-cause mortality or non-fatal myocardial infarction (MI) was the primary endpoint of the study. To obtain GDF-15- and NT-proBNP-adjusted 6-month estimated probabilities of death or non-fatal MI, statistical algorithms were developed in a derivation cohort (n = 754; n = 66 reached the primary endpoint) and applied to a validation cohort (n = 368; n = 33). Adjustment of the GRACE risk estimate by GDF-15 increased the area under the receiver-operating characteristic curve (AUC) from 0.79 to 0.85 (P < 0.001) in the validation cohort. Discrimination improvement was confirmed by an integrated discrimination improvement (IDI) of 0.055 (P = 0.005). A net 31% of the patients without events were reclassified into lower risk, and a net 27% of the patients with events were reclassified into higher risk, resulting in a total continuous net reclassification improvement [NRI(>0)] of 0.58 (P = 0.002). Addition of NT-proBNP to the GRACE score led to a similar improvement in discrimination and reclassification. Addition of GDF-15 to a model containing GRACE and NT-proBNP led to a further improvement in model performance [increase in AUC from 0.84 for GRACE plus NT-proBNP to 0.86 for GRACE plus NT-proBNP plus GDF-15, P = 0.010; IDI = 0.024, P = 0.063; NRI(>0) = 0.42, P = 0.022].
We show that a single measurement of GDF-15 on admission markedly enhances the predictive value of the GRACE score and provides moderate incremental information to a model including the GRACE score and NT-proBNP. Our study is the first to provide simple algorithms that can be used by the practicing clinician to more precisely estimate risk in individual patients based on the GRACE score and a single biomarker measurement on admission. The rigorous statistical approach taken in the present study may serve as a blueprint for future studies exploring the added value of biomarkers beyond clinical risk scores.
PMCID: PMC3888120  PMID: 22199121
GDF-15; NT-proBNP; GRACE score; Acute coronary syndrome; Risk stratification
13.  Isotopic Diversity Indices: How Sensitive to Food Web Structure? 
PLoS ONE  2013;8(12):e84198.
Recently revisited, the concept of niche ecology has lead to the formalisation of functional and trophic niches using stable isotope ratios. Isotopic diversity indices (IDI) derived from a set of measures assessing the dispersion/distribution of points in the δ-space were recently suggested and increasingly used in the literature. However, three main critics emerge from the use of these IDI: 1) they fail to account for the isotopic sources overlap, 2) some indices are highly sensitive to the number of species and/or the presence of rare species, and 3) the lack of standardization prevents any spatial and temporal comparisons. Using simulations we investigated the ability of six commonly used IDI to discriminate among different trophic food web structures, with a focus on the first two critics. We tested the sensitivity of the IDI to five food web structures along a gradient of sources overlap, varying from two distinct food chains with differentiated sources to two superimposed food chains sharing two sources. For each of the food web structure we varied the number of species (from 10 to 100 species) and the type of species feeding behaviour (i.e. random or selective feeding). Values of IDI were generally larger in food webs with distinct basal sources and tended to decrease as the superimposition of the food chains increased. This was more pronounced when species displayed food preferences in comparison to food webs where species fed randomly on any prey. The number of species composing the food web also had strong effects on the metrics, including those that were supposedly less sensitive to small sample size. In all cases, computing IDI on food webs with low numbers of species always increases the uncertainty of the metrics. A threshold of ∼20 species was detected above which several metrics can be safely used.
PMCID: PMC3877220  PMID: 24391910
14.  A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard 
The absence of a gold standard, i.e., a diagnostic reference standard having perfect sensitivity and specificity, is a common problem in clinical practice and in diagnostic research studies. There is a need for methods to estimate the incremental value of a new, imperfect test in this context.
We use a Bayesian approach to estimate the probability of the unknown disease status via a latent class model and extend two commonly-used measures of incremental value based on predictive values [difference in the area under the ROC curve (AUC) and integrated discrimination improvement (IDI)] to the context where no gold standard exists. The methods are illustrated using simulated data and applied to the problem of estimating the incremental value of a novel interferon-gamma release assay (IGRA) over the tuberculin skin test (TST) for latent tuberculosis (TB) screening. We also show how to estimate the incremental value of IGRAs when decisions are based on observed test results rather than predictive values.
We showed that the incremental value is greatest when both sensitivity and specificity of the new test are better and that conditional dependence between the tests reduces the incremental value. The incremental value of the IGRA depends on the sensitivity and specificity of the TST, as well as the prevalence of latent TB, and may thus vary in different populations.
Even in the absence of a gold standard, incremental value statistics may be estimated and can aid decisions about the practical value of a new diagnostic test.
PMCID: PMC4077291  PMID: 24886359
Area under the curve; Bayesian estimation; Incremental value; Informative priors; Integrated discrimination improvement; Imperfect diagnostic tests; Latent class models; Tuberculosis
15.  Assessing the performance of prediction models: a framework for some traditional and novel measures 
Epidemiology (Cambridge, Mass.)  2010;21(1):128-138.
The performance of prediction models can be assessed using a variety of different methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic (ROC) curve), and goodness-of-fit statistics for calibration.
Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision–analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.
We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n=544 for model development, n=273 for external validation).
We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for making clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.
PMCID: PMC3575184  PMID: 20010215
16.  Thirty-One Novel Biomarkers as Predictors for Clinically Incident Diabetes 
PLoS ONE  2010;5(4):e10100.
The prevalence of diabetes is increasing in all industrialized countries and its prevention has become a public health priority. However, the predictors of diabetes risk are insufficiently understood. We evaluated, whether 31 novel biomarkers could help to predict the risk of incident diabetes.
Methods and Findings
The biomarkers were evaluated primarily in the FINRISK97 cohort (n = 7,827; 417 cases of clinically incident diabetes during the follow-up). The findings were replicated in the Health 2000 cohort (n = 4,977; 179 cases of clinically incident diabetes during the follow-up). We used Cox proportional hazards models to calculate the relative risk of diabetes, after adjusting for the classic risk factors, separately for each biomarker. Next, we assessed the discriminatory ability of single biomarkers using receiver operating characteristic curves and C-statistics, integrated discrimination improvement (IDI) and net reclassification improvement (NRI). Finally, we derived a biomarker score in the FINRISK97 cohort and validated it in the Health 2000 cohort. A score consisting of adiponectin, apolipoprotein B, C-reactive protein and ferritin almost doubled the relative risk of diabetes in the validation cohort (HR per one standard deviation increase 1.88, p = 2.8 e-5). It also improved discrimination of the model (IDI = 0.0149, p<0.0001) and reclassification of diabetes risk (NRI = 11.8%, p = 0.006). Gender-specific analyses suggested that the best score differed between men and women. Among men, the best results were obtained with the score of four biomarkers: adiponectin, apolipoprotein B, ferritin and interleukin-1 receptor antagonist, which gave an NRI of 25.4% (p<0.0001). Among women, the best score included adiponectin, apolipoprotein B, C-reactive protein and insulin. It gave an NRI of 13.6% (p = 0.041).
We identified novel biomarkers that were associated with the risk of clinically incident diabetes over and above the classic risk factors. This gives new insights into the pathogenesis of diabetes and may help with targeting prevention and treatment.
PMCID: PMC2852424  PMID: 20396381
17.  The Use and Magnitude of Reclassification Measures for Individual Predictors of Global Cardiovascular Risk 
Annals of internal medicine  2009;150(11):795-802.
Models for risk prediction are widely used in clinical practice to risk stratify and assign treatment strategies. The contribution of new biomarkers has largely been based on the area under the receiver operating characteristic curve, but this measure can be insensitive to important changes in absolute risk. Methods based on risk stratification have recently been proposed to compare predictive models. These include the reclassification calibration statistic, the net reclassification improvement (NRI), and the integrated discrimination improvement (IDI). This work demonstrates the use of reclassification measures, and illustrates their performance for well-known cardiovascular risk predictors in a cohort of women. These measures are targeted at evaluating the potential of new models and markers to change risk strata and alter treatment decisions.
PMCID: PMC2782591  PMID: 19487714
18.  Critical elements of clinical follow-up after hospital discharge for heart failure: insights from the EVEREST trial 
European Journal of Heart Failure  2010;12(4):367-374.
Hospitalized heart failure (HF) patients are at high risk for death and readmission. We examined the incremental value of data obtained 1 week after HF hospital discharge in predicting mortality and readmission.
Methods and results
In the Efficacy of Vasopressin Antagonism in Heart Failure Outcome Study with tolvaptan, 1528 hospitalized patients (ejection fraction ≤40%) with a physical examination, laboratories, and health status [Kansas City Cardiomyopathy Questionnaire (KCCQ)] assessments 1 week after discharge were included. The ability to predict 1 year cardiovascular rehospitalization and mortality was assessed with Cox models, c-statistics, and the integrated discrimination improvement (IDI). Not using a beta-blocker, rales, pedal oedema, hyponatraemia, lower creatinine clearance, higher brain natriuretic peptide, and worse health status were independent risk factors for rehospitalization and death. The c-statistic for the base model (history and medications) was 0.657. The model improved with physical examination, laboratory, and KCCQ results, with IDI increases of 4.9, 7.0, and 3.2%, respectively (P < 0.001 each). The combination of all three offered the greatest incremental gain (c-statistic 0.749; IDI increase 10.8%).
Physical examination, laboratories, and KCCQ assessed 1 week after discharge offer important prognostic information, suggesting that all are critical components of outpatient evaluation after HF hospitalization.
PMCID: PMC3732083  PMID: 20197265
Heart failure; Prognosis; Health status; Hospitalization; Mortality; BNP
19.  CT findings for intracerebral hemorrhage have little incremental impact on post-stroke mortality prediction model performance 
Stroke outcome studies often combine cases of intracerebral hemorrhage (ICH) and ischemic stroke (IS). These studies of mixed stroke typically ignore computed tomography (CT) findings for ICH cases, though the impact of omitting these traditional predictors of ICH mortality is unknown. We investigated the incremental impact of ICH CT findings on mortality prediction model performance.
Cases of ICH and IS (2000–2003) were identified from the Brain Attack Surveillance in Corpus Christi (BASIC) project. Base models predicting 30-day mortality included demographics, stroke type, and clinical findings (National Institutes of Health Stroke Scale (NIHSS) +/− Glasgow coma scale (GCS)). The impact of adding CT data (volume, intraventricular hemorrhage, infratentorial location) was assessed with the area under the curve (AUC), unweighted sum of squared residuals (Ŝ), and integrated discrimination improvement (IDI). The model assessment was performed first for the mixed case of IS and ICH, and then repeated for ICH cases alone to determine whether any lack of improvement in model performance with CT data for mixed stroke type was due to IS cases naturally forming a larger proportion of the total sample than ICH.
A total of 1,256 cases were included (86% IS, 14% ICH). Thirty-day mortality was 16% overall (11% for IS; 43% for ICH). When both clinical scales (NIHSS and GCS) were included, none of the model performance measures showed improvement with the addition of CT findings whether considering IS and ICH together (ΔAUC: 0.002, 95% CI −0.01, 0.02; ΔŜ: −3.0, 95% CI −9.1, 2.6; IDI: 0.017, 95% CI −0.004, 0.05) or considering ICH cases alone (ΔAUC: 0.02, 95% CI −0.02, 0.08; Δ Ŝ: −2.0, 95% CI −9.7, 3.4); IDI 0.065, 95% CI −0.03, 0.21). If NIHSS was the only clinical scale included, there was still no improvement in AUC or Ŝ when CT findings were added for the sample with IS/ICH combined (ΔAUC: 0.005, 95%CI −0.01, 0.02; ΔŜ: −5.0, 95%CI −11.6, 1.0) or for ICH cases alone (ΔAUC: 0.05, 95% CI −0.002, 0.11; ΔŜ: −4.2, 95%CI −11.5, 2.3). However, IDI was improved when NIHSS was the only clinical scale for IS/ICH combined (IDI: 0.029, 95%CI 0.002, 0.065) and ICH alone (IDI: 0.12, 95%CI 0.005, 0.26).
Excluding ICH CT findings had only minimal impact on mortality prediction model performance whether examining ICH and IS together or ICH alone. These findings have important implications for the design of clinical studies involving ICH patients.
PMCID: PMC3466471  PMID: 22814203
epidemiology; cerebral infarction; prediction of outcome; intracerebral hemorrhage
20.  C-Reactive Protein and B-Type Natriuretic Peptide Yield Either a Non-Significant or a Modest Incremental Value to Traditional Risk Factors in Predicting Long-Term Overall Mortality in Older Adults 
PLoS ONE  2013;8(9):e75809.
New biomarkers may aid in preventive and end-of-life decisions in older adults if they enhance the prognostic ability of traditional risk factors. We investigated whether C-reactive protein (CRP) and/or B-type natriuretic peptide (BNP) improve the ability to predict overall mortality among the elderly of the Bambuí, Brazil Study of Aging when added to traditional risk factors.
From 1997 to 2007, 1,470 community-dwelling individuals (≥60 years) were followed-up. Death was ascertained by continuous verification of death certificates. We calculated hazard ratios per 1 standard deviation change (HR) of death for traditional risk factors only (old model), and traditional risk factors plus CRP and/or BNP (new models) and assessed calibration of the models. Subsequently, we compared c-statistic of each of the new models to the old one, and calculated integrated discriminative improvement (IDI) and net reclassification improvement (NRI).
544 (37.0%) participants died in a mean follow-up time of 9.0 years. CRP (HR 1.28, 95% CI 1.17-1.40), BNP (HR 1.31 95% CI 1.19-1.45), and CRP plus BNP (HR 1.26, 95% CI 1.15-1.38, and HR 1.29, 95% CI 1.16-1.42, respectively) were independent determinants of mortality. All models were well-calibrated. Discrimination was similar among the old (c-statistic 0.78 [0.78-0.81]) and new models (p=0.43 for CRP; p=0.57 for BNP; and p=0.31 for CRP plus BNP). Compared to the old model, CRP, BNP, and CRP plus BNP models led to an IDI of 0.009 (p<0.001), -0.005 (p<0.001) and -0.003 (p=0.84), and a NRI of 0.04 (p=0.24), 0.07 (p=0.08) and 0.06 (p=0.10), respectively.
Despite being independent predictors of long-term risk of death, compared to traditional risk factors CRP and/or BNP led to either a modest or non-significant improvement in the ability of predicting all-cause mortality in older adults.
PMCID: PMC3815403  PMID: 24244755
21.  Serum Uric Acid Predicts Declining of Circulating Proangiogenic Mononuclear Progenitor Cells in Chronic Heart Failure Patients 
Introduction: Serum uric acid (SUA) is considered a marker for natural progression of chronic heart failure (CHF) mediated cardiovascular remodelling. CHF associates with declining of circulating mononuclear progenitor cells (MPCs). The objective of this study was to evaluate the interrelationship between SUA concentrations and proangiogenic MPCs in ischemic CHF patients.
Methods: The study population was structured retrospectively after determining the coronary artery disease (CAD) by contrast-enhanced spiral computed tomography angiography in 126 subjects with symptomatic ischemic mild-to-severe CHF and 128 CAD subjects without CHF. Baseline biomarkers were measured in all patients. Cox proportional multivariate hazard ratio was calculated for predictors of MPCs declining in both CHF and non-CHF patient population predictors of MPCs declining in CHF subjects were examined in stepwise logistic regression. C-statistics, integrated discrimination indices (IDI) and net-reclassification improvement were utilized for prediction performance analyses.
Results: Cox proportional adjusted hazard ratio analyses for CD14+CD309+ and CD14+CD309+Tie2+ MPCs by SUA has shown that the higher quartiles (Q3 and Q4) of SUA compared to the lower quartiles (Q1 and Q2) are associated with increased risks of depletion of both CD14+CD309+ and CD14+CD309+Tie2+ MPCs. The addition of Q4 SUA to the ABC model improved the relative IDI by 13.8% for depletion of CD14+CD309+ MPCs and by 14.5% for depletion of CD14+CD309+Tie2+ MPCs.
Conclusion: Circulating levels of proangiogenic MPCs are declined progressively depending on the levels of SUA in the HF subjects with CHF. We suggest that even mild elevations of SUA might be used to predict of relative depletion of proangiogenic MPCs among chronic HF patients.
PMCID: PMC4195965  PMID: 25320662
Chronic Heart Failure; Serum Uric Acid; Circulating Mononuclear Progenitor Cells; Predictive Value
22.  Evaluation of the IDI-MRSA Assay for Detection of Methicillin-Resistant Staphylococcus aureus from Nasal and Rectal Specimens Pooled in a Selective Broth 
Journal of Clinical Microbiology  2006;44(4):1219-1223.
Rapid detection of methicillin-resistant Staphylococcus aureus (MRSA) by PCR can be performed directly from nasal specimens with the IDI-MRSA assay. To improve the efficiency of screening, we evaluated the performance of the IDI-MRSA assay for the detection of MRSA from pooled and unpooled specimens cultured in a selective broth. Of the 287 specimens evaluated, 71 were culture and PCR positive, 203 were culture and PCR negative, 3 were culture positive and PCR negative, 8 were culture negative and PCR positive, and 2 remained inhibited. A methicillin-susceptible Staphylococcus aureus isolate was recovered from five of the eight specimens with false-positive PCR results. Compared to the results of culture, the sensitivity, specificity, and negative and positive predictive values of the IDI-MRSA assay for detection of MRSA from broth were 96%, 96%, 90%, and 98%, respectively. Following implementation of the IDI-MRSA assay, PCR-positive broths were subcultured for evaluation of assay performance. Of the 298 IDI-MRSA assay-positive broths, the results for 103 could not be confirmed by culture. A methicillin-susceptible S. aureus (MSSA) isolate was recovered from 77 of these 103 broths. Repeat testing by the IDI-MRSA assay directly with the MSSA isolates confirmed the original positive PCR result. The positive predictive value of the IDI-MRSA assay fell from 90% during the evaluation phase to 65% postimplementation. The IDI-MRSA assay performed well for the detection of MRSA from a selective broth compared to the performance of the detection of MRSA from culture. However, because of the burden associated with implementation of infection control precautions, cultures remain essential in confirming positive IDI-MRSA results.
PMCID: PMC1448652  PMID: 16597841
23.  Plasma COOH-Terminal Proendothelin-1 
Diabetes Care  2012;35(11):2354-2358.
The aim of this study was to investigate the association between plasma COOH-terminal proendothelin-1 (CT-proET-1) and fatal cardiovascular events, all-cause mortality, and new-onset albuminuria in patients with type 2 diabetes.
A total of 1,225 patients with type 2 diabetes participated in this prospective observational study of two combined cohorts. Three clinical end points were studied: fatal cardiovascular events, all-cause mortality, and new-onset albuminuria. After a median follow-up of 3 or 10 years, Cox proportional hazard modeling was used to investigate the association between CT-proET-1 and the end points. Harrell C statistic, the Groennesby and Borgan test, the integrated discrimination improvement (IDI), and the net reclassification improvement (NRI) were used to evaluate whether CT-proET-1 is of additional value compared with classic cardiovascular and renal risk factors.
During follow-up, 364 (30%) patients died, 150 (42%) of whom died of cardiovascular disease; 182 (26.7%) of 688 patients with normoalbuminuria at baseline developed albuminuria. CT-proET-1 was associated with fatal cardiovascular events, all-cause mortality, and new-onset albuminuria with hazard ratios of 1.59 (95% CI 1.15–2.20), 1.41 (95% CI 1.14–1.74), and 1.48 (95% CI 1.10–2.01), respectively. Addition of CT-proET-1 to a model containing traditional risk factors leads only to improved prediction of fatal cardiovascular events. The IDI appeared significant for fatal cardiovascular events (0.82 [0.1–1.54]) and all-cause mortality (0.4 [0.05–0.92]), but not for new-onset albuminuria.
CT-proET-1 has additional value for the prediction of fatal cardiovascular events and new-onset albuminuria in patients with type 2 diabetes, compared with conventional risk factors, but not for all-cause mortality.
PMCID: PMC3476931  PMID: 22837372
24.  Temporal relationship and predictive value of urinary acute kidney injury biomarkers after pediatric cardiopulmonary bypass 
We investigated the temporal pattern and predictive value (alone and in combination) of four urinary biomarkers [neutrophil gelatinase-associated lipocalin (NGAL), interleukin-18 (IL-18), liver fatty-acid binding protein (L-FABP) and kidney injury molecule-1 (KIM-1)] for cardiac surgery-associated acute kidney injury (AKI).
Serum creatinine (SCr) is a delayed marker for AKI after cardiopulmonary bypass (CPB). Rapidly detectable AKI biomarkers could allow early intervention and improve outcomes.
Data from 220 pediatric patients were analyzed. Urine samples were obtained before and at intervals after CPB initiation. AKI was defined as a ≥50% increase in SCr from baseline within 48h after CPB. The temporal pattern of biomarker elevation was established and biomarker elevations were correlated with AKI severity and clinical outcomes. Biomarker predictive abilities were evaluated by AUC, net reclassification improvement (NRI) and integrated discrimination improvement (IDI).
AKI occurred in 27% of patients. Urine NGAL significantly increased in AKI patients at 2h after CPB initiation. IL-18 and L-FABP increased at 6h and KIM-1 increased at 12h. Biomarker elevations correlated with AKI severity and clinical outcomes, and improved AKI prediction above a clinical model. At 2h, addition of NGAL increased the AUC from 0.74 to 0.85 (p<0.0001). At 6h, NGAL, IL-18 and L-FABP each improved the AUC from 0.72 to 0.91, 0.84 and 0.77, respectively (all p<0.05). The added predictive ability of the biomarkers was supported by NRI and IDI. Biomarker combinations further improved AKI prediction.
Urine NGAL, IL-18, L-FABP, and KIM-1 are sequential predictive biomarkers for AKI and correlate with disease severity and clinical outcomes after pediatric CPB. These biomarkers, particularly in combination, may help establish the timing of injury and allow earlier intervention in AKI.
PMCID: PMC3220882  PMID: 22093507
Acute Kidney Injury; Cardiopulmonary Bypass; Ischemia; Biomarkers
25.  Prognostic Utility of Biomarkers in Predicting of One-Year Outcomes in Patients with Aortic Stenosis Treated with Transcatheter or Surgical Aortic Valve Implantation 
PLoS ONE  2012;7(12):e48851.
The aim of the work was to find biomarkers identifying patients at high risk of adverse clinical outcomes after TAVI and SAVR in addition to currently used predictive model (EuroSCORE).
There is limited data about the role of biomarkers in predicting prognosis, especially when TAVI is available.
The multi-biomarker sub-study included 42 consecutive high-risk patients (average age 82.0 years; logistic EuroSCORE 21.0%) allocated to TAVI transfemoral and transapical using the Edwards-Sapien valve (n = 29), or SAVR with the Edwards Perimount bioprosthesis (n = 13). Standardized endpoints were prospectively followed during the 12-month follow-up.
The clinical outcomes after both TAVI and SAVR were comparable. Malondialdehyde served as the best predictor of a combined endpoint at 1 year with AUC (ROC analysis) = 0.872 for TAVI group, resp. 0.765 (p<0.05) for both TAVI and SAVR groups. Increased levels of MDA, matrix metalloproteinase 2, tissue inhibitor of metalloproteinase (TIMP1), ferritin-reducing ability of plasma, homocysteine, cysteine and 8-hydroxy-2-deoxyguanosine were all predictors of the occurrence of combined safety endpoints at 30 days (AUC 0.750–0.948; p<0.05 for all). The addition of MDA to a currently used clinical model (EuroSCORE) significantly improved prediction of a combined safety endpoint at 30 days and a combined endpoint (0–365 days) by the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) (p<0.05).
Cystatin C, glutathione, cysteinylglycine, asymmetric dimethylarginine, nitrite/nitrate and MMP9 did not prove to be significant. Total of 14.3% died during 1-year follow-up.
We identified malondialdehyde, a marker of oxidative stress, as the most promising predictor of adverse outcomes during the 30-day and 1-year follow-up in high-risk patients with symptomatic, severe aortic stenosis treated with TAVI. The development of a clinical “TAVIscore” would be highly appreciated. Such dedicated scoring system would enable further testing of adjunctive value of various biomarkers.
PMCID: PMC3522688  PMID: 23272045

Results 1-25 (1138997)