# Related Articles

SUMMARY

For censored survival outcomes, it can be of great interest to evaluate the predictive power of individual markers or their functions. Compared with alternative evaluation approaches, the time-dependent ROC (receiver operating characteristics) based approaches rely on much weaker assumptions, can be more robust, and hence are preferred. In this article, we examine evaluation of markers’ predictive power using the time-dependent ROC curve and a concordance measure which can be viewed as a weighted area under the time-dependent AUC (area under the ROC curve) profile. This study significantly advances from existing time-dependent ROC studies by developing nonparametric estimators of the summary indexes and, more importantly, rigorously establishing their asymptotic properties. It reinforces the statistical foundation of the time-dependent ROC based evaluation approaches for censored survival outcomes. Numerical studies, including simulations and application to an HIV clinical trial, demonstrate the satisfactory finite-sample performance of the proposed approaches.

doi:10.1002/sim.5386

PMCID: PMC3743052
PMID: 22987578

time-dependent ROC; concordance measure; inverse-probability-of-censoring weighting; marker evaluation; survival outcomes

Summary

The predictiveness curve shows the population distribution of risk endowed by a marker or risk prediction model. It provides a means for assessing the model’s capacity for stratifying the population according to risk. Methods for making inference about the predictiveness curve have been developed using cross-sectional or cohort data. Here we consider inference based on case-control studies which are far more common in practice. We investigate the relationship between the ROC curve and the predictiveness curve. Insights about their relationship provide alternative ROC interpretations for the predictiveness curve and for a previously proposed summary index of it. Next the relationship motivates ROC based methods for estimating the predictiveness curve. An important advantage of these methods over previously proposed methods is that they are rank invariant. In addition they provide a way of combining information across populations that have similar ROC curves but varying prevalence of the outcome. We apply the methods to PSA, a marker for predicting risk of prostate cancer.

doi:10.1111/j.1541-0420.2009.01201.x

PMCID: PMC2794984
PMID: 19459841

biomarker; classification; predictiveness curve; risk prediction; ROC curve; total gain

The receiver operating characteristic (ROC) curve is often used to evaluate the performance of a biomarker measured on continuous scale to predict the disease status or a clinical condition. Motivated by the need for novel study designs with better estimation efficiency and reduced study cost, we consider a biased sampling scheme that consists of a SRC and a supplemental TDC. Using this approach, investigators can oversample or undersample subjects falling into certain regions of the biomarker measure, yielding improved precision for the estimation of the ROC curve with a fixed sample size. Test-result-dependent sampling will introduce bias in estimating the predictive accuracy of the biomarker if standard ROC estimation methods are used. In this article, we discuss three approaches for analyzing data of a test-result-dependent structure with a special focus on the empirical likelihood method. We establish asymptotic properties of the empirical likelihood estimators for covariate-specific ROC curves and covariate-independent ROC curves and give their corresponding variance estimators. Simulation studies show that the empirical likelihood method yields good properties and is more efficient than alternative methods. Recommendations on number of regions, cutoff points, and subject allocation is made based on the simulation results. The proposed methods are illustrated with a data example based on an ongoing lung cancer clinical trial.

doi:10.1093/biostatistics/kxs020

PMCID: PMC3577107
PMID: 22723502

Binormal model; Covariate-independent ROC curve; Covariate-specific ROC curve; Empirical likelihood method; Test-result-dependent sampling

Summary

This paper considers receiver operating characteristic (ROC) analysis for bivariate marker measurements. The research interest is to extend tools and rules from univariate marker to bivariate marker setting for evaluating predictive accuracy of markers using a tree-based classification rule. Using an and-or classifier, an ROC function together with a weighted ROC function (WROC) and their conjugate counterparts are proposed for examining the performance of bivariate markers. The proposed functions evaluate the performance of and-or classifiers among all possible combinations of marker values, and are ideal measures for understanding the predictability of biomarkers in target population. Specific features of ROC and WROC functions and other related statistics are discussed in comparison with those familiar properties for univariate marker. Nonparametric methods are developed for estimating ROC-related functions, (partial) area under curve and concordance probability. With emphasis on average performance of markers, the proposed procedures and inferential results are useful for evaluating marker predictability based on a single or bivariate marker (or test) measurements with different choices of markers, and for evaluating different and-or combinations in classifiers. The inferential results developed in this paper also extend to multivariate markers with a sequence of arbitrarily combined and-or classifier.

doi:10.1111/j.1541-0420.2012.01783.x

PMCID: PMC3530667
PMID: 23005264

Concordance probability; Prediction accuracy; Tree-based classification; U-statistics

Multiple biomarkers are frequently observed or collected for detecting or understanding a disease. The research interest of this paper is to extend tools of ROC analysis from univariate marker setting to multivariate marker setting for evaluating predictive accuracy of biomarkers using a tree-based classification rule. Using an arbitrarily combined and-or classifier, an ROC function together with a weighted ROC function (WROC) and their conjugate counterparts are introduced for examining the performance of multivariate markers. Specific features of the ROC and WROC functions and other related statistics are discussed in comparison with those familiar properties for univariate marker. Nonparametric methods are developed for estimating the ROC and WROC functions, and area under curve (AUC) and concordance probability. With emphasis on population average performance of markers, the proposed procedures and inferential results are useful for evaluating marker predictability based on multivariate marker measurements with different choices of markers, and for evaluating different and-or combinations in classifiers.

doi:10.1007/s10985-012-9233-5

PMCID: PMC3633731
PMID: 23054242

Concordance probability; Multiple markers; Prediction accuracy; U-statistics

Cho, Ki-Ho | Kim, Kyoo-Pil | Woo, Byung-Cheol | Kim, Young-Jee | Park, Joo-Young | Cho, Seung-Yeon | Park, Seong-Uk | Jung, Woo-Sang | Park, Jung-Mi | Moon, Sang-Kwan
Blood stasis syndrome (BSS) in traditional Asian medicine has been considered to correlate with the extent of atherosclerosis, which can be estimated using the cardioankle vascular index (CAVI). Here, the diagnostic utility of CAVI in predicting BSS was examined. The BSS scores and CAVI were measured in 140 stroke patients and evaluated with respect to stroke risk factors. Receiver operating characteristic (ROC) curve analysis was used to determine the diagnostic accuracy of CAVI for the diagnosis of BSS. The BSS scores correlated significantly with CAVI, age, and systolic blood pressure (SBP). Multiple logistic regression analysis showed that CAVI was a significant associate factor for BSS (OR 1.55, P = 0.032) after adjusting for the age and SBP. The ROC curve showed that CAVI and age provided moderate diagnostic accuracy for BSS (area under the ROC curve (AUC) for CAVI, 0.703, P < 0.001; AUC for age, 0.692, P = 0.001). The AUC of the “CAVI+Age,” which was calculated by combining CAVI with age, showed better accuracy (0.759, P < 0.0001) than those of CAVI or age. The present study suggests that the CAVI combined with age can clinically serve as an objective tool to diagnose BSS in stroke patients.

doi:10.1155/2012/696983

PMCID: PMC3369533
PMID: 22701508

We present a unified approach to nonparametric comparisons of receiver operating characteristic (ROC) curves for a paired design with clustered data. Treating empirical ROC curves as stochastic processes, their asymptotic joint distribution is derived in the presence of both between-marker and within-subject correlations. A Monte Carlo method is developed to approximate their joint distribution without involving nonparametric density estimation. The developed theory is applied to derive new inferential procedures for comparing weighted areas under the ROC curves, confidence bands for the difference function of ROC curves, confidence intervals for the set of specificities at which one diagnostic test is more sensitive than the other, and multiple comparison procedures for comparing more than two diagnostic markers. Our methods demonstrate satisfactory small-sample performance in simulations. We illustrate our methods using clustered data from a glaucoma study and repeated-measurement data from a startle response study.

PMCID: PMC2832229
PMID: 20209021

Area under the receiver operating characteristic curve; Clustered data; Confidence band; Intersection-union tests; Longitudinal data; Multiple comparison; Paired design; Partial area under the receiver operating characteristic curve; Quantile process; Repeated measurement

A major biomedical goal associated with evaluating a candidate biomarker or developing a predictive model score for event-time outcomes is to accurately distinguish between incident cases from the controls surviving beyond t throughout the entire study period. Extensions of standard binary classification measures like time-dependent sensitivity, specificity, and receiver operating characteristic (ROC) curves have been developed in this context (Heagerty, P. J., and others, 2000. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics
56, 337–344). We propose a direct, non-parametric method to estimate the time-dependent Area under the curve (AUC) which we refer to as the weighted mean rank (WMR) estimator. The proposed estimator performs well relative to the semi-parametric AUC curve estimator of Heagerty and Zheng (2005. Survival model predictive accuracy and ROC curves. Biometrics
61, 92–105). We establish the asymptotic properties of the proposed estimator and show that the accuracy of markers can be compared very simply using the difference in the WMR statistics. Estimators of pointwise standard errors are provided.

doi:10.1093/biostatistics/kxs021

PMCID: PMC3520498
PMID: 22734044

AUC curve; Survival analysis; Time-dependent ROC

The receiver operating characteristic (ROC) curve, the positive predictive value (PPV) curve and the negative predictive value (NPV) curve are three measures of performance for a continuous diagnostic biomarker. The ROC, PPV and NPV curves are often estimated empirically to avoid assumptions about the distributional form of the biomarkers. Recently, there has been a push to incorporate group sequential methods into the design of diagnostic biomarker studies. A thorough understanding of the asymptotic properties of the sequential empirical ROC, PPV and NPV curves will provide more flexibility when designing group sequential diagnostic biomarker studies. In this paper we derive asymptotic theory for the sequential empirical ROC, PPV and NPV curves under case-control sampling using sequential empirical process theory. We show that the sequential empirical ROC, PPV and NPV curves converge to the sum of independent Kiefer processes and show how these results can be used to derive asymptotic results for summaries of the sequential empirical ROC, PPV and NPV curves.

doi:10.1214/11-AOS937

PMCID: PMC3771874
PMID: 24039313

Group Sequential Methods; Empirical Process Theory; Diagnostic Testing

Background

In order to improve our understanding of the molecular pathways that mediate tumor proliferation and angiogenesis, and to evaluate the biological response to anti-angiogenic therapy, we analyzed the changes in the protein profile of glioblastoma in response to treatment with recombinant human Platelet Factor 4-DLR mutated protein (PF4-DLR), an inhibitor of angiogenesis.

Methodology/Principal Findings

U87-derived experimental glioblastomas were grown in the brain of xenografted nude mice, treated with PF4-DLR, and processed for proteomic analysis. More than fifty proteins were differentially expressed in response to PF4-DLR treatment. Among them, integrin-linked kinase 1 (ILK1) signaling pathway was first down-regulated but then up-regulated after treatment for prolonged period. The activity of PF4-DLR can be increased by simultaneously treating mice orthotopically implanted with glioblastomas, with ILK1-specific siRNA. As ILK1 is related to malignant progression and a poor prognosis in various types of tumors, we measured ILK1 expression in human glioblatomas, astrocytomas and oligodendrogliomas, and found that it varied widely; however, a high level of ILK1 expression was correlated to a poor prognosis.

Conclusions/Significance

Our results suggest that identifying the molecular pathways induced by anti-angiogenic therapies may help the development of combinaatorial treatment strategies that increase the therapeutic efficacy of angiogenesis inhibitors by association with specific agents that disrupt signaling in tumor cells.

doi:10.1371/journal.pone.0013710

PMCID: PMC2966411
PMID: 21060779

Rationale and Objectives

A basic assumption for a meaningful diagnostic decision variable is that there is a monotone relationship between the decision variable and the likelihood of disease. This relationship, however, generally does not hold for the binormal model. As a result, ROC-curve estimation based on the binormal model produces improper ROC curves that are not concave over the entire domain and cross the chance line. Although in practice the “improperness” is typically not noticeable, there are situations where the improperness is evident. Presently, standard statistical software does not provide diagnostics for assessing the magnitude of the improperness.

Materials and Methods

We show how the mean-to-sigma ratio can be a useful, easy-to-understand and easy-to-use measure for assessing the magnitude of the improperness of a binormal ROC curve by showing how it is related to the chance-line crossing. We suggest an improperness criterion based on the mean-to-sigma ratio.

Results

Using a real-data example we illustrate how the mean-to-sigma ratio can be used to assess the improperness of binormal ROC curves, compare the binormal method with an alternative proper method, and describe uncertainty in a fitted ROC curve with respect to improperness.

Conclusions

By providing a quantitative and easily computable improperness measure, the mean-to-sigma ratio provides an easy way to identify improper binormal ROC curves and facilitates comparison of analysis strategies according to improperness categories in simulation and real-data studies.

doi:10.1016/j.acra.2010.09.002

PMCID: PMC3053019
PMID: 21232682

receiver operating characteristic (ROC) curve; diagnostic radiology; mean-to-sigma ratio; binormal model; proper ROC model

Summary

Receiver operating characteristic (ROC) curves can be used to assess the accuracy of tests measured on ordinal or continuous scales. The most commonly used measure for the overall diagnostic accuracy of diagnostic tests is the area under the ROC curve (AUC). A gold standard test on the true disease status is required to estimate the AUC. However, a gold standard test may sometimes be too expensive or infeasible. Therefore, in many medical research studies, the true disease status of the subjects may remain unknown. Under the normality assumption on test results from each disease group of subjects, using the expectation-maximization (EM) algorithm in conjunction with a bootstrap method, we propose a maximum likelihood based procedure for construction of confidence intervals for the difference in paired areas under ROC curves in the absence of a gold standard test. Simulation results show that the proposed interval estimation procedure yields satisfactory coverage probabilities and interval lengths. The proposed method is illustrated with two examples.

doi:10.1002/sim.3661

PMCID: PMC2812057
PMID: 19691022

Area under the ROC curve; EM algorithm; bootstrap method; gold standard test; maximum likelihood estimation

In this paper, we extend the definitions of the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) in the context of multicategory classification. Both measures were proposed in Pencina and others (2008. Evaluating the added predictive ability of a new marker: from area under the receiver operating characteristic (ROC) curve to reclassification and beyond. Statistics in Medicine
27, 157–172) as numeric characterizations of accuracy improvement for binary diagnostic tests and were shown to have certain advantage over analyses based on ROC curves or other regression approaches. Estimation and inference procedures for the multiclass NRI and IDI are provided in this paper along with necessary asymptotic distributional results. Simulations are conducted to study the finite-sample properties of the proposed estimators. Two medical examples are considered to illustrate our methodology.

doi:10.1093/biostatistics/kxs047

PMCID: PMC3695653
PMID: 23197381

Area under the ROC curve; Integrated discrimination improvement; Multicategory classification; Multinomial logistic regression; Net reclassification improvement

ROC analysis occupies an increasingly important role in technology assessment.[1] ROC curves allow one to compare a set of ordinal estimates over the entire range of estimates. Sources of such estimates may include subjective probabilities, mathematical prediction models and empiric prediction models (like the APGAR score). The area under the ROC curve measures the ability of the estimation method to discriminate between two states (usually disease and non-disease). This paper discusses how one constructs ROC curves, what the area under the curve means, and how and why one compares two ROC curves. The computer program (ROC ANALYZER) allows easy performance of these analyses on MS-DOS compatible machines.

PMCID: PMC2245768

Background

Statistical evaluation of medical imaging tests used for diagnostic and prognostic purposes often employ receiver operating characteristic (ROC) curves. Two methods for ROC analysis are popular. The ordinal regression method is the standard approach used when evaluating tests with ordinal values. The direct ROC modeling method is a more recently developed approach that has been motivated by applications to tests with continuous values, such as biomarkers.

Objective

In this paper, we compare the methods in terms of model formulations, interpretations of estimated parameters, the ranges of scientific questions that can be addressed with them, their computational algorithms and the efficiencies with which they use data.

Results

We show that a strong relationship exists between the methods by demonstrating that they fit the same models when only a single test is evaluated. The ordinal regression models are typically alternative parameterizations of the direct ROC models and vice-versa. The direct method has two major advantages over the ordinal regression method: (i) estimated parameters relate directly to ROC curves. This facilitates interpretations of covariate effects on ROC performance; and (ii) comparisons between tests can be done directly in this framework. Comparisons can be made while accommodating covariate effects and comparisons can be made even between tests that have values on different scales, such as between a continuous biomarker test and an ordinal valued imaging test. The ordinal regression method provides slightly more precise parameter estimates from data in our simulated data models.

Conclusion

While the ordinal regression method is slightly more efficient, the direct ROC modeling method has important advantages in regards to interpretation and it offers a framework to address a broader range of scientific questions including the facility to compare tests.

doi:10.1177/0272989X09357477

PMCID: PMC2905510
PMID: 20147599

comparisons; covariates; diagnostic test; markers; ordinal regression; percentile values

Receiver operating characteristic (ROC) curve, plotting true positive rates against false positive rates as threshold varies, is an important tool for evaluating biomarkers in diagnostic medicine studies. By definition, ROC curve is monotone increasing from 0 to 1 and is invariant to any monotone transformation of test results. And it is often a curve with certain level of smoothness when test results from the diseased and non-diseased subjects follow continuous distributions. Most existing ROC curve estimation methods do not guarantee all of these properties. One of the exceptions is Du and Tang (2009) which applies certain monotone spline regression procedure to empirical ROC estimates. However, their method does not consider the inherent correlations between empirical ROC estimates. This makes the derivation of the asymptotic properties very difficult. In this paper we propose a penalized weighted least square estimation method, which incorporates the covariance between empirical ROC estimates as a weight matrix. The resulting estimator satisfies all the aforementioned properties, and we show that it is also consistent. Then a resampling approach is used to extend our method for comparisons of two or more diagnostic tests. Our simulations show a significantly improved performance over the existing method, especially for steep ROC curves. We then apply the proposed method to a cancer diagnostic study that compares several newly developed diagnostic biomarkers to a traditional one.

doi:10.1016/j.jspi.2010.05.026

PMCID: PMC3358774
PMID: 22639484

ROC curve; Smoothing spline; Bootstrap

Rationale and Objectives

To examine the effects of the number of categories in the rating scale used in an observer experiment on the results of ROC analysis by a simulation study.

Materials and Methods

We have previously evaluated the effects of computer-aided diagnosis (CAD) on radiologists’ characterization of malignant and benign breast masses in serial mammograms. The evaluation of the likelihood of malignancy was performed on a quasi-continuous (0-100 points) confidence-rating scale. In this study, we simulated the use of discrete confidence-rating scales with fewer number of categories and analyzed the results with receiver operating characteristic (ROC) methodology. The observers’ estimates of the likelihood of malignancy were also mapped to BI-RADS assessments with 5 and 7 categories and ROC analysis was performed. The area under the ROC curve and the partial area index obtained from ROC analysis of the different confidence-rating scales were compared.

Results

The fitted ROC curves and the performance indices do not change significantly when the confidence-rating scales were varied from 6 to 101 points if the estimated operating points obtained directly from the data are distributed relatively evenly over the entire range of true-positive fraction (TPF) and false-positive fraction (FPF). The mapping of the likelihood of malignancy observer data to the 7-category BI-RADS assessment scale allowed reliable ROC analysis, whereas mapping to the 5-category BI-RADS scale could cause erratic ROC curve fitting because of the lack of operating points in the mid-range or failure in ROC curve fitting because of data degeneration for some observers.

Conclusion

ROC analysis of discrete confidence rating scales with few but relatively evenly distributed data points over the entire FPF and TPF range is comparable to that of a quasi-continuous rating scale. However, ROC analysis of discrete confidence rating scales with few and unevenly distributed data points may cause unreliable estimations.

doi:10.1016/j.acra.2006.09.048

PMCID: PMC2976672
PMID: 17178364

Computer-Aided Diagnosis; Continuous and Discrete Confidence Rating Scales; ROC Observer Study; Classification; Mammography

The receiver operating characteristic (ROC) curve displays the capacity of a marker or diagnostic test to discriminate between two groups of subjects, cases versus controls. We present a comprehensive suite of Stata commands for performing ROC analysis. Non-parametric, semiparametric and parametric estimators are calculated. Comparisons between curves are based on the area or partial area under the ROC curve. Alternatively pointwise comparisons between ROC curves or inverse ROC curves can be made. Options to adjust these analyses for covariates, and to perform ROC regression are described in a companion article. We use a unified framework by representing the ROC curve as the distribution of the marker in cases after standardizing it to the control reference distribution.

PMCID: PMC2774909
PMID: 20161343

A new reporter system has been developed for quantifying gene expression in the yeast Saccharomyces cerevisiae. The system relies on two different reporter genes, Renilla and firefly luciferase, to evaluate regulated gene expression. The gene encoding Renilla luciferase is fused to a constitutive promoter (PGK1 or SPT15) and integrated into the yeast genome at the CAN1 locus as a control for normalizing the assay. The firefly luciferase gene is fused to the test promoter and integrated into the yeast genome at the ura3 or leu2 locus. The dual luciferase assay is performed by sequentially measuring the firefly and Renilla luciferase activities of the same sample, with the results expressed as the ratio of firefly to Renilla luciferase activity (Fluc/Rluc). The yeast dual luciferase reporter (DLR) was characterized and shown to be very efficient, requiring approximately 1 minute to complete each assay, and has proven to yield data that accurately and reproducibly reflect promoter activity. A series of integrating plasmids were generated that contain either the firefly or Renilla luciferase gene preceded by a multicloning region in two different orientations and the three reading frames to make possible the generation of translational fusions. Additionally, each set of plasmids contains either the URA3 or LEU2 marker for genetic selection in yeast. A series of S288C-based yeast strains, including a two-hybrid strain, were developed to facilitate the use of the yeast DLR assay. This assay can be readily adapted to a high-throughput platform for studies requiring numerous measurements.

doi:10.1128/EC.4.9.1539-1549.2005

PMCID: PMC1214206
PMID: 16151247

Background

The aim of this study was to assess the predictive capacity of body fat percentage (%BF) estimated by equations using body mass index (BMI) and waist circumference (WC) to identify hypertension and estimate measures of association between high %BF and hypertension in adults.

Methods

This is a cross-sectional population-based study conducted with 1,720 adults (20–59 years) from Florianopolis, southern Brazil. The area under the ROC curve, sensitivity, specificity, predictive values, and likelihood ratios of cutoffs for %BF were calculated. The association between %BF and hypertension was analyzed using Poisson regression, estimating the unadjusted and adjusted prevalence ratios and 95% CI.

Results

The %BF equations showed good discriminatory power for hypertension (area under the ROC curve > 0.50). Considering the entire sample, the cutoffs for %BF with better properties for screening hypertension were identified in the equation with BMI for men (%BF = 20.4) and with WC for women (%BF = 34.1). Adults with high %BF had a higher prevalence of hypertension.

Conclusions

The use of simple anthropometric measurements allowed identifying the %BF, diagnosing obesity, and screening people at risk of hypertension in order to refer them for more careful diagnostic evaluation.

doi:10.1186/1475-2891-11-112

PMCID: PMC3545980
PMID: 23244669

Anthropometry; Risk factors; Obesity; Hypertension; Adults

Summary

Covariate-specific ROC curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this paper, we propose a semiparametric estimation of the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates’ effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease research.

doi:10.1111/j.1541-0420.2011.01562.x

PMCID: PMC3596883
PMID: 21361890

Alzheimer's disease; covariate-specific ROC curve; ignorable missingness; verification bias; weighted estimating equations

Summary

To develop more targeted intervention strategies, an important research goal is to identify markers predictive of clinical events. A crucial step towards this goal is to characterize the clinical performance of a marker for predicting different types of events. In this manuscript, we present statistical methods for evaluating the performance of a prognostic marker in predicting multiple competing events. To capture the potential time-varying predictive performance of the marker and incorporate competing risks, we define time- and cause-specific accuracy summaries by stratifying cases based on causes of failure. Such definition would allow one to evaluate the predictive accuracy of a marker for each type of event and compare its predictiveness across event types. Extending the nonparametric crude cause-specific ROC curve estimators by Saha and Heagerty (2010), we develop inference procedures for a range of cause-specific accuracy summaries. To estimate the accuracy measures and assess how covariates may affect the accuracy of a marker under the competing risk setting, we consider two forms of semiparametric models through the cause-specific hazard framework. These approaches enable a flexible modeling of the relationships between the marker and failure times for each cause, while efficiently accommodating additional covariates. We investigate the asymptotic property of the proposed accuracy estimators and demonstrate the finite sample performance of these estimators through simulation studies. The proposed procedures are illustrated with data from a prostate cancer prognostic study.

doi:10.1111/j.1541-0420.2011.01671.x

PMCID: PMC3694786
PMID: 22150576

Biomarker evaluation; Cause-specific Hazard; Competing risk; Negative predictive value; Positive predictive value; Receiver Operating Characteristics Curve (ROC curve); Survival analysis

Receiver Operating Characteristic (ROC) analysis is a common tool for
assessing the performance of various classifications. It gained much popularity in medical and other fields including biological markers and, diagnostic test. This is particularly due to the fact that in real-world problems
misclassification costs are not known, and thus, ROC curve and related utility
functions such as F-measure can be more meaningful performance measures.
F-measure combines recall and precision into a global measure. In this paper, we propose a novel method through regularized F-measure maximization.
The proposed method assigns different costs to positive and negative samples and does simultaneous feature selection and prediction with L1 penalty. This method is useful especially when data set is highly unbalanced, or the
labels for negative (positive) samples are missing. Our experiments with the
benchmark, methylation, and high dimensional microarray data show that the performance of proposed algorithm is better or equivalent compared with the other popular classifiers in limited experiments.

doi:10.1155/2009/617946

PMCID: PMC2674633
PMID: 19421401

High-throughput studies have been extensively conducted in the research of complex human diseases. As a representative example, consider gene-expression studies where thousands of genes are profiled at the same time. An important objective of such studies is to rank the diagnostic accuracy of biomarkers (e.g. gene expressions) for predicting outcome variables while properly adjusting for confounding effects from low-dimensional clinical risk factors and environmental exposures. Existing approaches are often fully based on parametric or semi-parametric models and target evaluating estimation significance as opposed to diagnostic accuracy. Receiver operating characteristic (ROC) approaches can be employed to tackle this problem. However, existing ROC ranking methods focus on biomarkers only and ignore effects of confounders. In this article, we propose a model-based approach which ranks the diagnostic accuracy of biomarkers using ROC measures with a proper adjustment of confounding effects. To this end, three different methods for constructing the underlying regression models are investigated. Simulation study shows that the proposed methods can accurately identify biomarkers with additional diagnostic power beyond confounders. Analysis of two cancer gene-expression studies demonstrates that adjusting for confounders can lead to substantially different rankings of genes.

doi:10.1093/bib/bbs008

PMCID: PMC3431720
PMID: 22396461

ranking biomarkers; ROC; confounders; high-throughput data

Comparison of two samples can sometimes be conducted on the basis of analysis of ROC curves. A variety of methods of point estimation and confidence intervals for ROC curves have been proposed and studied well. We develop smoothed empirical likelihood based confidence intervals for ROC curves when the samples are censored and generated from semiparametric models. The resulting empirical log-likelihood function is shown to be asymptotically chi-squared. Simulation studies illustrate that the proposed empirical likelihood confidence interval is advantageous over the normal approximation based confidence interval. A real data set is analyzed using the proposed method.

doi:10.1111/j.1467-9469.2007.00580.x

PMCID: PMC2839253
PMID: 20300485

Estimating equation; confidence interval; coverage; Kaplan-Meier estimation; empirical likelihood ratio; empirical likelihood function