Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Kidney Int. Author manuscript; available in PMC 2013 November 1.
Published in final edited form as:
Published online 2013 January 23. doi:  10.1038/ki.2012.458
PMCID: PMC3642209

Urine neutrophil gelatinase-associated lipocalin levels do not improve risk prediction of progressive chronic kidney disease

Kathleen D Liu, MD, PhD,1 Wei Yang, PhD,2 Amanda H Anderson, PhD,2 Harold I Feldman, MD,2 Sevag Demirjian, MD,3 Takayuki Hamano, MD, PhD,2 Jiang He, MD, PhD,4 James Lash, MD,5 Eva Lustigova,4 Sylvia E Rosas, MD,2 Michael S Simonson, PhD,6 Kaixiang Tao, PhD,2 and Chi-yuan Hsu, MD1,7, on behalf of the Chronic Renal Insufficiency Cohort (CRIC) study investigators


Novel biomarkers may improve our ability to predict which patients with chronic kidney disease (CKD) are at higher risk for progressive loss of renal function. Here we assessed the performance of urine neutrophil gelatinase-associated lipocalin (NGAL) for outcome prediction in a diverse cohort of 3386 patients with CKD in the CRIC study. In this cohort, the baseline mean estimated glomerular filtration rate (eGFR) was 42.4 ml/min/1.73m2; the median 24-hour urine protein was 0.2 gm/day; and the median urine NGAL concentration was 17.2 ng/mL. Over an average follow-up of 3.2 years, there were 689 cases in which the eGFR was decreased by half or incident end-stage renal disease developed. Even after accounting for eGFR, proteinuria and other known CKD progression risk factors, urine NGAL remained a significant independent risk factor (Cox model hazard ratio 1.70 highest to lowest quartile). The association between baseline urine NGAL levels and risk of CKD progression was strongest in the first two years of biomarker measurement. Within this time frame, adding urine NGAL to a model which included eGFR, proteinuria and other CKD progression risk factors led to net reclassification improvement of 24.7%; but the C-statistic remained nearly identical. Thus, while urine NGAL was an independent risk factor of progression among patients with established CKD of diverse etiology, it did not substantially improve prediction of outcome events.


There is great interest currently in defining novel biomarkers that will improve our ability to predict which patients with chronic kidney disease (CKD) are at higher risk for progressive loss of renal function, adding to currently available risk factors such as amount of total proteinuria and glomerular filtration rate (GFR).1, 2 Better risk stratification may potentially improve clinical outcomes by facilitating the application or intensification of evidence-based therapies among higher-risk patients.

One such promising novel biomarker is urine neutrophil gelatinase-associated lipocalin (NGAL).3 A recent comprehensive review identified urine NGAL as the most promising novel biomarker among those being evaluated as predictors of CKD progression.2 NGAL was originally identified in animal models by microarray analysis to be one of the earliest induced genes and proteins in the kidney after ischemic or nephrotoxic injury.4 It is a ubiquitous lipocalin iron-carrying protein, highly expressed in the tubular epithelium and released from tubular epithelial cells following damage.2 Although initially studied in the context of acute kidney injury, urine NGAL levels are also abnormally elevated (albeit at a much lower level) in a large number of individuals with chronic kidney disease.

A number of cross-sectional studies have shown that urine NGAL levels correlate with level of GFR or severity of underlying renal parenchyma injury.5-10 Several small studies in selected populations have provided conflicting reports about whether urine NGAL is an independent risk factor of more rapid loss of renal function after controlling for established CKD progression risk factors (such as proteinuria and blood pressure level).11-18

We undertook the present study to assess the performance of urine NGAL as a novel independent risk factor of CKD progression in a large prospective diverse cohort of individuals with CKD.


Of the 3939 enrolled Chronic Renal Insufficiency Cohort (CRIC) study participants, 229 did not have a specimen available for NGAL testing and 3386 had valid NGAL measurements. The characteristics of the 3386 study participants are shown in Table 1. Urine NGAL distribution was highly skewed with a median of 17.2 ng/mL and interquartile range (IQR) 8.1 to 39.2 ng/mL; 5% of the highest measurements were between 178.9 and 3069.6 ng/mL. The 24-hour urine protein distribution was also highly skewed with a median of 0.2 gm/day, and interquartile range of 0.1 to 1.0 gm/day; 5% of the highest measurements were between 5.3-30.1 gm/day. Mean estimated glomerular filtration rate (eGFR) was 42.4 ± 13.6 (SD) ml/min/1.73m2. Mean serum creatinine was 1.9 ± 0.7 mg/dL, blood urea nitrogen 30 ± 14 mg/dL, total cholesterol 183 ± 46 mg/dL, hemoglobin 12.6 ± 1.8 gm/dL, parathyroid hormone 76 ± 71 ng/L, and C-reactive protein 5.7 ± 10.0 mg/L.

Table 1
Baseline patient characteristics: overall and by quintiles of urine NGAL concentration

We divided the participants into quintiles of urine NGAL concentrations as shown in Table 1 (≤6.9, >6.9 to ≤12.9, >12.9 to ≤ 22.6, >22.6 to ≤49.5, >49.5 ng/mL). Patients with higher urine NGAL concentrations were more likely to have lower eGFR and have higher levels of proteinuria. They were also more likely to be female, non-white, have diabetes mellitus and higher blood pressure.

Over a mean follow-up period of 3.2 years, there were 689 cases of halving of eGFR or incident end-stage renal disease (ESRD)(incidence rate 6.37 per 100 person-years). In unadjusted analysis, there was a strong graded relationship between urine NGAL concentration and risk of CKD progression (Table 2). Compared with those with urine NGAL concentration ≤6.9 ng/mL, CRIC participants with urine NGAL concentration >49.5 were nearly 10 times more likely to experience ESRD or halving of eGFR (hazard ratio [HR] 9.34; 95% confidence interval [CI] 6.86-12.72; p <0.001). This relationship was similar after adjustment for demographic attributes (Table 2). The association between urine NGAL concentration and ESRD or halving of eGFR was substantially attenuated when eGFR and 24-urine protein were taken into account (HR 1.58 for urine NGAL concentration >49.5 vs. ≤6.9ng/mL; 95% CI 1.09-2.29; p=0.01). Additional adjustment for other risk factors for CKD progression gave similar results (HR 1.70 for urine NGAL concentration >49.5 vs. ≤6.9ng/mL; 95% CI 1.16-2.48; p=0.006)(Table 2).

Table 2
Association between quintiles of urine NGAL concentration and risk of progressive CKD (halving of eGFR or ESRD)

Results were consistent when urine NGAL was analyzed as a linear term with a HR of 1.11 per one standard deviation increase in log urine NGAL in the fully adjusted model (HR 1.11; 95% CI 1.01-1.21; p = 0.02). Compared with urine protein and eGFR the effect size of urine NGAL is much more modest (Figure 1).

Figure 1Figure 1Figure 1
A. Multivariable adjusted association between risk of progressive CKD and amount of 24-hour urine protein; log (HR) is log of the adjusted hazard ratio.

Similar results were seen when the exposure was changed to urine NGAL/urine creatinine ratio or when the outcome was defined using ESRD alone in sensitivity analyses (results not shown).

There were no statistically significant interactions between urine NGAL and proteinuria (p=0.27) or eGFR (p=0.42); nor with diabetes status (p= 0.78), age (p=0.13), sex (p=0.32) or race (p=0.18).

The C-statistic for the final model was 0.847. Removal of urine NGAL from the final model did not change the C-statistic to the third decimal place (0.847).

In exploratory analysis, we relaxed the assumption of a constant hazard ratio over time by modeling urine NGAL and renal progression within each of the four time intervals (<2, 2-<3, 3-<4 and >4 years). We noted that the effect of urine NGAL was stronger in the first two years after measurement of urine NGAL vs. more than 2 years of follow-up (Figure 2). The HR per SD of urine NGAL for outcomes occurring within 2 years of biomarker measurement was 1.27 (95% CI 1.14-1.42). The HRs were 0.97 (95% CI 0.83-1.12), 1.00 (95% CI 0.84-1.20) and 0.98 (95% CI 0.78-1.22) for the later three time intervals respectively.

Figure 2
Multivariable adjusted hazard ratio (HR)(per increase in log urine NGAL concentration) overall and by duration of follow-up.

The C-statistic for the model limited to two years of follow-up was 0.880. Removal of urine NGAL from the final model lowered the C-statistic minimally to 0.879.

To assess reclassification in arena of potentially most promising clinical use which would be prediction of outcome in the first two years following biomarker ascertainment, we examined category free as well as category-based Net Reclassification Improvement (NRI) within a two year horizon. We a priori defined clinically meaningful differences in risks of 0-5%, 5-10% and >10% event rate per year. The category free NRI for events was 4.4% (95% CI -5.9% to 16.8%) and for non-events was 20.3% (95% CI 6.5% to 26.0%). The overall category free NRI was thus 24.7% (95% CI 0.4% to 38.5%). The three-category NRI for events was -0.3% (95% CI -2.8% to 2.9%), and for non-events was 0.4% (95% CI -0.4% to 1.1%). The overall three-category NRI was thus 0.1% (95% CI -2.7% to 3.5%). In the entire cohort, predicted probability of CKD progression was similar in models with and without urine NGAL. Table 3 summarizes the different metrics used to assess incremental improvement in risk reclassification.

Table 3
Summary of measures of risk reclassification


In this large and diverse cohort of individuals with CKD, urine NGAL level was elevated in a substantial fraction of the study population. Urine NGAL levels correlated with severity of CKD at baseline as patients with lower eGFR and greater 24-hour proteinuria had higher urine NGAL levels.

Baseline urine NGAL levels correlated strongly with risk of CKD progression, defined as halving of estimated GFR or development of ESRD. The strength of this association was substantially attenuated after adjustment for a priori defined “traditional” risk factors, but urine NGAL remained an independent risk factor for CKD progression (with those in the highest quintile at 70% increased risk compared with those in the lowest quintile).

Our results are consistent with some but not all of the prior literature. In a study of 63 patients with type 1 diabetes mellitus (DM) and mean GFR of 87 ml/min/1.73m2, Nielsen et al showed that elevated NGAL was predictive of more rapid decline in 51Cr-EDTA measured GFR; but it was no longer an independent risk factor after adjusting for known promoters of progression.15 In a study of 65 patients with IgA nephropathy, Peters found that urine NGAL was a risk factor of ESRD risk in univariable analysis but not after adjusting for other factors such as serum creatinine.14 Nielson showed in another cohort of 78 type 1 DM patients that elevated urine NGAL was not related to decline in GFR during a 4-year follow-up; it was associated with the development of ESRD, but not after adjustment (albeit confidence intervals were wide).16 Nauta found in 606 kidney transplant recipients that urine NGAL did not predict graft failure after albuminuria was accounted for.18

Viau et al examined 87 subjects with polycystic kidney disease who had GFR 33±20 ml/min/1.73m2 and reported that urine NGAL levels were higher in patients who progressed to ESRD. No attempts were made to adjust for other known risk factors for CKD progression.11 In contrast, Parikh et al17 found that that urine NGAL levels did not correlate with changes in total kidney volume or kidney function in 209 patients with polycystic kidney disease and preserved GFR (mean 89 ml/min/1.73m2). In a study by Wu et al of 36 patients with drug induced chronic tubulointerstitial nephritis (GFR 37±20 ml/min/1.73m2), urine NGAL was predictive of renal function decline and the only risk factor with a p-value <0.05 in their multivariable models.12 Bolignano et al reported that among 96 patients with CKD of various etiologies (GFR 42±19 ml/min/1.73m2), urine NGAL (but not proteinuria) was an independent risk factor for CKD progression.13

We noted that the association between baseline urine NGAL levels and risk of CKD progression was strongest within the first two years of the biomarker measurement. Even within this time frame though, urine NGAL did not substantially improve risk classification.

The major strengths of this study include the large sample size, the detailed and rigorous phenotyping of study participants, the rigorous statistical analysis (encompassing both assessment of prediction and risk reclassification) and the diversity of the study population (in terms of both race-ethnicity as well as disease etiology).

Limitations of this study include the fact that NGAL measurements were made on urine samples which were not rapidly processed after voiding under conditions fully controlled by the research team. To the extent that suboptimal handling may have resulted in random error from protein degradation or other processes, this would bias our results towards the null. The CRIC study is composed of persons who volunteered for research and so our results may not be generalizable to all CKD patients. By design, some underlying causes of CKD, such as polycystic kidney disease, were not represented. Exact underlying kidney disease etiologies were not ascertained systematically or with certainty in CRIC through means such as kidney biopsies. Urine NGAL concentration was ascertained only at one point in time. Our subgroups and the 2-year time horizon were defined post-hoc and hence those results should be regarded as hypothesis generating findings which may be due to chance. With regard to possible explanations for the latter finding, we speculate that perhaps as an injury marker, urine NGAL levels may fluctuate over time due to (subtle) acute renal insults such as excessive lowering of blood pressure, for example. The impact of these transitory insults may be most evident in the short term.

To summarize, we determined that urine NGAL levels correlated with severity of CKD and were independently predictive of CKD progression, particularly in the first two years following measurement. However, this novel marker only very modestly improved prediction of outcome events. Thus it would be premature to consider introducing urine NGAL into clinical practice. Further studies, are needed to refine our understanding, in particular, ones in which urine collection and handling conditions were optimized.


Study population

The Chronic Renal Insufficiency Cohort (CRIC) study is a multi-center observational cohort which enrolled patients from seven clinical centers (consisting of 13 enrolling sites) located throughout the United States. Patients with reduced glomerular filtration rate (in the range of stages 2-4 CKD) between the ages of 21 and 74 were eligible for study participation. Those with polycystic kidney disease, multiple myeloma, or glomerulonephritis on active immunosuppression were excluded, as were those who had undergone kidney transplantation. Enrollment started June 2003 and ended August 2008. A total of 3939 participants were enrolled. The CRIC basic study design and baseline characteristics have been published.19, 20


The CRIC protocol called for collection of a 24-hour urine specimen at baseline. Enrollees were allowed to start this timed collection at home up to a week prior to their in-person visit. They were instructed to keep the urine refrigerated or on ice prior to bringing it to the clinical center. Samples were rejected and re-collection attempted if total urine volumes were below 500 cc or collection times below 22 hours or more than 26 hours. After proper mixing, samples from the 24-hr urine were then transferred into airtight 10 mL cryovials (filled to 9 mL). The urine NGAL measurements for this analysis were made on urine aliquots stored at 4°C and then shipped overnight within 72 hours to the CRIC central lab with cold gel packs placed in the bottom of the mailing container.

Upon receipt at the central lab, the urine samples were further aliquoted into four 2mL aliquots and frozen at −80°C. The urine for the current studies therefore have undergone one freeze-thaw cycle. Urine NGAL concentration was measured using a two-step assay using Chemiluminescent Microparticle Immunoassay (CMIA) technology on an Abbott ARCHITECT i2000SR (Abbott Laboratories, Abbott Park, IL)(total imprecision was 3.8%).

All but 12 of the urine creatinines at baseline were measured on a BioTek Plate Reader ELX 808 using a Jaffe reaction with a colorimetric end point and reagents from Sigma (mean CV was 5.0%). A Roche Module P analyzer was used to quantify urine total protein using a turbidimetric reaction with benzothnium chloride (CV 5.2%).


The primary outcome was progressive CKD, defined as a composite endpoint of incident end-stage renal disease (ESRD) or halving of estimated glomerular filtration rate (eGFR)(from baseline) using the Modification of Diet in Renal Disease (MDRD) study equation after calibration of serum creatinine.21 ESRD was defined as receipt of chronic dialysis or kidney transplant. Ascertainment of ESRD in the CRIC study has been supplemented by cross-linkage with U.S. Renal Data System. Time to eGFR halving was imputed assuming a linear decline in kidney function between in-person annual visit measures. Follow-up for this analysis was through June 2009.


Time-to-event analysis was conducted using Cox proportional hazards models. Recognizing that ESRD or halving of eGFR represent different degrees of loss of renal function depending on baseline renal function,22 we adjusted for baseline eGFR in all analyses.

We also adjusted for 24-hr urine proteinuria and other potential confounders, based on the literature of known risk factors for CKD progression. For the latter, we a prior selected demographics (age, sex, race/ethnicity), presence or absence of diabetes mellitus status (defined by fasting blood glucose ≥126 mg/dl or random blood glucose ≥200 mg/dL or patient self-report of use of insulin or oral diabetes medication), systolic blood pressure (SBP), body mass index (BMI), use of an angiotensin-converting enzyme inhibitor (ACE-I) or angiotensin receptor blocker (ARB) at baseline, history of cardiovascular disease (CVD), and education attainment (as a proxy for socio-economic status).22, 23 (We also adjusted for CRIC clinical centers in final analyses).

We explored the non-linear relationship between urine NGAL and renal progression using splines24 and discovered that by expressing urine NGAL on a natural logarithmic scale, a linear term of urine NGAL is sufficient to describe the relationship with CKD progression. For proteinuria and eGFR, the two strongest predictors for CKD progression,25 we relaxed the linearity assumptions for their relations with progression by using quadratic splines (with one knot at the median) of eGFR and natural log transformed proteinuria. We calculated the C-statistic to measure the overall model fit.26

To explore whether there was effect modification, we entered multiplicative interaction terms into our statistical models for urine NGAL and proteinuria, eGFR, diabetes status, age, sex, or race.

To explore whether and how the association between urine NGAL and progression changed over time, we refit the Cox proportional hazard model assuming different hazard ratios for urine NGAL in the following four time intervals: <2, 2 to <3, 3 to <4 and >4 years. We found that the effect of NGAL was strongest over the first two years after measurement. Thus, improved model predictability due to NGAL was quantified using Net Reclassification Improvement (NRI) over a two year horizon. We calculated both category-free NRI and NRI based on three categories (a priori defined risks of <5%, 5-10% and >10% annual event rate)27 for the risk of renal progression.


Funding for the CRIC Study was obtained under a cooperative agreement from National Institute of Diabetes and Digestive and Kidney Diseases (U01DK060990, U01DK060984, U01DK061022, U01DK061021, U01DK061028, U01DK60980, U01DK060963, and U01DK060902). In addition, this work was supported in part by: K01DK92353, K24DK02651, University of Pennsylvania CTRC CTSA UL1 RR-024134, Johns Hopkins University UL1 RR-025005, University of Maryland GCRC M01 RR-16500, Case Western Reserve University Clinical and Translational Science Collaborative (University Hospitals of Cleveland, Cleveland Clinic Foundation, and MetroHealth) UL1 RR-024989, University of Michigan GCRC grant number M01 RR-000042 CTSA grant number UL1 RR-024986, University of Illinois at Chicago CTSA UL1RR029879, Tulane Clinical and Translational Research, Education, and Commercialization Project (CTRECP), Kaiser NIH/NCRR UCSF-CTSI UL1 RR-024131. Drs. Liu, Feldman and Hsu are also supported by U01DK85649 as members of the CKD Biomarker Consortium and Dr. Hsu is additionally supported by K24 DK92291. We would like to thank Ted Mifflin and Steve Masters at Penn Central Lab for technical assistance with the urine NGAL assays.


DISCLOSURE All the authors declared no competing interests.


1. Kronenberg F. Emerging risk factors and markers of chronic kidney disease progression. Nature reviews Nephrology. 2009;5:677–689. [PubMed]
2. Fassett RG, Venuthurupalli SK, Gobe GC, et al. Biomarkers in chronic kidney disease: a review. Kidney Int. 2011;80:806–821. [PubMed]
3. Bolignano D, Donato V, Coppolino G, et al. Neutrophil gelatinase-associated lipocalin (NGAL) as a marker of kidney damage. Am J Kidney Dis. 2008;52:595–605. [PubMed]
4. Nickolas TL, Barasch J, Devarajan P. Biomarkers in acute and chronic kidney disease. Curr Opin Nephrol Hypertens. 2008;17:127–132. [PubMed]
5. Brunner HI, Mueller M, Rutherford C, et al. Urinary neutrophil gelatinase-associated lipocalin as a biomarker of nephritis in childhood-onset systemic lupus erythematosus. Arthritis Rheum. 2006;54:2577–2584. [PubMed]
6. Ding H, He Y, Li K, et al. Urinary neutrophil gelatinase-associated lipocalin (NGAL) is an early biomarker for renal tubulointerstitial injury in IgA nephropathy. Clin Immunol. 2007;123:227–234. [PubMed]
7. Bolignano D, Coppolino G, Campo S, et al. Neutrophil gelatinase-associated lipocalin in patients with autosomal-dominant polycystic kidney disease. Am J Nephrol. 2007;27:373–378. [PubMed]
8. Bolignano D, Coppolino G, Campo S, et al. Urinary neutrophil gelatinase-associated lipocalin (NGAL) is associated with severity of renal disease in proteinuric patients. Nephrol Dial Transplant. 2008;23:414–416. [PubMed]
9. Suzuki M, Wiers KM, Klein-Gitelman MS, et al. Neutrophil gelatinase-associated lipocalin as a biomarker of disease activity in pediatric lupus nephritis. Pediatr Nephrl. 2008;23:403–412. [PubMed]
10. Malyszko J, Malyszko JS, Bachorzewska-Gajewska H, et al. Neutrophil gelatinase-associated lipocalin is a new and sensitive marker of kidney function in chronic kidney disease patients and renal allograft recipients. Transplant Proc. 2009;41:158–161. [PubMed]
11. Viau A, El Karoui K, Laouari D, et al. Lipocalin 2 is essential for chronic kidney disease progression in mice and humans. J Clin Invest. 2010;120:4065–4076. [PMC free article] [PubMed]
12. Wu Y, Su T, Yang L, et al. Urinary neutrophil gelatinase-associated lipocalin: A potential biomarker for predicting rapid progression of drug-induced chronic tubulointerstitial nephritis. Am J Med Sci. 2010;339:537–542. [PubMed]
13. Bolignano D, Lacquaniti A, Coppolino G, et al. Neutrophil gelatinase-associated lipocalin (NGAL) and progression of chronic kidney disease. Clin J Am Soc Nephrol. 2009;4:337–344. [PubMed]
14. Peters HP, Waanders F, Meijer E, et al. High urinary excretion of kidney injury molecule-1 is an independent predictor of end-stage renal disease in patients with IgA nephropathy. Nephrol Dial Transplant. 2011;26:3581–3588. [PubMed]
15. Nielsen SE, Andersen S, Zdunek D, et al. Tubular markers do not predict the decline in glomerular filtration rate in type 1 diabetic patients with overt nephropathy. Kidney Int. 2011;79:1113–1118. [PubMed]
16. Nielsen SE, Hansen HP, Jensen BR, et al. Urinary neutrophil gelatinase-associated lipocalin and progression of diabetic nephropathy in type 1 diabetic patients in a four-year follow-up study. Nephron Clinical practice. 2011;118:c130–135. [PubMed]
17. Parikh CR, Dahl NK, Chapman AB, et al. Evaluation of urine biomarkers of kidney injury in polycystic kidney disease. Kidney Int. 2012;81:784–790. [PMC free article] [PubMed]
18. Nauta FL, Bakker SJ, van Oeveren W, et al. Albuminuria, proteinuria, and novel urine biomarkers as predictors of long-term allograft outcomes in kidney transplant recipients. Am J Kidney Dis. 2011;57:733–743. [PubMed]
19. Feldman HI, Appel LJ, Chertow GM, et al. The Chronic Renal Insufficiency Cohort (CRIC) Study: Design and Methods. J Am Soc Nephrol. 2003;14:S148–153. [PubMed]
20. Lash JP, Go AS, Appel LJ, et al. Chronic Renal Insufficiency Cohort (CRIC) Study: baseline characteristics and associations with kidney function. Clin J Am Soc Nephrol. 2009;4:1302–1311. [PubMed]
21. Levey AS, Coresh J, Greene T, et al. Expressing the Modification of Diet in Renal Disease Study equation for estimating glomerular filtration rate with standardized serum creatinine values. Clin Chem. 2007;53:766–772. [PubMed]
22. Hunsicker LG, Adler S, Caggiula A, et al. Predictors of the progression of renal disease in the Modification of Diet in Renal Disease Study. Kidney Int. 1997;51:1908–1919. [PubMed]
23. Hsu CY, Iribarren C, McCulloch CE, et al. Risk factors for end-stage renal disease: 25. year follow-up. Arch Intern Med. 2009;169:342–350. [PMC free article] [PubMed]
24. Greenland S. Dose-response and trend analysis in epidemiology: alternatives to categorical analysis. Epidemiology (Cambridge, Mass. 1995;6:356–365. [PubMed]
25. Gansevoort RT, Matsushita K, van der Velde M, et al. Lower estimated GFR and higher albuminuria are associated with adverse kidney outcomes. A collaborative meta-analysis of general and high-risk population cohorts. Kidney Int. 2011;80:93–104. [PMC free article] [PubMed]
26. Heagerty PJ, Zheng Y. Survival model predictive accuracy and ROC curves. Biometrics. 2005;61:92–105. [PubMed]
27. Pencina MJ, D’Agostino RB, Sr., Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11–21. [PMC free article] [PubMed]