Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Am Coll Cardiol. Author manuscript; available in PMC Feb 22, 2012.
Published in final edited form as:
PMCID: PMC3061352
Improvement in Mortality Risk Prediction Following Percutaneous Coronary Intervention Through Addition of a “Compassionate Use” Variable to the National Cardiovascular Data Registry CathPCI® Dataset: A Study from the Massachusetts Angioplasty Registry
Frederic S. Resnic, MD, MSc, FACC,a Sharon-Lise T. Normand, PhD, FACC,bc Thomas C. Piemonte, MD, FACC,d Samuel J. Shubrooks, MD, FACC,e Katya Zelevinsky, BA,b Ann Lovett, RN, MA,b and Kalon K.L. Ho, MD, MSc, FACCe
a Brigham and Women’s Hospital and Harvard Medical School, Boston, MA
b Massachusetts Data Analysis Center, Department of Health Care Policy, Harvard Medical School, Boston, MA
c Department of Biostatistics, Harvard School of Public Health, Boston, MA
d Lahey Clinic, Burlington MA and Tufts School of Medicine, Boston, MA
e Beth Israel Deaconess Medical Center and Harvard Medical School, Boston MA
Correspondence: Frederic S. Resnic, MD MSc, Brigham and Women’s Hospital, 75 Francis Street, Boston, MA 02115. Office: (857) 307-1992; Fax: 857-307-1955; fresnic/at/
This study investigated the impact of adding novel elements to models predicting in-hospital mortality following percutaneous coronary interventions (PCIs).
Massachusetts (MA) mandated public reporting of hospital-specific PCI mortality in 2003. In 2006, a physician advisory group recommended adding to the prediction models three attributes not collected by the National Cardiovascular Data Registry instrument. These “compassionate use” (CU) features included coma on presentation, active hemodynamic support during PCI, and cardiopulmonary resuscitation at PCI initiation.
From October 2005 through September 2007, PCI was performed during 29,784 admissions in MA non-federal hospitals. Of these, 5,588 involved patients with ST segment elevation myocardial infarction or cardiogenic shock. Cases with CU criteria identified were adjudicated by trained physician reviewers. Regression models with and without the CU composite variable (presence of any of the 3 features) were compared using areas under the receiver operator characteristic curves (AUC).
Unadjusted mortality in this high-risk subset was 5.7%. Among these admissions, 96 (1.7%) had at least one CU feature, with 69.8% mortality. The adjusted odds ratio for in-hospital death for CU PCIs (vs. no CU criteria) was 27.3 (95% CI 14.5–47.6). Discrimination of the model improved after including CU, with AUC increasing from 0.87 to 0.90 (p<0.01), while goodness of fit was preserved.
A small proportion of patients at extreme risk for post-PCI mortality can be identified using pre-procedural factors not routinely collected, but that heighten predictive accuracy. Such improvements in model performance may result in greater confidence in reporting of risk-adjusted PCI outcomes.
Keywords: Percutaneous coronary intervention (PCI), predictive models, Hierarchical risk prediction models, ACC-NCDR CathPCI
Beginning in 2003, Massachusetts (MA) followed New York, becoming the second state in the nation to require public release of risk-adjusted mortality statistics following percutaneous coronary interventional (PCI) procedures. The program was established to monitor the quality of invasive cardiac procedures performed within the state and to provide rigorous statistical assessments of any differences in the risk-adjusted mortality among hospitals and individual PCI operators. The Massachusetts program utilizes Bayesian hierarchical logistic regression models to evaluate the quality of cardiac care as assessed by the in-hospital, risk-adjusted, all-cause mortality following PCI(1). The hierarchical model methodology has been shown to be more specific in the detection of performance outliers than non-hierarchical methods(2,3).
The public release of risk-adjusted mortality reports is controversial(48) and can have significant impact on the hospitals and operators through adverse publicity associated with being identified as a negative outlier in quality performance(8). From the outset of the PCI reporting effort, the Massachusetts program actively engaged interventional cardiologists in the case review and adjudication process for all patients with high risk clinical features. In addition, clinicians were directly involved in the process of developing the risk-prediction models, based on the covariates available in the National Cardiovascular Data Registry (NCDR®) CathPCI Registry® (version 3.04) in an effort to establish the maximum confidence by the clinical community in the risk adjustment methodology. Based on physician input, Massachusetts divided the population of patients undergoing PCI into two groups: those presenting with ST segment elevation myocardial infarction (STEMI) or cardiogenic shock (the shock or STEMI, or “SOS” group), and all other, typically less acute PCI admissions (non-SOS group).
While the NCDR CathPCI Registry dataset is broadly used as a quality monitoring tool throughout the United States, it had not been originally intended for nor designed to be applied to mandatory public reporting as utilized in Massachusetts. Recognizing limitations in the breadth of the variables collected in the existing CathPCI Registry dataset, interventional cardiologists in Massachusetts became concerned that the risk adjustment models were limited in their ability to account fully for the clinical acuity of the most critically ill patients. There was a specific concern that clinicians could become increasingly risk-averse due to fears that the risk model methodology would not adequately adjust for the highest-risk patient populations being treated. The concerns centered on the possibility that uncaptured (and thus unmodeled) covariates, such as impaired neurologic status on presentation, might be independently predictive of in-hospital mortality. Exclusion of such potentially strong, independent predictors of mortality that could vary in prevalence across hospitals would reduce the clinical credibility of the models and might lead to inaccurate prediction of post-PCI mortality.
The concerns raised by Massachusetts clinicians was based, in part, on the observation that there had been a gradual decline in the proportion of patients treated with PCI for cardiogenic shock in New York in the years since that state began publishing reports of clinician-specific risk adjusted outcomes(4). Supporting this view, several surveys of New York State PCI operators and cardiac surgeons indicated an increasing unwillingness to treat the most severely ill patients after the introduction of public reporting of mortality outcomes; despite validated risk adjustment models(7). Moreover, it had been observed that New York interventionalists and cardiac surgeons were unique among geographic locales in providing lower rates of revascularization for patients included in a national cardiogenic shock registry(9), despite mounting evidence of the clinical benefits of early revascularization of most patients presenting in cardiogenic shock(10). In 2008, New York State began excluding patients presenting in cardiogenic shock from the publicly released risk adjusted mortaility reports. In light of these findings, clinical advisors to the Massachusetts public reporting program recommended inclusion of additional high-risk clinical factors not routinely collected in the CathPCI Registry dataset in an effort to improve the performance of the risk adjustment models as well as bolster clinician acceptance of the risk adjustment process.
We sought to examine the feasibility of collecting a set of novel factors likely predictive of post-PCI demise and their impact in the Massachusetts risk adjusted models of in-hospital mortality.
Patient and Hospital Populations
We studied all patients aged 18 years or older presenting with cardiogenic shock or ST segment elevation myocardial infarction and who underwent PCI at any of the Massachusetts, non-federal, acute care hospitals between October 1, 2005 through September 30, 2007. Because it was possible for patients to undergo more than one PCI during a single admission, we selected the first PCI as the unit of observation in accordance with NCDR risk adjustment methods and to reflect the clinical status of the patient on presentation. Patients were assigned to the hospital where the initial PCI was performed, even if transferred elsewhere for subsequent procedures.
Data Sources
We used clinical and billing data submitted to the Massachusetts Data Analysis Center (Mass-DAC), the data coordinating center located in Harvard Medical School under contract with the Department of Public Health of the Commonwealth of Massachusetts to collect, clean, and analyze the PCI data. Three distinct data sources were used to assemble information. Detailed clinical information obtained during the hospital admission was collected prospectively by trained data managers using variables defined in the NCDR CathPCI Registry dataset(11) and submitted electronically every 3 months. Massachusetts hospitals were also mandated by regulation to supplement the NCDR data with additional elements incorporating patient identification information and details regarding the identity of hospitals from which and to which patients may have been transferred. To ensure completeness of the admissions submitted to Mass-DAC, inpatient discharge billing data available from the Division of Health Care Finance and Policy in the Massachusetts Department of Public Health were linked to the Mass-DAC data using patient and hospital identifiers. The billing data included patient demographic information, as well as diagnoses and procedures performed during the hospitalization. Finally, while data managers report patient vital status at discharge, Mass-DAC confirmed this information by linking to the Massachusetts Registry of Vital Records and Statistics.
Compassionate Use
Beginning October 1, 2005, Mass-DAC added three unique covariates to those available in the NCDR CathPCI dataset and collected this information electronically at regular intervals. The new covariates were termed “compassionate use” (CU) criteria and were intended to identify extremely high risk clinical scenarios not currently characterized by the existing NCDR data instrument. The CU criteria included: coma on presentation for emergent PCI, use of a percutaneous ventricular assist device or extracorporeal bypass, and cardiopulmonary resuscitation (CPR) at the initiation of the procedure. Of note, both coma on presentation and CPR at initiation of procedure required that the case also be coded as an emergent or salvage procedure, while a planned or emergent ventricular assist device-supported procedure would also qualify as CU. Table 1 summarizes the definitions of the additional compassionate use covariates.
Table 1
Table 1
Definition of Compassionate Use Criteria
All CU cases were independently reviewed, adjudicated, and confirmed by physician reviewers trained in the definitions of items in the NCDR data instrument (e.g., cardiogenic shock, PCI status as emergent or salvage) as well as the CU criteria. All reviewers underwent training in research on human subjects and were approved by the Harvard Medical School’s Institutional Review Board. The CU case adjudication process scrutinized every case coded as CU using supplemental clinical documentation forwarded to Mass-DAC. This documentation included emergency department and emergency medical services records (when appropriate), cardiac catheterization laboratory reports, and clinical documentation for the inpatient hospital course. If a physician reviewer was uncertain about a particular adjudication, the case was discussed with the other members of the review committee, and a decision was reached by consensus. An appeal process was established so that medical centers could challenge the adjudication of any particular case which was originally ruled to not meet the CU criteria. All appeals required additional clinical documentation explicitly justifying the use of CU designation. Appeal adjudications were made by board certified interventional cardiologists with extensive experience in the adjudication process.
Primary Outcome
The primary patient outcome was mortality, regardless of cause, during the hospitalization (or outpatient procedure, if applicable) in which the PCI was performed. Secondary outcomes included survival at 30 days and 12 months as assessed by linked data analysis with Massachusetts death index, and assessment of impact of inclusion of compassionate use adjustment on hospital quality classification.
Statistical Analysis
Univariate analyses were performed using Chi-squared tests or Fisher Exact tests were implemented using SAS 9 (SAS Institute, Cary, NC)(12). Hierarchical logistic-normal regression models that included random hospital-specific intercepts were used to account for within-hospital clustering and estimated using the WinBugs application (13). The CU covariate was operationalized as a binary composite variable that assumed a value of 1 if any of the specified conditions was observed to be true; otherwise it assumed a value of 0. In-hospital mortality prediction models were developed using hierarchical regression models, including and excluding the compassionate use covariate.
The discriminating ability of the model was assessed by the area under the receiver operator characteristic (AUC) curve. We calculated the difference in the AUC between a model with and without the CU variables using a paired test(14). Because we were also interested in how the addition of the variables would impact risk prediction, we undertook a reclassification analysis to assess the proportion of patients whose predicted mortality changed by at least one quintile of risk(15). This was accomplished by calculating risk quintile for each patient using both the predictive model excluding and including the compassionate risk covariate. The Net Reclassification Improvement (NRI) measure that reflects the overall gain in prediction due to the inclusion of the compassionate use covariate was also calculated (16).
Between October 1, 2005 and September 30, 2007, at least one PCI occurred in a total of 29,784 admissions in Massachusetts licensed (non-Federal) hospitals. Among these admissions, 5,588 patients (18.8%) underwent a PCI after presenting with shock or ST segment elevation myocardial infarction (SOS cases). Within the SOS cohort of patients, a total of 96 patients (1.7% of SOS cases) were adjudicated as having qualified for compassionate use.
Compassionate use admissions were significantly more likely to have presented in cardiogenic shock (Table 2: 65.6% vs. 8.2%, p<0.001), more likely to have received an intra-aortic balloon pump (49.0% vs. 12.5%, p<0.001), and more likely to have pre-existing renal insufficiency on presentation (15.6% vs. 4.1%, p<0.001) as compared with SOS patients without compassionate use clinical features.
Table 2
Table 2
Patient presenting factors in the cardiogenic shock or ST segment myocardial infarction PCI cohort, overall and stratified by compassionate use.
In-hospital outcomes for compassionate use cases were dramatically worse than for the high risk SOS cohort of patients without compassionate use clinical features as a whole (Table 3). The likelihood of procedural success was significantly lower for CU cases as compared with non-CU SOS cases (79.2% vs. 94.2%, p<0.001). The CU cases were significantly more likely to suffer new post-procedural cardiogenic shock and more likely to experience bleeding and renal complications following the index procedure than non-CU cases. The unadjusted in-hospital mortality rate was 15.6 times higher for CU cases than for non CU SOS cases (69.8% vs. 4.5%, p<0.001). Although CU cases represented only 1.7% of overall SOS cases, these cases accounted for more than 21% of the overall mortality following PCI in the SOS cohort. Post-procedural death from neurologic causes was significantly more frequent in the compassionate use population as compared with the standard risk SOS cases, likely driven by the concentration of patients presenting with coma in the CU cohort. The observed mortality for CU cases with use of percutaneous ventricular support, rescucitation at the start of the procedure and coma on presentation were 50%, 84% and 70% respectively. Due to the relatively small individual sample sizes, there was no significant difference in the mortality rates between these three categories.
Table 3
Table 3
In-hospital complications and death, overall and stratified by compassionate use.
After adjusting for all other known predictors of in-hospital mortality, compassionate use designation was associated with an odds ratio for in-hospital death of 27.3 (95% CI:14.5–47.6) relative to the non-CU SOS patients. The final multiple logistic regression model included: patient age, pre-procedure renal insufficiency, documented pre-procedure left ventricular ejection fraction <30%, emergent or salvage procedure status, presence of left main coronary artery disease (of severity >50%), presentation with cardiogenic shock and the compassionate use indicator (Table 4).
Table 4
Table 4
Adjusted odds ratios of risk of in-hospital all-cause mortality following PCI in the Commonwealth of Massachusetts. Based on 5,588 PCI admissions from October 2005 through September 2007 and 312 deaths.
Among those CU patients who survived to hospital discharge, 83% (24 of 29) were alive at 30 days, while 76% (22 of 29) were alive at one year.
Inclusion of the compassionate use covariates significantly improved the inhospital mortality risk prediction model performance. The discrimination of the hierarchical mortality prediction model significantly improved from an ROC of 0.87 to 0.90 (p<0.001) with preserved goodness of fit. Individual cases were assigned to five risk strata according to predicted risk before and after incorporation of compassionate use criteria. Incorporation of the compassionate use covariate led to the reclassification of the risk by at least one risk strata for 347 SOS cases (6.2%). The NRI was 8.7% (p = 0.43), indicating that 8.7% more patients who died appropriately moved up a category of risk than down when compared with survivors. Most of the impact of inclusion of the new covariate was observed for those who died within the hospitalization (Figure 1).
Figure 1
Figure 1
Impact of inclusion of compassionate use (CU) on predicted mortality
In this analysis, overall classification of hospital performance did not change significantly with the inclusion of the CU indicator variable (Figure 2). However, there were measurable changes in the width and range of the estimated posterior intervals which made certain centers “closer” to a change in classification as an outlier institution. For example, the posterior interval for Hospital 1 was shifted to the left after including CU indicating a reclassification of cases which results in improved estimated quality relative to the overall State performance, as compared with the method that excluded the compassionate use indicator. In contrast, the posterior interval estimate for Hospital 12 was shifted to the right after inclusion of the CU indicator, indicating that reclassification of cases resulted in a less favorable estimate of risk adjusted mortality performance. Therefore, Hospital 12 was closer to being identified as an outlier for poorer performance after incorporation of CU indicators, than for the models that did not include CU. However, there were no changes in the posterior interval estimates that led to a change in overall classification by institution as either above, below or within expectations for risk adjusted mortality, though the observed changes in posterior intervals confirms the potential impact o inclusion of CU indicators on assessment of hospital quality.
Figure 2
Figure 2
Hospital risk adjusted mortality with and without compassionate use variable
While the impact on physician perception of the risk adjustment methods was not directly measured in this study, a temporal association was apparent after the introduction of the CU indicator in October 2005, with a substantial increase in the prevalence of cardiogenic shock among PCI admissions (Figure 3). The proportion of overall PCI patients presenting in cardiogenic shock monotonically declined from 2.3% in 2003 to 1.3% in 2005. Beginning with the inclusion of compassionate use criteria in late 2005, this trend reversed with an increasing proportion of patients treated with PCI for cardiogenic shock for both 2006 and 2007, to a level of 1.7%. Of note, there was no concomitant change to the risk adjustment methodologies used for predicting mortality after coronary artery bypass graft (CABG) surgery, and the proportion of CABG cases with shock has declined steadily since the inception of public reporting of post-revascularization outcomes (Figure 3).
Figure 3
Figure 3
Emergent revascularization for cardiogenic shock in MA, 2003–2007
The Massachusetts experience demonstrates that a small proportion of patients at extremely high risk for in-hospital mortality can be identified using objective, pre-procedural clinical factors that had not been previously collected as part of traditional quality monitoring efforts. Incorporation of these compassionate use covariates in risk adjustment models lead to significant improvements in model performance as well as reclassification of predicted risk in a substantial proportion of cases. The NCDR dataset was supplemented with compassionate use covariates without significant changes in data collection methods, and these additional variables predictive of mortality were conducive to comprehensive case level review and adjudication.
In Massachusetts between 2005–2007, compassionate use cases represented 1.72% of all high-risk cardiogenic shock or STEMI cases and experienced an in-hospital mortality rate of 69.8% (vs. 4.45% for other shock or STEMI patients). The adjusted odds ratio for compassionate use was 27.3 (95% CI: 14.5–47.6) when compared to the non-compassionate use shock or STEMI patients. Nearly two thirds of compassionate use cases were already coded as presenting in cardiogenic shock; indicating the extreme instability of these CU patients on arrival to the cardiac catheterization laboratory. Importantly, the presence of compassionate use covariate was highly, and independently, predictive of in-hospital mortality, conferring substantial additional risk of in-hospital death beyond that of cardiogenic shock alone. Despite a prevalence of compassionate use criteria in only 1.7% of the analyzed cohort, the inclusion of the covariate significantly improved model discrimination and risk prediction.
Beyond improving the performance of the risk prediction models, the incorporation of compassionate use covariates was intended to increase the confidence of participating physicians in the risk adjustment methodology utilized in Massachusetts. While the potential association between inclusion of compassionate use covariate and changes in physician willingness to treat high risk patients with PCI cannot be directly inferred from the available data, the observed change in proportion of patients treated for cardiogenic shock (Figure 3) may indicate a positive influence on physician acceptance of the risk adjustment methodology.
There was debate within the medical and regulatory communities within MA as to the best approach to handling cases identified as CU. While many physicians advocated excluding such cases altogether from the risk prediction models, the regulator community held that CU was an appropriately “behaved” predictor variable for in-hospital mortality, which added to the discrimination of the prediction models and should therefore be included in the final model. The current study confirms the utility of including CU as an independent predictor of in-hospital mortality, and the temporal association of inclusion of CU in the model with the increasing proportion of the sickest patients in Cardiogenic shock being treated with PCI may demonstrate a physician acceptance of the improvement of modeling performance and therefore patient selection behavior.
This analysis was based on clinical data from a mandatory, state-wide observational registry with a well-publicized and rigorous audit program, and its generalizability is therefore limited. Specifically, each fatal, high risk and compassionate use case was individually reviewed and key variables adjudicated by volunteer, practicing interventional cardiologists trained in use of definitions from both the NCDR data instrument and the CU variables, resulting in high credibility of the final dataset. Not all PCI cases and not all variables in the risk prediction models could be uniformly reviewed, however. Post-PCI mortality occurring after transfer to another facility was also not evaluated.
The generalizability of incorporation of CU variables into voluntary PCI registries such as the NCDR CathPCI registry hinges on the accuracy of institutionally reported CU status, as compared with the central adjudication process utilized in MA. In the first year of use of the CU criteria, 69 cases were coded by the participating hospitals as CU of which 49 (71%) were adjudicated as meeting criteria for CU. This proportion of adjudication rejection for a high risk clinical factor is similar to the MA experience for the institutional coding of cardiogenic shock, and may therefore be representative of the accuracy of institutionally reported high risk features. Therefore, it appears that incorporation of CU in a voluntary registry may be reasonable, given the current reliance on non-centrally adjudicated high risk factors with similar adjudication rejection rates.
In conclusion, incorporation of compassionate use covariates in a statewide clinical outcomes registry is feasible, significantly improves mortality risk prediction model performance, and was temporally associated with an increase in the prevalence of cardiogenic shock among patients undergoing PCI in Massachusetts. Improvements to risk model performance with regards to extremely high risk cases may provide clinicians, regulators and health care consumers with increased confidence regarding the results of mandatory public reporting of risk adjusted PCI outcomes.
The authors thank the members of the Mass-DAC PCI Data Adjudication Committee who participated in numerous data audits; Matthew Cioffi (Mass-DAC) for data management; Treacy Silbaugh (Mass-DAC) for programming expertise; and Hillary Greene (Mass-DAC) for technical assistance. Dr. Resnic’s effort was supported, in part, by the National Library of Medicine (NIH R01-LM008142). The efforts of Normand, Lovett, and Zelevinsky were supported through a contract with the Department of Public Health of the Commonwealth of Massachusetts (620022A4PRE).
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
1. Shahian DM, Torchiana DF, Normand SL. Implementation of a cardiac surgery report card: lessons from the Massachusetts experience. Ann Thorac Surg. 2005;80:1146–50. [PubMed]
2. Shahian DM, Torchiana DF, Shemin RJ, Rawn JD, Normand SL. Massachusetts cardiac surgery report card: implications of statistical methodology. Ann Thorac Surg. 2005;80:2106–13. [PubMed]
3. Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group: cosponsored by the Council on Epidemiology and Prevention and the Stroke Council. Endorsed by the American College of Cardiology Foundation. Circulation. 2006;113:456–62. [PubMed]
4. Resnic FS, Welt FG. The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention. J Am Coll Cardiol. 2009;53:825–30. [PMC free article] [PubMed]
5. Werner RM, Asch DA. The unintended consequences of publicly reporting quality information. JAMA. 2005;293:1239–44. [PubMed]
6. Cutlip DE, Ho KK, Kuntz RE, Baim DS. Risk assessment for percutaneous coronary intervention: our version of the weather report? J Am Coll Cardiol. 2003;42:1896–9. [PubMed]
7. Narins CR, Dozier AM, Ling FS, Zareba W. The influence of public reporting of outcome data on medical decision making by physicians. Arch Intern Med. 2005;165:83–7. [PubMed]
8. Guru V, Naylor CD, Fremes SE, Teoh K, Tu JV. Publicly reported provider outcomes: the concerns of cardiac surgeons in a single-payer system. Can J Cardiol. 2009;25:33–8. [PMC free article] [PubMed]
9. Apolito RA, Greenberg MA, Menegus MA, et al. Impact of the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System on the management of patients with acute myocardial infarction complicated by cardiogenic shock. Am Heart J. 2008;155:267–73. [PubMed]
10. Hochman JS, Sleeper LA, Webb JG, et al. Early revascularization in acute myocardial infarction complicated by cardiogenic shock. SHOCK Investigators. Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock. N Engl J Med. 1999;341:625–34. [PubMed]
11. Description of NCDR CathPCI Registry. American College of Cardiology; 2010.
12. Matheny ME, Ohno-Machado L, Resnic FS. Discrimination and calibration of mortality risk prediction models in interventional cardiology. J Biomed Inform. 2005;38:367–75. [PubMed]
13. Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS -- a Bayesian modelling framework: concepts, structure, and extensibility. Statistics and Computing. 2000;10:325–337.
14. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45. [PubMed]
15. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928–35. [PubMed]
16. Pencina MJ, D’Agostino RB, Sr, D’Agostino RB, Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157–72. discussion 207–12. [PubMed]