PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Circ Cardiovasc Qual Outcomes. Author manuscript; available in PMC 2011 February 4.
Published in final edited form as:
PMCID: PMC3033831
NIHMSID: NIHMS260017

C-Reactive Protein and Reclassification of Cardiovascular Risk In the Framingham Heart Study

Abstract

Background

The relationship of circulating levels of high sensitivity C-reactive protein (CRP) with cardiovascular disease (CVD) risk, particularly with consideration of effects at intermediate levels of risk, has not been fully assessed.

Methods

Among 3006 Offspring participants in the Framingham Heart Study free of CVD (mean age 46 years at baseline), there were 129 Hard coronary heart disease (CHD) events and 286 Total CVD events during 12 years of follow up. Cox regression, discrimination with area under the receiver operating characteristic curve, and net reclassification improvement were used to assess the role of CRP on vascular risk.

Results

In an age-adjusted model that included both sexes the hazard ratios for new Hard CHD and Total CVD were significantly associated with higher CRP levels. Similar analyses according to increasing homocysteine (Hcys) level showed significant protective associations for Hard CHD but not for Total CVD. In multivariable analyses that included age, sex, systolic blood pressure, total cholesterol, HDL-cholesterol, diabetes mellitus, current smoking, hypertension treatment and homocysteine, the log CRP level remained significantly related to developing Hard CHD and Total CVD and provided moderate improvement in the discrimination of events. The net reclassification improvement when CRP was added to traditional factors was 5.6% for Total CVD (P=0.014) and 11.8% for Hard CHD (P=0.009).

Conclusions

Circulating levels of CRP help to estimate risk for initial cardiovascular events and may be used most effectively in persons at intermediate risk for vascular events, offering moderate improvement in reclassification of risk.

Keywords: risk factors, coronary disease, homocysteine, C-reactive protein

Introduction

Traditional risk factors such as age, blood pressure, cholesterol, HDL cholesterol, and diabetes mellitus have been shown to be predictive of coronary heart disease (CHD) and cardiovascular disease (CVD) in a large number of prospective observational studies.1 Novel biomarkers have also been suggested as indicators of increased risk and may contribute to vascular disease risk assessment over and above the use of traditional risk factors.26 Descriptive statistics such as the relative risk of a new factor that is added to a multivariable prediction and the c statistic have been considered inadequate to convey how such new factors may mediate risk in a population setting and there is great interest to employ new methods to quantify such risks.7,8

Information on circulating levels of C-reactive protein may be used to refine estimates of cardiovascular risk stratification using newer methods of assessement.911 This biomarker has generated interest as a potentially important biomarker of inflammation and cardiovascular risk and recommendations for testing with this biomarker were made by the American Heart Association and Centers for Disease Control in 2003.12 We undertook analyses related to the development of both coronary disease and total cardiovascular disease endpoints because there is growing interest in the prediction of total cardiovascular disease as a vascular disease endpoint that is worthy of primary prevention, and a recent Framingham publication has estimated risk of CVD as an initial vascular disease endpoint.13

With this background we investigated the potential benefit of adding information on circulating levels of CRP and Hcys to prediction equations that estimate vascular disease risk in a prospective study of middle-aged and older Framingham adults. We first analyzed the effects of these newer biomarkers using conventional assessment methods, and subsequently evaluated a newly described reclassification approach that used a multivariable model to predict an individual’s risk of developing or not developing a vascular outcome.14

Methods

The second clinic visit of the Framingham Offspring Study in 1979–1983 served as the baseline for this study.15 The examination included assessment of vascular disease prevalence and evaluation of vascular disease risk factors, which was followed by surveillance for the development of new vascular events over the ensuing 12 years. New cases of “Hard CHD” events during the follow-up interval included myocardial infarction and CHD-related death.1 “Total CVD” was a composite measure of the CHD events listed above as well as angina pectoris, transient ischemic attack, stroke, and intermittent claudication. The diagnosis of angina pectoris was determined by information obtained at regular clinic visits and from personal medical records,1 transient ischemic attack and stroke were determined by history, medical records and adjudication by a panel of neurologists,16 and intermittent claudication was evaluated on the basis of a structured questionnaire related to lower extremity symptoms when walking,17

The clinic visit obtained information on cigarette smoking during the past year and use of medications. Blood pressure after sitting for five minutes was measured using standardized methods.18 Phlebotomy took place under fasting conditions. Lipid determinations were made at the time of the examination in the Framingham Heart Study laboratory. Plasma cholesterol was measured according to the Lipid Research Clinics Program Protocol and high density lipoprotein cholesterol (HDL-C) was determined after precipitation of non-HDL lipoproteins with heparin-manganese.19 Aliquots were frozen at −20°C after the initial phlebotomy at the time of the baseline examination. In 2003 the previously unthawed specimens were thawed for measurement of high sensitivity CRP and Hcys. High sensitivity CRP assays were performed in the Framingham laboratory using a previously described nephelometric method with Dade-Behring reagents.20 Hcys was determined in laboratory of Dr. Selhub using high pressure liquid chromatography, as previously described.21

Logarithmic transformations were used for Hcys and CRP for analyses with continuous variables to decrease the effect of extreme observations. The hazard ratios (HR) were estimated using a traditional Cox model that first evaluated age and sex-adjusted effects followed by a multivariable model that included the variables age, sex, cholesterol, HDL-cholesterol, systolic blood pressure, diabetes mellitus, blood pressure treatment, and cigarette smoking. The discriminatory capability of traditional variables and the novel risk factors CRP and Hcys were evaluated using c-statistics, as described previously.22,23 Similar analytic methods were used to test for the effects of three pre-specified CRP categories (<1.00, 1–2.99, ≥3.00 mg/L) and tertiles of Hcys on the risk of Hard CHD and Total CVD.

The effects of reclassification using CRP were assessed using recently published methods that estimated the Net Reclassification Improvement (NRI),14 which expands and improves upon previously published reclassification methods.23,24 The prediction model for each individual was re-estimated with the information for the new factor included in the estimate. This method provides a more rigorous statistical approach to assess the improvement in reclassification by including new biomarker information into prediction models. The analyses used continuous variable information with evaluation of the effects on risk category reclassification for those cases and non-cases during the follow up interval. This approach analyzed separately the reclassification of persons who developed events and those who did not develop events. Reclassification to a risk group with a higher risk was considered upward movement and was considered improvement in classification for those experiencing an event. On the other hand, reclassification downward was considered a failure for persons who developed an event. Conversely, reclassification upward was considered disadvantageous for persons who did not develop an event and advantageous for those who did not develop an event. Improvement in reclassification was estimated by taking the sum of differences in proportions of individuals reclassified upward minus the proportion reclassified downward for people who developed events and the proportion of individuals moving down minus the proportion moving up for those who did not develop events. This overall reclassification sum with this method is the NRI, and the statistical significance of the overall improvement is assessed with an asymptotic test, as described by Pencina.14 The follow up experience over 12 years was adjusted to 10-year categories of risk (0–6%, 6–20%, >20% risk) identified by the National Cholesterol Education Program and other experts for the reclassification analysis.25,26 These estimates were made according to traditional variables (age, sex, systolic pressure, total cholesterol, HDL-cholesterol, diabetes mellitus, smoking status, and blood pressure therapy).

Results

Table 1 shows the baseline characteristics of the participants and number of vascular events. Their mean age was approximately 46 years, diabetes mellitus was uncommon and affected less than 5% of the men and women, and approximately 10% of the individuals were taking blood pressure medications. The mean CRP was 2.67 mg/l in men and 2.28 mg/l in women. The prevalence of CRP <1, 1–3, and >3 mg/L was 43%, 33%, 24% in men and 52%, 28%, and 20% in women. The mean Hcys levels were 7.69 umol/L in men and 6.59 umol/L in women. There were 129 Hard CHD events and 286 Total CVD events (intermittent claudication events: 26 in men and 16 in women; cerebrovascular disease events: 28 events in men and 14 in women) during 12 years of follow up.

Table 1
Baseline Characteristics and Vascular Outcomes for Participants

Table 2 displays the age- and sex-adjusted HR for new vascular disease events. The HRs show the effect on vascular disease risk per unit of log(Hcys) and log(CRP). For example, the HR for the association of log(CRP) with Hard CHD was 1.52 (95% confidence interval [CI] 1.32–1.76) after age and sex adjustment. The multivariable HR for the association of log(CRP) with Hard CHD was 1.34 (95% CI 1.14–1.58) in an analysis that included age, sex, systolic blood pressure, blood pressure therapy, cholesterol, HDL cholesterol, smoking, and diabetes mellitus. Statistically significant associations were also observed for associations of log(CRP) with Total CVD in the age- and sex-adjusted analyses (HR=1.42, 95% CI 1.29–1.57), as well as for the multivariable analyses (HR=1.26, 95% confidence interval 1.12–1.40). In age- and sex-adjusted models, and in the multivariable models, lower Hcys levels were associated with Hard CHD with a hazard ratio in the 0.53 to 0.62 range that was statistically significant. On the other hand, these effects were not observed for Total CVD in the age and sex-adjusted or the multivariable-adjusted models.

Table 2
C-Reactive Protein and Homocysteine Level Hazard Ratios* for Vascular Events

Tertiles of Hcys and commonly used categories of CRP (<1, 1–3, >3 mg/l) were analyzed in age-and sex-adjusted proportional hazards analyses to test for associations with risk for new vascular events (Table 3). The referent group in each instance was the lowest category. Significantly lower risk for Hard CHD events was observed at the higher categories of Hcys in the age and sex adjusted models. On the other hand, no significant associations were observed for Hcys categories on Total CVD risk. Statistically significant increased multivariable HR were observed in persons with CRP > 3.0 mg/L for Hard CHD (HR=1.88, 95% confidence interval 1.18–3.00) and for Total CVD (HR=1.58, 95% confidence interval 1.16–2.15).

Table 3
C-Reactive Protein and Homocysteine Level Hazard Ratios* for Vascular Events

Table 4 shows the c-statistics for Hard CHD and Total CVD outcomes according to different prediction models. The traditional variables were included in a core multivariable proportional hazards regression analyses and the effects of adding information on Hcys and CRP were analyzed. The c-statistic for the traditional multivariable approach was 0.863 for Hard CHD and 0.795 for Total CVD. Only a small increment in the c-statistic was observed when CRP or Hcys information was added to the prediction model.

Table 4
Discrimination of Vascular Disease with Various Prediction Models

A total of 110 persons developed Hard CHD and 2896 did not develop this event during 10 years of follow up (Figure 1), and the follow up interval for this analysis was statistically adjusted from 12 years to 10 years, which resulted in a smaller number of events shown in the reclassification figures compared to Table 1. We initially undertook this analysis to test the net reclassification effect of adding CRP to a predetermined multivariable prediction model that included age, sex, systolic blood pressure, blood pressure therapy, cholesterol, HDL cholesterol, smoking and diabetes mellitus. Persons were classified in 3 different categories according to traditional multivariable risk models (initial probability) and models that included the traditional variables and log(CRP). The majority of persons remained at the same level of risk (along the diagonal from upper left to lower right) after the CRP information was included as an additional variable in the prediction model. Some persons were reclassified upward (above the diagonal) and some were reclassified downward (below the diagonal).

Figure 1
Number of Participants According to Initial Probability of Hard CHD and Probability after CRP Included

The estimates in Figure 1 allow calculation of the Net Reclassification Index using methods reported by Pencina.14 A total of 8 + 10 = 18 persons who developed Hard CHD and were reclassified upward and 5 +1 = 6 persons who developed an event were reclassified downward. The net estimate for the percentage classified upward was the difference between these two estimates divided by the total number of events (18-6)/110= 10.91%, which was statistically significant (P=0.014). Similar calculations for persons who did not develop an event showed a total of 74 + 15 = 89 who were reclassified downward, and 51 + 13 = 64 who were reclassified upward. The net estimate for those not developing a vascular event was the difference between these two estimates divided by the total number of persons who did not develop an event (89 - 64)/2896 for a proportion of 0.86%, which was statistically significant (P=0.043). The NRI was then estimated by taking the sum of the net estimates for those who developed an event and those who did not develop an event, which was 10.91% + 0.86%=11.77% (P=0.009). Analogous calculations for Total CVD from Figure 1 led to an estimated 4.98% (P=0.023) reclassified for those developing an event and 0.61% (P=0.289) for those not developing an event, which yielded an NRI=5.59% (P=0.014).

As an exploratory analysis concerning effects of reclassification, we undertook a stepwise strategy that sequentially estimated the effects for several risk factors with the outcome of Total CVD. We included age and sex as core factors and estimated the percent net reclassified and the c statistic for each variable added. For the addition of systolic blood pressure and treatment to the model, the percent net reclassification and c statistic were 10.8% and 0.740, respectively; for the addition of lipids, 7.0% and 0.767, respectively; for the addition of smoking, 7.7% and 0.787, respectively; for the addition of diabetes, −0.5% and 0.795, respectively; and finally for the addition of log(CRP), 5.6% and 0,799, respectively. The order of adding the variables can affect the statistical significance of the contribution for each factor, and it is interesting that most factors add several percent to reclassification index at the same time that modest increments in the c-statistic are observed. The net reclassification effect for Total CVD is identical to what we reported for the Figure 1 calculation of the net reclassification related to the multivariable model from Table 4.

Discussion

This prospective study tested whether circulating levels of Hcys and high sensitivity CRP affected risk of first CHD events in middle-aged adults. The results showed a statistically significant association of high sensitivity CRP with the incidence of Hard CHD and Total CVD in continuous variable analyses (Table 2) and in categorical analyses for CRP levels > 3.0 mg/L (Table 3). The overall effect on discrimination with Hcys, CRP or both biomarkers was relatively insignificant for both Hard CHD and for Total CVD (Table 4). These results suggested that including each of these factors in an initial screening for vascular disease risk assessment did not measurably improve the ability to discriminate future cases and non-cases. We report a significant association of lower Hcys levels with greater risk for Hard CHD in age- and sex- and multivariable-adjusted regression models, but there was no significant association in models of Hcys with Total CVD (Table 2, Table 3).

Reclassification methods that assessed CRP and vascular disease risk showed significant multivariable adjusted effects of CRP on Total CVD, although adding CRP to the list of traditional CHD risk factors had a minimal effect on discrimination of future events using the c-statistic (Table 3), as shown in previous publications.23,27 In our reclassification analysis that tested for CRP effects at different strata of CHD risk according to traditional risk factors, the analyses showed highly significant CRP effects (Table 4), and a NRI in the 7 – 9 per cent range for Hard CHD or Total CVD as compared to models without CRP information being used in the initial estimation. Similar results have been reported by Cook for CRP in a recently published letter, and she reported a 5.7% reclassification index value for CRP in the Women’s Health Study and a range of 4.7–8.4% for net reclassification in the various models used to develop the Reynolds risk score.28 Our results and those recently reported by Cook are relatively concordant and provide more conservative estimates of reclassification than what was reported previously by Cook4 or in the original Reynolds risk score by Ridker,24 when both CRP and parental history were used in the reclassification model. Unfortunately parental history of heart disease was not available for all Framingham participants at the 1979 examination and the analyses did not allow incorporation of that element into the effects on reclassification without substantial reduction in sample size.

Reclassification can be considered from a variety of vantage points and the exact utility of the method is not clear at the present. Differences in reclassification with CRP or other new variables can stem from a variety of sources. The choice and number of categories used and absolute event rates for the study participants are key sources of variation that can help to explain the differences, as well as the use of NRI, described by Pencina,14 as the performance measures. The NRI considers effects of upward, neutral, and downward reclassification of cases and non-cases during follow up, leading to a net reclassification that provides a more accurate estimate than that obtained with other approaches.

There is considerable interest in the development of effective strategies to identify persons at risk for CHD. Traditional risk factors such as age, sex, blood pressure, cholesterol, HDL cholesterol, cigarette use, and diabetes mellitus have been used to screen persons at risk for CHD in the United States,29 Europe,30,31 and around the world.32 Other biomarkers have been tested in prospective cohort studies for effects in prediction models that have included traditional vascular disease risk models. As with our results, newer biomarkers such as CRP may be statistically related to the development of CHD but using the test for screening or clinical practice is less certain. The addition of newer biomarkers in a previous Framingham publication and results of other studies of CRP in risk prediction have generally shown modest effects in terms of their discriminatory ability to help identify new cases of CHD during follow up, and there was minimal change in the AROC with the new biomarker added.23,33

The Framingham Offspring experience reported in this study reflects that of a suburban, community-based population sample that is largely Caucasian, and follow-up took place from the middle of the 1990’s onward. At baseline the participants often had relatively normal blood pressure levels, and during follow up blood pressure medication was common. The overall effects of baseline blood pressure on CHD risk has been previously reported as smaller in the Framingham Offspring than in the first generation Framingham cohort during the first 12 years of follow up, and effects were largely related to diastolic blood pressure levels.34 Similarly, drug treatment for lipoprotein cholesterol abnormalities was uncommon at the 1990 index examination for the Offspring included in this investigation. Additionally, the participants had a mean age of 46 years at baseline, their risk factor burden was relatively light concerning hypertension and diabetes mellitus, and the incidence of cardiovascular events during follow up was relatively modest in comparison to older population groups. Different results may be obtained in other settings, especially in more recent times when treatment of risk factors has become much more prevalent and both excess adiposity and diabetes mellitus are more common.

The significant reclassification effects of CRP highlight the need to use risk factor assessment strategies that focus testing on those most likely to benefit. One possible approach would be a two-step strategy which first would identify persons at intermediate risk for the vascular outcome via traditional risk factors and then further stratify risk based upon follow-up testing.26,35 Further research is needed regarding the effectiveness of two-step approaches that use reclassification of risk after consideration of additional risk factor information, and cost effectiveness strategies should provide even more information concerning the absolute degree of risk and costs to detect persons at higher risk.

Identifying persons at risk for cardiovascular disease is a dynamic field and newer tests and analytic strategies are constantly being evaluated to improve our ability to assess risk more accurately so that the most appropriate follow up and care can be provided. Our findings in a cohort with a moderate proportion of persons at intermediate risk for CVD showed no improvement in the c-statistic, but with a reclassification approach we saw a net reclassification that was in the 5–10% range. There are many unanswered questions concerning estimation of cardiovascular risk that are related to discrimination and reclassification. The order in which variables are included in risk prediction equations can affect some of the results and interpretation of the findings. Intermediate risk is an arbitrary condition that will change as risk factors are more effectively controlled. Such changes alone will pose new challenges to researchers. Our analytic approach included traditional cardiovascular risk factors first and then evaluated the role of CRP as a new biomarker, but other analytic strategies are possible. We have a better understanding of how risk prediction works at the present time compared to the past, but much is left to be accomplished to improve the identification of persons who will later develop cardiovascular events.

Acknowledgements

This study was supported by R01 HL073272 (Dr. Wilson). From the Framingham Heart Study of the National Heart Lung and Blood Institute of the National Institutes of Health and Boston University School of Medicine. This work was supported by the National Heart, Lung and Blood Institute's Framingham Heart Study (Contract No. N01-HC-25195). The authors acknowledge the computer programming assistance of Peter Shrader.

Abbreviations

CVD
cardiovascular disease
CHD
coronary heart disease
CRP
C-reactive protein
HDL-C
high density lipoprotein cholesterol
HR
hazard ratio
Hcys
homocysteine
NRI
Net Reclassification Improvement

Footnotes

Disclosure: The authors have no potential conflicts of interest to disclose.

Reference List

1. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18):1837–1847. [PubMed]
2. Ridker PM, Hennekens CH, Buring JE, Rifai N. C-reactive protein and other markers of inflammation in the prediction of cardiovascular disease in women. N Engl J Med. 2000;342(12):836–843. [PubMed]
3. Blake GJ, Ridker PM. Novel clinical markers of vascular wall inflammation. Circ Res. 2001;89(9):763–771. [PubMed]
4. Cook NR, Buring JE, Ridker PM. The effect of including C-reactive protein in cardiovascular risk prediction models for women. Ann Intern Med. 2006;145(1):21–29. [PubMed]
5. Ridker PM, Rifai N, Pfeffer MA, Sacks F, Braunwald E. Long-term effects of pravastatin on plasma concentration of C- reactive protein. The Cholesterol and Recurrent Events (CARE) Investigators. Circulation. 1999;100(3):230–235. [PubMed]
6. Ridker PM, Stampfer MJ, Rifai N. Novel risk factors for systemic atherosclerosis: a comparison of C- reactive protein, fibrinogen, homocysteine, lipoprotein(a), and standard cholesterol screening as predictors of peripheral arterial disease. JAMA. 2001;285(19):2481–2485. [PubMed]
7. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928–935. [PubMed]
8. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159(9):882–890. [PubMed]
9. Bostom AG, Silbershatz H, Rosenberg IH, Selhub J, D'Agostino RB, Wolf PA, Jacques PF, Wilson PW. Nonfasting plasma total homocysteine levels and all-cause and cardiovascular disease mortality in elderly Framingham men and women. Arch Intern Med. 1999;159(10):1077–1080. [PubMed]
10. Selhub J, Jacques PF, Bostom AG, D'Agostino RB, Wilson PWF, Belanger AJ, O'Leary DH, Wolf PA, Schaefer EJ, Rosenberg IH. Association between plasma homocysteine and extracranial carotid stenosis. N Engl J Med. 1995;332:286–291. [PubMed]
11. Vasan RS, Beiser A, D'Agostino RB, Levy D, Selhub J, Jacques PF, Rosenberg IH, Wilson PW. Plasma homocysteine and risk for congestive heart failure in adults without prior myocardial infarction. JAMA. 2003;289(10):1251–1257. [PubMed]
12. Pearson TA, Mensah GA, Alexander RW, Anderson JL, Cannon RO, III, Criqui M, Fadl YY, Fortmann SP, Hong Y, Myers GL, Rifai N, Smith SC, Jr, Taubert K, Tracy RP, Vinicor F. Markers of inflammation and cardiovascular disease: application to clinical and public health practice: A statement for healthcare professionals from the Centers for Disease Control and Prevention and the American Heart Association. Circulation. 2003;107(3):499–511. [PubMed]
13. D'Agostino RB, Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117(6):743–753. [PubMed]
14. Pencina MJ, D'Agostino RB, Sr, D'Agostino RB, Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157–172. [PubMed]
15. Feinleib M, Kannel WB, Garrison RJ, McNamara PM, Castelli WP. The Framingham Offspring Study. Design and preliminary data. Prev Med. 1975;4(4):518–525. [PubMed]
16. Wolf PA, D'Agostino RB, Belanger AJ, Kannel WB. Probability of stroke: a risk profile from the Framingham Study. Stroke. 1991;3:312–318. [PubMed]
17. Murabito JM, D'Agostino RB, Silbershatz H, Wilson PWF. Intermittent claudication: a risk profile from the Framingham Heart Study. Circulation. 1997;96:44–49. [PubMed]
18. Wilson PW, Garrison RJ, Castelli WP, Feinleib M, McNamara PM, Kannel WB. Prevalence of coronary heart disease in the Framingham Offspring Study: role of lipoprotein cholesterols. Am J Cardiol. 1980;46(4):649–654. [PubMed]
19. Manual of Laboratory Operations: Lipid Research Clinics Program, Lipid and Lipoprotein Analysis. 2 ed. Washington, D.C.: National Institutes of Health, US Dept of Health and Human Services; 1982.
20. Wang TJ, Larson MG, Levy D, Benjamin EJ, Kupka MJ, Manning WJ, Clouse ME, D'Agostino RB, Wilson PW, O'Donnell CJ. C-reactive protein is associated with subclinical epicardial coronary calcification in men and women: the Framingham Heart Study. Circulation. 2002;106(10):1189–1191. [PubMed]
21. Selhub J, Jacques PF, Wilson PWF, Rush D, Rosenberg IH. Vitamin status and intake as primary determinants of homocysteinemia in the elderly. JAMA. 1993;270:2693–2698. [PubMed]
22. Pencina MJ, D'Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med. 2004;23(13):2109–2123. [PubMed]
23. Wilson PW, Nam BH, Pencina M, D'Agostino RB, Sr, Benjamin EJ, O'Donnell CJ. C-reactive protein and risk of cardiovascular disease in men and women from the framingham heart study. Arch Intern Med. 2005;165(21):2473–2478. [PubMed]
24. Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA. 2007;297(6):611–619. [PubMed]
25. Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III) JAMA. 2001. pp. 2486–2497. [PubMed]
26. Greenland P, Smith JS, Jr, Grundy SM. Improving coronary heart disease risk assessment in asymptomatic people: role of traditional risk factors and noninvasive cardiovascular tests. Circulation. 2001;104(15):1863–1867. [PubMed]
27. Folsom AR, Aleksic N, Catellier D, Juneja HS, Wu KK. C-reactive protein and incident coronary heart disease in the Atherosclerosis Risk In Communities (ARIC) study. Am Heart J. 2002;144(2):233–238. [PubMed]
28. Cook NR. Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine. Stat Med. 2008;27(2):191–195. (DOI: 10.1002/sim.2929) [PubMed]
29. D'Agostino RB, Sr, Grundy S, Sullivan LM, Wilson P. Validation of the Framingham coronary heart disease prediction scores: Results of a multiple ethnic groups investigation. JAMA. 2001;286(2):180–187. [PubMed]
30. Assmann G, Schulte H, Oberwittler W. New aspects in the prediction of coronary artery disease: The Prospective Cardiovascular Munster Study. In: Fidge NH, Nestel PJ, editors. Atherosclerosis VII. Amsterdam: Elsevier; 1986. pp. 19–24.
31. Ferrario M, Chiodini P, Chambless LE, Cesana G, Vanuzzo D, Panico S, Sega R, Pilotto L, Palmieri L, Giampaoli S. Prediction of coronary events in a low incidence population. Assessing accuracy of the CUORE Cohort Study prediction equation. Int J Epidemiol. 2005;34(2):413–421. [PubMed]
32. Liu J, Hong Y, D'Agostino RB, Sr, Wu Z, Wang W, Sun J, Wilson PW, Kannel WB, Zhao D. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA. 2004;291(21):2591–2599. [PubMed]
33. Albert MA, Glynn RJ, Ridker PM. Plasma concentration of C-reactive protein and the calculated Framingham Coronary Heart Disease Risk Score. Circulation. 2003;108(2):161–165. [PubMed]
34. Wilson PWF, Anderson KM, Castelli WP, Kannel WB. Twelve-year incidence of coronary heart disease in middle-aged adults during the era of hypertensive therapy: the Framingham Offspring Study. Am J Med. 1991;90:11–16. [PubMed]
35. Wilson PW, Smith SC, Jr, Blumenthal RS, Burke GL, Wong ND. 34th Bethesda Conference: Task force #4--How do we select patients for atherosclerosis imaging? J Am Coll Cardiol. 2003;41(11):1898–1906. [PubMed]