PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Glob Heart. Author manuscript; available in PMC 2014 March 1.
Published in final edited form as:
Glob Heart. 2013 March; 8(1): 11–23.
doi:  10.1016/j.gheart.2013.01.001
PMCID: PMC3673738
NIHMSID: NIHMS442532

Cardiovascular Disease Risk Assessment: Insights from Framingham

SUMMARY

Cardiovascular disease (CVD) is among the leading causes of death and disability worldwide. Since its beginning, the Framingham study has been a leader in identifying CVD risk factors. Clinical trials have demonstrated that when the modifiable risk factors are treated and corrected, the chances of CVD occurring can be reduced. The Framingham study also recognized that CVD risk factors are multifactorial and interact over time to produce CVD. In response, Framingham investigators developed the Framingham Risk Functions (also called Framingham Risk Scores) to evaluate the chance or likelihood of developing CVD in individuals. These functions are multivariate functions (algorithms) that combine the information in CVD risk factors such as sex, age, systolic blood pressure, total cholesterol, high-density lipoprotein cholesterol, smoking behavior, and diabetes status to produce an estimate (or risk) of developing CVD or a component of CVD (such as coronary heart disease, stroke, peripheral vascular disease, or heart failure) over a fixed time, for example, the next 10 years. These estimates of CVD risk are often major inputs in recommending drug treatments such as cholesterol-lowering drugs.

In this paper, we review briefly the history and original aims of the Framingham study and, in more detail, the history of the Framingham Risk Functions. For the latter, we describe their objectives; the essentials needed for developing them; the methods of evaluating their performance, transportability, and validity in non-Framingham settings; and their recalibration, if needed, for the valid use in these non-Framingham settings. Further, we discuss issues of major current interest involving evaluation of new and novel biomarkers for improving CVD risk prediction, long-term risk evaluation, and the concept of heart (or vascular) age to quantify the state of the vascular system. Along the way, we review the history and development of the Framingham Risk Functions for specific components of CVD and the more recent expansion to assessing the risk for global CVD, which includes coronary and cerebrovascular disease and congestive heart failure.

EPIDEMIOLOGIC BACKGROUND AND DESIGN OF THE FRAMINGHAM HEART STUDY

The Framingham Heart Study is widely acknowledged as a premier longitudinal cohort study. Several historical reviews of its early years exist [15]. During the decades of the 1930s to the 1950s, infectious diseases came under control. The major efforts of public health prior to World War II were directed at control of these diseases, for they were the major causes of morbidity and mortality [1,2]. Improved sanitation greatly decreased diarrheal disease. Strides were made in the control of tuberculosis and pneumococcal pneumonia, and with the introduction of penicillin in 1942, still further dramatic reductions were made. The problem of the infectious diseases was replaced in the 1940s and 1950s by the mounting epidemic of cardiovascular disease (CVD). By the 1950s, 1 of every 3 men in the United States developed CVD before reaching the age of 60 years. Though less prevalent in women, the development of CVD in women had debilitating and often fatal consequences [1,2]. Its prevalence was twice that of cancer. It had become the leading cause of death and the reason why life expectancy beyond age 45 years did not increase. Furthermore, there were no known treatments capable of prolonging life, even in those who managed to survive an attack. Added to this was the fact that little was known about the determinants of the disease process itself, so methods for reversing the epidemic were not even conjectured.

Action was a needed. There were serious activities in developing methods to treat and reverse the process of CVD, but these were still mainly in the conceptual and development stages. Given the preceding circumstances, there were many who believed a primary prevention approach would be promising and possibly more important than a search for cures [1]. Dawber [1] presents the following. Most people will ultimately succumb to some degenerative disease, including CVD. Complete avoidance is not possible. However, the onset of CVD might be delayed by preventive approaches. If the onset could be delayed, possibly life expectancy could be significantly increased. To develop a preventive approach, the preventable and modifiable pre-disposing factors had to be identified. Further, CVD is a disease that is multifactorial and develops over time, so a longitudinal study was necessary. To study CVD appropriately, it was necessary to identify people without CVD, note their lifestyle and possible other factors such as age and sex, follow them over time, and relate the factors to the development of CVD. A longitudinal cohort epidemiological study was deemed necessary to identify factors and relate them to the development of CVD [1,3]. This approach explores, “certain relationships in health and disease which, with present technological methods, cannot be observed directly” [3]. The factors that did relate to the development of CVD were later labeled by Dr. William B. Kannel as CVD risk factors.

The thinking just described led to the initiation of the Framingham Heart Study (or, because of its breadth over time, the Framingham study). To achieve the objectives, a systematic sample of 2 of every 3 families in the town of Framingham, Massachusetts, was selected. People in those families between the ages of 30 and 59 years were invited to participate in the study. Ultimately, 5,209 individuals (2,336 men and 2,873 women) joined the study. The major aim of the study was to secure epidemiological data on CVD. This encompassed the establishment of the relation of risk factors (e.g., clinical parameters such as age, sex, blood pressure, cholesterol, body weight, diabetes, and lifestyle parameters such as smoking, physical activity, and alcohol consumption) to CVD. Biennial history and physical examinations were administered in which the risk factors were evaluated. Procedures such as electrocardiography, spirometry, and blood and urine testing were administered. Continuous surveillance methods identified when a CVD event occurred. Clinical and statistical issues of the relations of risk factors to disease never addressed before in longitudinal studies were raised and met.

The Framingham study was very successful with an offspring cohort initiated in 1971 (2,489 men and 2,646 women) and a third-generation cohort in 2001 with over 4,000 subjects [6,7].

GENESIS OF MULTIVARIABLE FRAMINGHAM RISK FUNCTIONS

Originally, the Framingham study examined the relation of individual risk factors to the development of CVD. Within a decade of the study, it was clear that the hypothesized risk factors do contribute to CVD. Further, it became clear that the presence of multiple risk factors did increase risk. Figure 1 uses 6-year follow-up data to show the impact of the joint absence or presence of elevated blood pressure (blood pressure ≥ 160/95 mm Hg), elevated total cholesterol (≥260 mg/dl) and the presence of left ventricular hypertrophy on developing coronary heart disease (CHD). As early as 1961, Kannel et al. [8], using “risk factor” explicitly, reported on this 6-year follow-up data and stated that combinations of the 3 risk factors appear to augment further the risk of subsequent development of coronary disease. He then went on to say that as additional longitudinal observations are made, it is hoped that additional risk factors will be determined. With this, emphasis shifted to the question as to whether the individual risk factors could be combined into a multivariate function to give an assessment (or the probability of or the risk) of developing a CVD event over a specific period (say, 10 years). The original hypothesis of the Framingham study was that CVD was multifactorial. A multivariable assessment was the logical consequence. Furthermore, the attention focused on the development of a primary CVD event (that is, a first CVD event in a person free of CVD at the time of risk factor evaluation). At this time, the focus was often on the prediction of CHD. The era of the Framingham Risk Functions began.

FIGURE 1
Risk of CHD according to elevated blood pressure (BP), elevated cholesterol, and left ventricular hypertrophy: Framingham cohort 6-year follow-up. Elevated BP = ≥160/95; elevated cholesterol = ≥260 mg/dl

BRIEF LISTING OF MAJOR FRAMINGHAM RISK FUNCTIONS

Dr. William B. Kannel said, “Multivariable risk formulations [now called Framingham Risk Functions or Framingham Risk Score] for estimating the probability of CVD conditional on the burden of a number of specified risk factors [were and] have been produced to facilitate evaluation of candidates for CVD in need of preventive management” [9]. The 1960s saw the first formal presentation of Framingham Risk Functions, which employed the statistical technique of discriminant analysis and the then-new logistic regression analysis, presented for the first time in a computationally doable manner, that was explicitly devised for computing Framingham Risk Functions [1012]. The 1970s and 1980s saw the development of coronary and general CVD functions [1315]. The newly developed statistical methods of survival analysis (time-to-event analyses) were employed in the 1990s [1618]. The Anderson et al. [17] CHD function received major attention, support, and use from European and U.S. societies. However, in the United States, what was probably the Framingham Risk Function that had the first real major impact on guidelines was the Wilson, D’Agostino et al. [18] primary CHD function, which incorporated explicitly, as risk factor variables, the risk categories used for cholesterol [19] and hypertension [20] national guidelines. This Framingham Risk Function estimated the risk for general CHD (angina, myocardial infarction [MI], and coronary death) over a 10-year period.

Other important Framingham Risk Functions that have received widespread attention and use are for hard CHD (coronary death or nonfatal MI) [21], stroke including transient ischemic attack [22,23], secondary CHD events [24], intermittent claudication [25], heart failure [26], stroke or death after atrial fibrillation [27], and the development of atrial fibrillation [28]. Of special note, the hard CHD function of D’Agostino, Grundy, et al. [21] was the motivation of the Framingham Risk Function that was used for the Adult Treatment Panel III assessment tool for the risk of hard CHD [2931].

More recently, the Framingham investigators have presented a Framingham Risk Function for global CVD (including CVD death, general CHD, stroke [including transient ischemic attack], intermittent claudication, and congestive heart failure) [32]. This function has raised the question that functions that focused mainly on coronary disease may be too narrow and a broader outcome is warranted. The new cholesterol, blood pressure, and dietary guidelines will focus on such an approach where hard CVD (coronary deaths, MI, and stroke) will be the CVD event of interest.

DEVELOPMENT AND INTERNAL EVALUATION ON FRAMINGHAM STUDY DATA

The development of the Framingham Risk Functions has been well documented [818,2133]. Most of these references do not systematically discuss the steps involved in developing them and evaluating their performance. In this section, we give a more detailed presentation of the issues and methods involved in developing and evaluating a Framingham Risk Function focusing on how well the function works on the Framingham data on which it was developed. Later, we will discuss their transportability.

Table 1 lists 11 issues in development and evaluation of risk prediction models. We use for illustration the development of the hard CHD (coronary deaths and MI) of D’Agostino, Grundy et al. [21]. In this section, we focus on the first 8 issues. The first 6 items (steps) deal with the development of the model. The important components in the development are to have: 1) a clearly defined component of CVD that is of clinical relevance and interest such as MI or coronary death; 2) a clearly defined set of individuals who are the at-risk population, for example, those free of CVD; 3) a selected follow-up time such as 10 years; 4) a well-defined and obtainable set of CVD risk factors such as systolic blood pressure, total cholesterol, and smoking; 5) a mathematical model to relate the CVD risk factors to the development of the disease; and 6) the ability of the model to produce risk estimates such as relative and absolute risks.

TABLE 1
Issues in development, evaluation, and validation of multivariate risk models (internal, external, and extensions)

Specific to the illustration, we have as the first step the selection of the endpoint (event/outcome) for which we desire to estimate the risk. For the model under discussion, the endpoint is first (primary) hard CHD event.

The second step is the identification of the at-risk population. Framingham Risk Functions are traditionally sex-specific. For this particular activity, sex-specific functions were to be developed. They were derived from 2,439 men and 2,818 women, 30 to 74 years of age who were free of all CVD at the time of their Framingham study examination from 1971 to 1974. Participants attended either the 11th examination of the original Framingham cohort or the initial examination of the Framingham Offspring Study. (It should be noted that it is possible that the at-risk population is the first step followed by the endpoint of interest as the second step.)

Third, the follow-up time is selected. The functions were developed so that they could be used to produce 10-year risk estimates. It is useful for the dataset to have a follow-up that is slightly longer than the desired estimation time. In line with this, 12-year follow-up was obtained on all subjects for the development of hard CHD.

Fourth, the risk factors of the Framingham Risk Function need to be selected. Table 2 contains the risk factors. They include age, blood pressure categories, total cholesterol categories, high-density lipoprotein (HDL) cholesterol categories, smoking behavior (yes/no), and diabetes status (yes/no).

TABLE 2
Risk factors for hard CHD model

Next (step 5), the mathematical model is selected. Originally, discriminant models were used [10,11]. These were replaced with logistic regression models [12]. Presently, time-to-event survival models such as the Cox proportional hazard regression [18,2123,32] or accelerated failure models (Weibull) [17,24] are used. Figure 2 displays the Cox model.

FIGURE 2
Cox model: (A) mathematical model; (B) general structure of Cox model survival to time t.

For step 6, the mathematical model should have the capability of supplying the needed estimates of risk. The Cox model contains a linear function (function L in Fig. 2B), the coefficients (betas) of which are related to the hazard ratios (relative risks), which are equal of the exponential of beta of each risk factor. Further, the Cox model readily produces an estimation of the absolute risk of an event occurring within a period from time 0 to time t (see Fig. 2). In many Framingham Risk Functions, the absolute risk of interest is the absolute risk at time t =10 years. Note, whereas relative risk is essential for the mathematical model to produce, it is widely accepted that absolute risk is the risk of interest and that treatment decisions should be based on absolute risks [3436]. Table 3 contains the Framingham Risk Function for the hard CHD as developed previously [21].

TABLE 3
Framingham functions (Cox regression coefficients) hard CHD model (MI + coronary death)

After the risk function is developed, it is essential to evaluate its performance and internal validation. Issue (step) 7 of Table 1 identifies 2 major areas of performance measures—discrimination and calibration—for evaluation of performance. Discrimination refers to the function’s ability to discriminate cases from noncases. The area under the receiver-operating characteristic, also called the C-statistic, is the most used measurement of discrimination [3740]. The C-statistic summarizes the sensitivity and specificity of the risk function. Figure 3 displays a typical receiver-operating characteristic curve. The C-statistic has a probability interpretation, namely it is the probability that the risk function will assign a higher risk (absolute probability) to a randomly selected person who will develop an event than a randomly selected person who will not. A C-statistic =0.50 means the function is no better than a guess. Values above 0.50 indicate discriminatory ability. For example, a C-statistic =0.80 indicates that 80% of the time the risk function will give a higher absolute risk to a randomly selected person who will develop the disease than a randomly selected person who will not. Although it is arbitrary to decide what is a good value of the C-statistic for a risk function, often a value below 0.70 is considered suboptimal, a value from 0.70 to 0.80 is considered good, and a value of 0.80 or above is excellent. For the Framingham Risk Functions of Table 3, the C-statistics are 0.79 and 0.83 for the male and female functions, respectively, indicating good to excellent discrimination. It should be noted that the C-statistic is basically a measure of the risk function to rank risk. Once good discrimination is established, the evaluation of its ability to produce the correct absolute risks is important [37,38]. This is the function of calibration.

FIGURE 3
Receiver-operating characteristic curve for prediction of coronary heart disease: Framingham men.

Calibration refers to the risk function’s ability to produce the correct absolute probability. For example, if 15% of a group of individuals with a particular risk profile develops CHD over a 10-year period, we want the risk function to produce a 10-year risk (probability) of 15% for subjects from that group. Calibration for time-to-event analyses can be evaluated by the Nam-D’Agostino chi-square test [38]. This test quantifies the difference between the predicted probabilities versus the estimated event rate in the deciles of risk. The deciles of risk are generated by listing in order of magnitude the predicted probabilities and dividing them into 10 equal groups. Within each decile, the predicted probabilities are averaged. Furthermore, within each decile, the Kaplan-Meier estimate for the selected time of follow-up (often 10 years) is computed [21,38]. These are estimated event rates. Figure 4 shows a calibration plot contrasting the predicted probabilities and event rates. The calibration statistics for Nam-D’Agostino chi-square test are as follows:

χ2=j=1Mnf=[KMjpj¯]2pj¯(1pj¯)

(Kaplan-Meier estimates as observed. Chi-square distribution with M −1 degrees of freedom). Under the null hypothesis that the risk model is appropriate, the test statistics has a chi-square distribution with M −1 degrees of freedom, where M is the number of categories (M = 10 when deciles are used). Small values of the chi-square statistics (as a rule of thumb, less than 20) have become conventional in deciding the calibration is good. It should be mentioned that to this day many investigators developing and evaluating calibration of risk function do not perform the correct chi-square test. They often inappropriately use a Hosmer-Lemeshow test [41], which is appropriate to the logistic regression, not to a time-to-event risk function. The Nam-D’Agostino chi-square statistics for the models of Table 3 are, respectively, 3.3 and 3.7, indicating excellent calibration.

FIGURE 4
Calibration plot showing mean risk calibrated by decile.

The last aspect of evaluation of the risk model on the data on which it was generated is interval validation. This is item 8 of Table 1. It must be anticipated that a model will fit the data on which it was developed better than it will fit another dataset. Internal validation attempts to take into account the overfitting that comes from developing a model on a particular dataset and then evaluating it on the same data.

Several internal validation procedures have been proposed and used over the years. These include variations of split-sample, cross validation, and bootstrapping. One method often used on Framingham data is a 10-fold split in which the original sample is randomly divided into 10 samples; models are fit on all sets of 9 of these samples; and the resulting 10 models are used individually to estimate the risks on the 10th samples (the sample not used in the model development). This procedure consists of 10 models being developed on the different sets of 90% of the data, and then each of these being used to estimate risks on the remaining 10th dataset. The C-statistic, the calibration plots, and Nam-D’Agostino calibration statistic are computed on the estimated results. This procedure was applied to the preceding model (Table 3) and produced no meaningful differences from what were given previously for the C-statistics and chi-square tests.

Some have argued that bootstrapping provides the best estimate of internal validation at least when applied to logistic regression models [42]. Framingham investigators have not found it to give different results from the 10-fold split for logistic regression or time-to-event models such as the Cox model. Whereas internal validation is essential, the best way to validate a prediction/risk model is to apply it to a new independent dataset. This has been done extensively to Framingham models and we now turn our attention to it.

TRANSPORTING AND VALIDATING FRAMINGHAM RISK FUNCTIONS IN NON-FRAMINGHAM POPULATIONS: EXTERNAL VALIDATION

Framingham investigators have long held that external validation provides the best evidence of the ability to generalize with a multivariate risk model [43]. This is item 9 (Transporting [Performance and Recalibration]) of Table 1. The Framingham Risk Functions were developed in a white middle-class sample, which raised concerns about whether these functions can be generalized to other populations [21]. There were a number of early appraisals (pre-1990) of the Framingham Risk Functions for CHD on non-Framingham populations, and these basically found that the Framingham functions did reasonably well except in areas in which the incidence of CHD was very low [4447]. However, even in the low-incidence regions, it was possible to distinguish people at high risk from those at low risk by adjusting the intercepts of the discriminant and logistic models to adjust for the true absolute risk. Since the use of time-to-event models such as the Cox and Weibull models (1990 onward), there have been numerous validation studies of the Framingham Risk Functions by non-Framingham and Framingham investigators [21,4859]. Some of the most rigorous external evaluations were those involving the hard CHD function displayed in Table 3 [21,48,58,59].

In January 1999, a National Heart, Lung, and Blood Institute workshop on CVD risk assessment was convened to address whether Framingham Risk Functions for CHD were valid in (or could be validly transported to) diverse U.S. populations. To assess their transportability, they were compared with those developed from other prospective studies [21,48]. Sex-specific Cox regression models were developed for each U.S. study using the risk factors of Table 2. The external validation study had 3 major components. First, for each of the sex-specific models developed on the non-Framingham study data, each risk factor’s coefficient was compared with the corresponding coefficient of the Framingham sex-specific model of Table 3 and tested if it was statistically different from the Framingham model coefficient. Second, the discrimination C-statistic was computed for each of the sex-specific models developed. These C-statistics were compared with the C-statistics obtained by using the sex-specific Framingham model of Table 3 to generate absolute risk probabilities. Third, the Nam-D’Agostino chi-square statistic was computed, again, for the sex-specific models developed using the particular study’s data, and this was compared to the chi-square computed using the sex-specific Framingham function of Table 3. (See D’Agostino, Grundy, et al. [21] and Liu, et al. [59] for more details on procedure.)

The Framingham Risk Function of Table 3 fit the data from the ARIC (Atherosclerosis Research in Communities) study and the PHS (Physicians Health Study) basically as well as the best function obtained by developing a risk function directly on their data. For each cohort, as would be expected, the use of the study-specific risk function was numerically better in predicting their CHD events than was the Framingham Risk Function. However, the differences were usually well within what would be explained by the overfitting of the study-specific functions. The workshop concluded that the Framingham Risk Functions could be used broadly in white population samples in the United States. Although the blood pressure variable appeared to be a stronger risk factor in African Americans, the Framingham Risk Functions appeared to apply well to white and African American population samples [21,48].

In Asian men from Honolulu and Hispanics from Puerto Rico, the Framingham Risk Functions of Table 3 overestimated the CHD risk. However, a simple calibration adjustment of the Framingham Risk Models made to compensate for the lower average baseline risk for CHD greatly improved their performance [21]. The procedure for the calibration adjustment is presented in Figure 5. It consists of replacing the average Framingham underlying risk (S0[t] of the Cox model [see Fig. 2 and Fig. 5]) with the average event rate for the study estimated by the Kaplan-Meier estimate [60] and the means of the risk factors from the Framingham model with the means for the particular study (see Fig. 2B and Fig. 5). See D’Agostino, Grundy et al. [21] for further elaboration on recalibration.

FIGURE 5
Calibration adjustment for transportation to a different population. See Harrell et al. [37].

Of special note related to the recalibration of Framingham Risk Scores are the adaptations to European Mediterranean areas [58] and to the CMCS (Chinese Multi-Provincial Cohort Study) [59]. The latter consisted of 30,121 Chinese adults from 16 centers in 11 provinces of China and also from Beijing, ages 35 to 64 years at baseline, and who were followed for up to 10 years (follow-up rate of 86%). Figure 6 and Figure 7 show the results of the discrimination and calibration comparison, respectively, of the best Cox regression risk function computed on women from the CMCS to the Framingham Risk Function of Table 3 applied to the female CMCS data. The C-statistics are 0.759 and 0.742, respectively (Fig. 6). The value of the Nam-D’Agostino chi-square statistics is 147.6 for the Framingham Risk Function applied directly to the CMCS data. After recalibration, it is 16.9. The chi-square statistic applied to the best CMCS model is 14.2 (Fig. 7).

FIGURE 6
C-statistics for women in the Chinese Multi-Provincial Cohort Study (CMCS) and Framingham Heart Study (FHS).
FIGURE 7
Chi-square calibration for women in the Chinese Multi-Provincial Cohort Study (CMCS) and Framingham Heart Study (FHS)

Four useful references dealing with development, performance evaluation, and validation are D’Agostino, Griffith, and Schmidt [61], Brindle, May et al. [62], Barzi et al. [52], and Whittemore [63]. These offer both general ideas and specific examples.

RISK FUNCTIONS IN TREATMENT GUIDELINES

CHD is the most common outcome of Framingham standard risk functions. It equals all the other CVD manifestations in incidence and is the most lethal. As was mentioned earlier, much of the attention and focus of Framingham investigators was directed at developing risk functions for CHD [10,13,15,17,18,21]. These efforts culminated in the development of the Adult Treatment Panel III (ATP III) hard CHD function [2931]. Table 4 contains explicitly all the input needed to use the function for men. The risk factor variables include logs of age, total cholesterol, HDL cholesterol, and systolic blood pressure. Also, it contains a dichotomous variable for blood pressure medication (yes/no), smoking (yes/no), and interaction variables with the log age. It produces 10 year absolute probability estimates of the risk of developing hard CHD (coronary death or MI).

TABLE 4
ATP III hard CHD function for men

This function was developed to evaluate if a patient was suitable for statin therapy. The ATP III guidelines classify people into high risk (CHD or a CHD equivalent or 2 or more risk factors and 10-year risk greater than 20%), moderately high risk (10-year risk between 10% and 20% and 2 or more risk factors), moderate risk (10-year risk less than 10% and 2 or more risk factors), and low risk (0 or 1 risk factors). Based on these risk categories, treatment recommendations are made [29]. It should be mentioned that the Framingham ATP III function and other major Framingham Risk Functions are available on the Framingham Web page including formula and Excel spread sheets (Microsoft, Redmond, Washington, USA) [64].

RISK PREDICTION FOR GLOBAL CVD

Whereas CHD is a major component of overall CVD, it does not include other important manifestations such as stroke, heart failure, and peripheral arterial disease. Framingham investigators have considered that for risk functions, a broader class of CVD components would be appropriate, especially for public health. In line with this in 2008, Framingham investigators presented a global CVD risk function for estimating the risk of developing any manifestations of CVD (including CHD, cerebrovascular disease, intermittent claudication, and congestive heart failure) [32]. The global CVD function produces an absolute risk estimate for global CVD and, by simple adjustments, the risk for any of the components of CVD [32]. Table 5 is the score sheet for the female global CVD function, which also can be used to obtain absolute risks of CVD. As an example, for a woman age 61 years, with HDL of 47, total cholesterol of 230, nontreated systolic blood pressure of 124, and who is a nonsmoker and nondiabetic, the corresponding points are 9, 0, 3, 0, and 0 for a total of 12 points, which corresponds to a 10-year risk of CVD of 8.6%. For comparison, the ATP III function produces for this woman a 10-year risk of developing a hard CHD of 3%.

TABLE 5
CVD points and risk for women

The global CVD function was not developed to replace the risk functions for individual CVD components, but rather to bring unity to the subject. First, it deals with primary prevention. That is, the person is free of any manifestation of CVD. Second, it can produce a risk estimate of CVD along with estimates of the separate components of CVD. It supplies a full assessment across the spectrum of CVD.

SCORE SHEETS VERSUS MATHEMATICAL MODEL’S ESTIMATES

Whereas Table 5 can be used to produce a 10-year estimate of CVD, the Framingham Risk Functions are based on a mathematical model such as the male ATP III model of Table 4. Most of the risk functions are based on Cox regression models [38]. The score sheet is only an approximation to the mathematical function. Excel programs have been developed to produce the risk based on the mathematical functions. These are available on the Framingham Website [64]. A score sheet for the global CVD function was used to illustrate multivariate relations among risk factors. For estimating risk estimates in practice, we strongly suggest the use of the Excel programs.

ADDING NEW RISK FACTORS TO RISK FUNCTIONS

CVD often occurs in people with what is considered acceptable or average risk factors values. This suggests that there are other risk factors that must be responsible and which, if added to the present set of CVD risk factors, would improve risk estimation. A number of risk factors (some new and often called novel risk factors) have been proposed. Among these are inflammatory markers such as C-reactive protein, coronary calcium, lipoprotein (a), interleukin-6, fibrinogen, homocysteine, small-dense low-density lipoprotein, insulin resistance, metabolic syndrome, function assess by tissue-type plasminogen activator and plasminogen activator inhibitor 1 antigens, platelet function, and genetic scores [6576]. These new variables, for the most part, have had little effect on changing the discrimination C-statistics. There have been some positive results. Polak et al. [75] showed that the carotid-wall intima-media thickness does add significantly to CVD prediction. Furthermore, coronary artery calcium scores also appear to add significantly [76].

There seems to be at least 2 major problems. First, once the C-statistic is in the range of 0.75, it is very hard mathematically to improve it. It is confined to the range 0.5 to 1.0. Even when the change in the C-statistic is statistically significant, it appears numerically often to be trivial. Second, the procedure often used for testing if the C-statistic increases has been misused; significant changes were possibly not recognized [77,78].

For evaluating a new risk factor, an approach that did not focus solely on the change in the discrimination C-statistic was needed. Emphasis shifted to asking if a new risk factor could reclassify subjects that improve risk estimation. For example, in the ATP III model classifications mentioned earlier, would C-reactive protein shift a person appropriately from a moderate- to high-risk category? Framingham has contributed substantially to this effort. Pencina, D’Agostino, and other Framingham investigators have produced a number of papers dealing with this reclassification concept [40,7983]. Pencina, D’Agostino, Sr., D’Agostino, Jr., and Vasan [79] defined 2 statistics to measure the new risk factors’ ability to improve prediction. The Net Reclassification Index measures how much more reclassification is appropriate (moves those who will develop events into higher categories) versus inappropriate, so that reclassification can be attributed to the new risk factor. The Integrative Discrimination Index is basically a measurement of how the R2 (explained variance) improves with the addition of the new risk factor. These measures do have solid interpretations, and quantification of meaningful improvements is now possible [8183].

LIFETIME AND LONG-TERM RISK FUNCTIONS

In addition to short-term (up to 10 years) risks, Framingham investigators have had an interest in lifetime and long-term (for example, 30 years) risk estimation. In 1997, Seshardri et al. [84] published a paper on the lifetime risk of dementia and Alzheimer’s disease. Methodologically, this paper identified the need to consider competing risks (such as death from other causes) when generating long-term estimates of the effect of risk factors. Given its importance, a methodological paper, including a computer macro was published in 2002 by Beiser et al. [85]. Lifetime risk estimation work was extended to CVD, led by Drs. Donald Lloyd-Jones and Daniel Levy, and resulted in a number of important papers on the lifetime risk of CHD, congestive heart failure, and atrial fibrillation [8689]. These papers basically consisted of identifying a group of individuals and following them for an extended period. They did not have the feature of looking at a number of risk factors and evaluating the impact of these on the long-term outcomes as did the 10-year Framingham Risk Functions such as the hard CHD and ATP III functions of Table 3 and Table 4. To address this, Pencina, D’Agostino, Larson et al. [90] generated risk models for estimating the risk of CVD over a 30-year period. For a subject between the ages of 20 and 40 years, a 30-year risk can be computed that relates the standard CVD risk factors at baseline to the development of developing CVD over the next 30 years.

The need and usefulness of a long-term risk is becoming recognized as an important addition to risk estimation [91]. Many individuals have a low 10-year CVD risk, but a substantial 30-year risk, such as a 30-year-old woman with total cholesterol of 250 and other risk factors at normal levels, who would have a 10-year risk lower than 10%. Incorporating long-term risk estimation in the evaluation of this individual could lead to better care. For example, a subclinical test may be suggested based on the 30-year high risk or a change in lifestyle.

HEART (VASCULAR) AGE

Continuing the theme that some people (especially young women or older people with only 1 elevated risk factor) could have low 10-year probabilities, another risk feature that Framingham investigators have presented is the so-called heart (or vascular) age. The heart age is the age that corresponds to a person with normal risk factors and the same 10-year absolute risk. It is possible for a person to have a low 10-year risk, but have a vascular age much older than their chronological age. Consider the woman used in the global CVD section of this paper. She was 61 with total cholesterol of 230, HDL of 47, a nontreated systolic blood pressure of 124, a nonsmoker, and was without diabetes. The total number of points from Table 5 was 12 and her 10-year risk was 8.6%. Using Table 6, which converts points into heart (vascular) age, we see that the heart age is 68. With respect to CVD risk, this woman has a heart age or vascular system equivalent to a 68-year-old woman who has normal risk values on all CVD risk factors. That is, the 61-year-old woman has the same 10-year risk as a 68-year-old woman whose risk factors are all at normal levels. This heart age does give information as to the status of the woman’s health that is not clear from the 10-year risk alone.

TABLE 6
Heart age/vascular age for women

The global CVD risk paper contains charts that can be used to give heart age [32]. Again, the Framingham Web page [64] can also supply these.

OTHER CVD RISK FUNCTIONS

Whereas the Framingham Risk Functions are the most widely used for clinical guidelines, there are a number of other important risk functions [91]. Of special note are the European Systematic Coronary Risk Evaluation (SCORE) function [92], the PROCAM (Prospective Cardiovascular Münster) model [93], the QRISK function [94] and the Reynolds functions [95,96]. The important SCORE function is actually a set of functions used in Europe and consists of both region-specific and country-specific functions. It is applicable for the risk estimation of only fatal CVD events. The PROCAM function is applicable for CHD deaths and nonfatal MI. The QRISK function applies to CHD, stroke, and transient ischemic attack. The Reynolds risk scores apply to CVD death, MI, stroke, and revascularization. With regard to discrimination, most of these and other risk functions perform in a similar fashion. Calibration, however, may vary and, as we have discussed, the ability to recalibrate the Framingham Risk Functions is an important feature. An informative review of CVD risk functions discussing in depth their rationale, comparing them in content and performance, discussing whether they improve patient outcomes, and suggesting their future is given by Cooney et al. [97].

WHAT IS NEXT?

The Framingham study has played a major leading role in the development and dissemination of risk prediction functions. The Framingham Risk Functions have been used and tested in numerous settings. They do have validity and transportability. In the United States, there are other epidemiologic studies with similar data [98100]. It is logical to combine the data from these studies and produce risk models that are applicable to most, if not all, of the United States. This activity is underway. The models from this effort will focus on short-term risks (10 years), but they will also realize that long-term risk such as 30-year risk and concepts such as heart (vascular) age are essential for quantifying and understanding risk. Further, the possible addition of new variables added to improve prediction must be pursued. In particular, genetic markers offer tremendous possibilities [76]. Most important, it is essential that these new models be used in treatment guidelines [101]. Determining the best ways to convey the risks, convert the risk to treatment action, and judge the usefulness and success of these are imperative [82,97,101105]. The lessons that we have learned and continue to learn from the Framingham study and the development and implementation of its Framingham Risk Functions can help bring success to these future endeavors.

Acknowledgments

Dr. D’Agostino is the coprincipal investigator of the Framingham contract from the National Heart, Lung, and Blood Institute’s Framingham Heart Study, National Institutes of Health. Framingham Heart Study research is supported by National Institutes of Health/National Heart, Lung, and Blood Institute contract #N01-HC-25195

REFERENCES

1. Dawber TR. The Framingham Study: The Epidemiology of Atherosclerotic Disease. Cambridge, MA: Harvard University Press; 1980.
2. D’Agostino RB, Sr, Kannel WB. Proceedings of the American Statistical Association Sesquicentennial Invited Paper Sessions. Alexandria, VA: American Statistical Association; 1989. Epidemiological background and design: the Framingham study.
3. Dawber TR, Meadors GF, Moore FE., Jr Epidemiological approaches to heart disease: the Framingham study. Am J Public Health Nations Health. 1951;41:279–281. [PubMed]
4. Dawber TR, Kannel WB, Lyell LP. An approach to longitudinal studies in a community: the Framingham study. Ann N Y Acad Sci. 1963;107:539–556. [PubMed]
5. Dawber TR, Moore FE. Longitudinal study of heart disease in Framingham, Massachusetts: an interim report. Research in Public Health: Papers Presented at the 1951 Annual Conference of the Milbank Memorial Fund. 1952:241–247.
6. Kannel WB, Feinleib M, McNamara PM, Garrison RJ, Castelli WP. An investigation of coronary heart disease in families: the Framingham Offspring Study. Am J Epidemiol. 1979;110:281–290. [PubMed]
7. Splansky GL, Corey D, Yang Q, et al. The Third Generation Cohort of the National Heart, Lung, and Blood Institute’s Framingham Heart Study: design, recruitment, and initial examination. Am J Epidemiol. 2007;165:1328–1335. [PubMed]
8. Kannel WB, Dawber TR, Kagan A, Revotskie N, Stokes J., 3rd Factors of risk in the development of coronary heart disease—six year follow up experience. The Framingham Study. Ann Intern Med. 1961;55:33–50. [PubMed]
9. Kannel WB. Multivariate evaluation of candidates for cardiovascular disease. In: Rosendorff C, editor. Essential Cardiology Principles and Practice. 2nd. Totowa, NJ: Humana Press; 2006. pp. 3–18.
10. Truett J, Cornfield J, Kannel W. A multivariate analysis of the risk of coronary heart disease in Framingham. J Chronic Dis. 1967;20:511–524. [PubMed]
11. Cornfield J, Gordon T, Smith W. Quantal response curves for experimentally uncontrolled variables. Bull Int Stat Inst. 1961:28.
12. Walker SH, Duncan DB. Estimation of the probability of an event as a function of several independent variables. Biometrika. 1967;54:167–179. [PubMed]
13. Coronary Risk Handbook. Estimating Risk of Coronary Heart Disease in Daily Practice. New York, NY: American Heart Association; 1973. pp. 1–35.
14. Kannel WB, McGee D, Gordon T. A general cardiovascular risk profile: the Framingham study. Am J Cardiol. 1976;38:46–51. [PubMed]
15. Gordon T, Kannel WB. Multiple risk functions for predicting coronary heart disease: the concepts, accuracy, and application. Am Heart J. 1982;103:1031–1039. [PubMed]
16. D’Agostino RB, Lee ML, Belanger AJ, Cupples LA, Anderson K, Kannel WB. Relation of pooled logistic regression to time dependent Cox regression analysis: the Framingham Heart Study. Stat Med. 1990;9:1501–1515. [PubMed]
17. Anderson KM, Wilson PW, Odell PM, Kannel WB. An updated coronary risk profile: a statement for health professionals. Circulation. 1991;83:356–362. [PubMed]
18. Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97:1837–1847. [PubMed]
19. National Cholesterol Education Program: Second Report of the Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adults Treatment Panel II) Circulation. 1994;89:1333–1445. [PubMed]
20. The fifth report of the Joint National Committee on Detection, Evaluation, and Treatment of High Blood Pressure (JNC V) Arch Intern Med. 1993;153:154–183. [PubMed]
21. D’Agostino RB, Sr, Grundy S, Sullivan LM, et al. CHD Risk Predication Group Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic group investigation. JAMA. 2001;286:180–187. [PubMed]
22. Wolf PA, D’Agostino RB, Belanger AJ, Kannel WB. Probability of stroke: a risk profile from the Framingham study. Stroke. 1991;22:312–318. [PubMed]
23. D’Agostino RB, Wolf PA, Belanger AJ, Kannel WB. Stoke risk profile: adjustment for antihypertensive medication. The Framingham study. Stroke. 1994;25:40–43. [PubMed]
24. D’Agostino RB, Russell MW, Huse DM, et al. Primary and subsequent coronary risk appraisal: new results from the Framingham study. Am Heart J. 2000;139:272–281. [PubMed]
25. Murabito JM, D’Agostino RB, Sibershatz H, Wilson WF. Intermittent claudication: a risk profile from the Framingham study. Circulation. 1997;96:44–49. [PubMed]
26. Kannel WB, D’Agostino RB, Silbershatz H, Belanger AJ, Wilson PW, Levy D. Profile for estimating risk of heart failure. Arch Intern Med. 1999;159:1197–1204. [PubMed]
27. Wang TJ, Massaro JM, Levy D, et al. A risk score for predicting stroke or death in individuals with new-onset atrial fibrillation in the community: the Framingham Heart Study. JAMA. 2003;290:1049–1056. [PubMed]
28. Schnabel RB, Sullivan LM, Levy D, et al. Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort. Lancet. 2009;373:739–745. [PMC free article] [PubMed]
29. Executive Summary of the Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol In Adults (Adult Treatment Panel III) JAMA. 2001;285:2486–2497. [PubMed]
30. Grundy SM. Primary prevention of coronary heart disease: integrating risk assessment with intervention. Circulation. 1999;100:988–998. [PubMed]
31. Sullivan LM, Massaro JM, D’Agostino RB., Sr Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med. 2004;23:1631–1660. [PubMed]
32. D’Agostino RB, Sr, Vasan RS, Pencina MJ, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117:743–753. [PubMed]
33. Abbott RD, McGee D. National Heart, Lung, and Blood Institute, U.S. Department of Commerce. The Framingham Study: An Epidemiological Investigation of Cardiovascular Disease. Springfield, VA: National Technical Information Service; 1987. National Technical Information Service. The probability of developing certain cardiovascular diseases in eight years specified values of some characteristics. sec. 37.
34. Vine DL, Hastings GE. Ischaemic heart disease and cholesterol: absolute risk more informative than relative risk. BMJ. 1994;308:1040–1041. [PMC free article] [PubMed]
35. Jackson R. Guidelines on preventing cardiovascular disease in clinical practice. BMJ. 2000;320:656–661. [PMC free article] [PubMed]
36. Jackson R. Updated New Zealand cardiovascular disease risk-benefit prediction guidelines. BMJ. 2000;320:709–710. [PMC free article] [PubMed]
37. Harrell FE, Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387. [PubMed]
38. D’Agostino RB, Nam BH. Evaluation of the performance of survival analysis models: discrimination and calibration measures. In: Balakrishnan N, Rao CR, editors. Handbook of Statistics. Vol. 23. New York, NY: Elsevier; 2003. pp. 1–25.
39. Pencina MJ, D’Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med. 2004;23:2109–2123. [PubMed]
40. Pencina MJ, D’Agostino RB, Sr, Song L. Quantifying discrimination of Framingham risk functions with different survival C statistics. Stat Med. 2012;31:1543–1553. [PMC free article] [PubMed]
41. Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd. New York, NY: John Wiley and Sons; 2000.
42. Steyerberg EW, Harrell FE, Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–781. [PubMed]
43. Kannel WB, D’Agostino RB, Sullivan L, Wilson PW. Concepts and usefulness of cardiovascular risk profiles. Am Heart J. 2004;148:16–26. [PubMed]
44. Pooling Project Research Group. Relationship of blood pressure, serum cholesterol, smoking habit, relative weight, and ECG abnormalities to the incidence of major coronary events: final report of the pooling project. J Chronic Dis. 1978;31:201–306. [PubMed]
45. Brand RJ, Rosenman RH, Sholtz RI, Friedman M. Multivariate prediction of coronary heart disease in Western Collaboration Group Study compared to findings of the Framingham study. Circulation. 1976;53:348–355. [PubMed]
46. Leaverton PE, Sorlie PD, Kleinnman JC, et al. Representativeness of the Framingham risk model for coronary heart disease mortality: a comparison with a national cohort study. J Chronic Dis. 1987;40:775–784. [PubMed]
47. McGee D, Gordon T. The results of the Framingham Study applied to four other US-based studies of cardiovascular disease. In: Kannel WB, Gordon T, editors. The Framingham Study: An epidemiological Investigation of Cardiovascular Disease. Bethesda, MD: U.S. Government Printing Office; 1976. U.S. Dept of Health Education and Welfare Publication No. 76-1083. sec. 31.
48. Grundy SM, D’Agostino RB, Sr, Mosca L, et al. Cardiovascular risk assessment based on US cohort studies: findings from a National Heart, Lung, and Blood Institute workshop. Circulation. 2001;104:491–496. [PubMed]
49. Empana JP, Ducimetière P, Arveiler D, et al. for the PRIME Study Group. Are the Framingham and PROCAM coronary heart disease risk functions applicable to different Europeans populations? The PRIME study. Eur Heart J. 2003;24:1903–1911. [PubMed]
50. Scheltens T, Verschuren WM, Boshuizen HC, et al. Estimation of cardiovascular risk: a comparison between Framingham and the SCORE model in people under 60 years of age. Eur J Cardiovasc Prev Rehabil. 2008;15:562–566. [PubMed]
51. Aktas MK, Ozduran V, Pothier CE, Lang R, Lauer MS. Global risk scores and exercise testing for predicting all-cause mortality in preventive medicine program. JAMA. 2004;292:1462–1468. [PubMed]
52. Barzi F, Patel A, Gu D, et al. for the Asia Pacific Cohort Studies Collaboration. Cardiovascular risk prediction tools for populations in Asia. J Epidemiol Community Health. 2007;61:115–121. [PMC free article] [PubMed]
53. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Brindle P. Performance of the QRISK cardiovascular risk prediction algorithm in an independent UK sample of patients from general practice: a validation study. Heart. 2008;94:34–39. [PubMed]
54. Simmons RK, Sharp S, Boekholdt SM, et al. Evaluation of the Framingham risk score in the European Prospective Investigation of Cancer-Norfolk cohort: does adding glycated hemoglobin improve the prediction of coronary heart disease events? Arch Intern Med. 2008;168:1209–1216. [PubMed]
55. Brindle PM, McConnachie A, Upton MN, Hart CL, Davey Smith G, Watt GC. The accuracy of the Framingham risk-score in different socioeconomic groups: a prospective study. Br J Gen Pract. 2005;55:838–845. [PMC free article] [PubMed]
56. May M, Lawlor DA, Brindle P, Patel R, Ebrahim S. Cardiovascular disease risk assessment in older women: can we improve on Framingham? British Women’s Heart and Health prospective cohort study. Heart. 2006;92:1396–1401. [PMC free article] [PubMed]
57. Ferrario M, Chiodini P, Chambless LE, et al. for the CUORE Project Research Group. Prediction of coronary events in a low incidence population. Assessing accuracy of the CUORE Cohort Study prediction equation. Int J Epidemiol. 2005;34:413–421. [PubMed]
58. Marrugat J, D’Agostino R, Sullivan L, et al. An adaptation of the Framingham coronary heart disease risk function to European Mediterranean areas. J Epidemiol Community Health. 2003;57:634–638. [PMC free article] [PubMed]
59. Liu J, Hong Y, D’Agostino RB, Sr, et al. Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study. JAMA. 2004;291:2591–2599. [PubMed]
60. Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. J Am Stat Assoc. 1958;53:457–481.
61. D’Agostino RB, Griffith JL, Schmidt CH, Terrin N. Proceedings of the Biometrics Section. Alexandria, VA: American Statistical Association, Biometrics section; 1997. Measures for evaluating model performance; pp. 253–258.
62. Brindle P, May M, Gill F, et al. Primary prevention of cardiovascular disease: a web-based risk score for seven British black and minority ethnic groups. Heart. 2006;92:1595–1602. [PMC free article] [PubMed]
63. Whittemore AS. Evaluating health risk models. Stat Med. 2010;29:2438–2452. [PMC free article] [PubMed]
64. Risk Score Profiles Framingham Heart Study. [Accessed November 30, 2011]; Available at: http://www framinghamheartstudy.org/risk/index.html.
65. Koenig W. Haemostatic risk factors for cardiovascular disease. Eur Heart J. 1998;19(Suppl C):C39–C43. [PubMed]
66. Ridker PM, Cushman M, Stampfer MJ, Tracy RP, Hennekens CH. Inflammation, aspirin, and the risk of cardiovascular disease in apparently healthy men. N Engl J Med. 1997;336:973–979. [PubMed]
67. Welch GN, Loscalzo J. Homocysteine and atherothrombosis. N Engl J Med. 1998;338:1042–1050. [PubMed]
68. Ridker PM, Rifai N, Rose L, Buring JE, Cook NR. Comparison of C-reactive protein and low-density lipoprotein cholesterol levels in prediction of first cardiovascular events. N Engl J Med. 2002;347:1557–1565. [PubMed]
69. Rutter MK, Meigs JB, Sullivan LM, D’Agostino RB, Sr, Wilson PW. C-reactive protein, the metabolic syndrome, and prediction of cardiovascular events in the Framingham Offspring Study. Circulation. 2004;110:380–385. [PubMed]
70. Wilson PW, Nam BH, Pencina M, D’Agostino RB, Sr, Benjamin EJ, O’Donnell CJ. C-reactive protein and risk of cardiovascular disease in men and women from the Framingham Heart Study. Arch Intern Med. 2005;165:2473–2478. [PubMed]
71. Detrano R, Guerci AD, Carr JJ, et al. Coronary calcium as a predictor of coronary events in four racial or ethnic groups. N Engl J Med. 2008;358:1336–1345. [PubMed]
72. Lloyd-Jones DM, Nam BH, D’Agostino RB, Sr, et al. Parental cardiovascular disease as a risk factor for cardiovascular disease in middle-aged adults: a prospective study of parents and offspring. JAMA. 2004;291:2204–2211. [PubMed]
73. Folsom AR, Chambless LE, Ballantyne CM, et al. An assessment of incremental coronary risk prediction using C-reactive protein and other novel risk markers: the Atherosclerosis Risk in Communities Study. Arch Intern Med. 2006;166:1368–1373. [PubMed]
74. Wang TJ, Gona P, Larson MG, et al. Multiple biomarkers for the prediction of first major cardiovascular events and death. N Engl J Med. 2006;355:2631–2639. [PubMed]
75. Polak JF, Pencina MJ, Pencina KM, O’Donnell CJ, Wolf PA, D’Agostino RB., Sr Carotid-wall intima-media thickness and cardiovascular events. N Engl J Med. 2011;365:213–221. [PMC free article] [PubMed]
76. Thanassoulis G, Peloso GM, Pencina MJ, et al. A genetic risk score is associated with incident cardiovascular disease and coronary artery calcium: the Framingham Heart Study. Circ Cardiovasc Genet. 2012;5:113–121. [PMC free article] [PubMed]
77. Demler OV, Pencina MJ, D’Agostino RB., Sr Equivalence of improvement in area under ROC curve and linear discriminant analysis coefficient under the assumption of normality. Stat Med. 2011;30:1410–1418. [PubMed]
78. Demler OV, Pencina MJ, D’Agostino RB., Sr Misuse of DeLong test to compare AUCs for nested models. Stat Med. 2012;31:2577–2587. [PMC free article] [PubMed]
79. Pencina MJ, D’Agostino RB, Sr, D’Agostino RB, Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157–172. [PubMed]
80. Pencina MJ, D’Agostino RB, Sr, D’Agostino RB, Jr, Vasan RS. Comments on integrated discrimination and net reclassification improvements—practical advice. Stat Med. 2008;27:207–212.
81. Pencina MJ, D’Agostino RB, Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11–21. [PMC free article] [PubMed]
82. Pencina MJ, D’Agostino RB, Demler OV. Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med. 2012;31:101–113. [PMC free article] [PubMed]
83. Pencina MJ, D’Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176:473–481. [PMC free article] [PubMed]
84. Seshadri S, Wolf PA, Beiser A, et al. Lifetime risk of dementia and Alzheimer’s disease: the impact of mortality on risk estimates in the Framingham study. Neurology. 1997;49:1498–1504. [PubMed]
85. Beiser A, D’Agostino RB, Sr, Seshadri S, Sullivan LM, Wolf PA. Computing estimates of incidence, including lifetime risk: Alzheimer’s disease in the Framingham Study. The practical Incidence Estimators (PIE) macro. Stat Med. 2002;19:1495–1522. [PubMed]
86. Lloyd-Jones DM, Larson MG, Beiser A, Levy D. Lifetime risk of developing coronary heart disease. Lancet. 1999;353:89–92. [PubMed]
87. Lloyd-Jones DM, Larson MG, Leip EP, et al. Lifetime risk for developing congestive heart failure: the Framingham Heart Study. Circulation. 2002;106:3068–3072. [PubMed]
88. Lloyd-Jones DM, Wang TJ, Leip EP, et al. Lifetime risk for development of atrial fibrillation: the Framingham Heart Study. Circulation. 2004;110:1042–1046. [PubMed]
89. Lloyd-Jones DM, Wilson PW, Larson MG, et al. Lifetime risk for coronary heart disease by cholesterol levels at selected age. Arch Intern Med. 2003;163:1966–1972. [PubMed]
90. Pencina MJ, D’Agostino RB, Sr, Larson MG, Massaro JM, Vasan RS. Predicting the 30-year risk of cardiovascular disease: the Framingham Heart Study. Circulation. 2009;119:3078–3084. [PMC free article] [PubMed]
91. Lloyd-Jones DM. Cardiovascular risk prediction: basic concepts, current status, and future directions. Circulation. 2010;121:1768–1777. [PubMed]
92. Conroy RM, Pyörälä K, Fitzgerald AP, et al. for the SCORE Project Group. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003;24:987–1003. [PubMed]
93. Assmann G, Cullen P, Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the Prospective Cardiovascular Münster (PROCAM) study. Circulation. 2002;105:310–315. [PubMed]
94. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ. 2007;335:136. [PMC free article] [PubMed]
95. Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA. 2007;297:611–619. [PubMed]
96. Ridker PM, Paytner NP, Rifai N, Gaziano JM, Cook NR. C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men. Circulation. 2008;118:2243–2251. [PMC free article] [PubMed]
97. Cooney MT, Dudina A, D’Agostino R, Graham IM. Cardiovascular risk-estimation system in primary prevention: do they differ? Do they make a difference? Can we see the future? Circulation. 2010;122:300–310. [PubMed]
98. The ARIC Investigators. The Atherosclerosis Risk in Communities (ARIC) Study: design and objective. Am J Epidemiol. 1989;129:687–702. [PubMed]
99. Fried LP, Borhani NO, Enright P, et al. The Cardiovascular Health Study: design and rationale. Ann Epidemiol. 1991;1:263–276. [PubMed]
100. Friedman GD, Cutter GR, Donahue RP, et al. CARDIA: study design, recruitment, and some characteristics of the examined subjects. J Clin Epidemiol. 1988;41:1105–1116. [PubMed]
101. Graham IM, D’Agostino RB. Therapeutic Strategies in Cardiovascular Risk. Oxford, UK: Atlas Medical Publishing Ltd; 2008.
102. Greenland S, et al. The need for reorientation toward cost-effective prediction: comments on “Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification, beyond” by MJPencina, Statistics in Medicine (DOI: 101002/sim.2929) Stat Med. 2008;27:199–206. [PubMed]
103. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–138. [PMC free article] [PubMed]
104. Khalili D, Hadaegh F, Soori H, Steyerberg EW, Bozorgmanesh M, Azizi F. Clinical usefulness of the Framingham cardiovascular risk profile beyond its statistical performance: the Tehran Lipid and Glucose Study. Am J Epidemiol. 2012;176:177–186. [PubMed]
105. D’Agostino RB, Sr, Pencina MJ. Invited commentary: clinical usefulness of the Framingham cardiovascular risk profile beyond its statistical performance. Am J Epidemiol. 2012;176:187–189. [PubMed]