|Home | About | Journals | Submit | Contact Us | Français|
To examine the features of available Framingham-based risk calculation tools and review their accuracy and feasibility in clinical practice.
medline, 1966–April 2003, and the google search engine on the Internet.
We included risk calculation tools that used the Framingham risk equations to generate a global coronary heart disease (CHD) risk. To determine tool accuracy, we reviewed all articles that compared the performance of various Framingham-based risk tools to that of the continuous Framingham risk equations. To determine the feasibility of tool use in clinical practice, we reviewed articles on the availability of the risk factor information required for risk calculation, subjective preference for 1 risk calculator over another, or subjective ease of use.
Two reviewers independently reviewed the results of the literature search, all websites, and abstracted all articles for relevant information.
Multiple CHD risk calculation tools are available, including risk charts and computerized calculators for personal digital assistants, personal computers, and web-based use. Most are easy to use and available without cost. They require information on age, smoking status, blood pressure, total and HDL cholesterol, and the presence or absence of diabetes. Compared to the full Framingham equations, accuracy for identifying patients at increased risk was generally quite high. Data on the feasibility of tool use was limited.
Several easy-to-use tools are available for estimating patients' CHD risk. Use of such tools could facilitate better decision making about interventions for primary prevention of CHD, but further research about their actual effect on clinical practice and patient outcomes is required.
Drs. Sheridan and Pignone have participated in the development of Heart-to-Heart, one of the risk tools evaluated within. They have also received speaking and consulting fees from Bayer, Inc. Bayer, Inc. has licensed the Heart-to-Heart tool.
Clinical practice guidelines recommend that providers and patients base treatment decisions regarding coronary heart disease (CHD) prevention on assessment of underlying global CHD risk.1–4 In addition, the American Heart Association has recommended that adults aged 40 and older with no previous history of cardiovascular disease have their global CHD risk calculated every 5 years.5 To implement these guidelines in clinical practice, providers need an accurate and feasible means of calculating global CHD risk.
Previous research has shown that providers do not accurately estimate the risk of CHD events on their own.6–11 Fortunately, multivariate risk prediction equations have been developed to better estimate CHD risk. These equations have been derived from large prospective cohort studies or randomized trials12–23 and estimate a patient's risk of having a CHD event over 5 to 10 years. They provide better estimates of CHD risk than either assessment of single risk factors or simple counting of multiple risk factors and appear to be more cost effective in guiding CHD treatment decisions.24 Some of the available risk equations, however, have limitations: they include relatively few risk factors; are derived from truncated middle-aged or male-only populations; use logistic regression models that require fixed follow-up periods (e.g., 10 years); treat events occurring at 1 year the same as events occurring at 5 or 10 years; and have been prospectively validated in limited populations.
Among the various risk prediction equations, those derived from the Framingham Heart Study are most commonly recommended for use in the United States.13 These equations calculate the absolute risk of CHD events for patients with no known previous history of CHD, stroke, or peripheral vascular disease (primary prevention). Compared to other risk equations, the Framingham risk equations have favorable characteristics: they were developed in a large prospective cohort of U.S. men and women aged 30 to 74 years, have been subsequently validated in multiple diverse populations,17 and discriminate well among those who will have a CHD event and those who won't.21,25–28 In general, the Framingham equations also predict the degree of risk well in middle-aged white and African-American adults, although hypertension is somewhat underweighted as a risk factor in African Americans (particularly for women)27,28 and the risk associated with diabetes mellitus is undervalued.13,29,30 The equations predict the degree of risk less well in men and women younger than age 30 or over age 65, Japanese-American men, Hispanic men, and Native-American women.13,27,28 They also are less precise in patients with diabetes, severe hypertension, or left ventricular hypertrophy because fewer numbers of participants in the original Framingham cohort had these risk factors.13
For use in clinical practice, the Framingham equations have been operationalized into several risk assessment “tools.” Common formats of available risk tools include risk charts (simple tables or wall charts) and electronic calculators, which are available as stand-alone applications for personal computers or personal digital assistants, and web-based tools. We sought to review available CHD risk calculation tools based on Framingham equations to help guide providers in selecting the best tools for their practices.
To identify Framingham-based CHD risk calculation tools and review their accuracy and feasibility in clinical practice, we conducted a search of medline 1966–April 2003 using the MeSH terms coronary heart disease and risk assessment. To identify web-based tools that are readily available to the clinician, we also performed an Internet search in April 2002 using a popular search engine, google, and the search term “cardiac risk calculator.” Finally, we used our own literature files, and hand-checking of identified bibliographies and web links to identify other risk tools or articles evaluating risk assessment tools.
To identify available CHD risk calculation tools, we included articles and websites that used the Framingham risk equations to generate a global CHD risk, expressed either as the proportion of similar patients who would have a CHD event over a defined time period or as the movement of a patient across a predefined treatment threshold. We excluded articles and websites that used non-Framingham risk equations, did not specify the equation used for calculation, were designed for secondary prevention, did not clearly define the calculated risk outcome, or calculated risk using nontraditional risk factors such as blood type or measures of psychological stress.
To determine the accuracy of CHD risk tools, we included articles that compared the performance of various Framingham-based risk tools to that of the continuous Framingham equation in clinical practice. We included articles that tabulated the sensitivity and specificity of the risk tools or provided enough information that these could be calculated.
Because we wanted to focus on tools available for clinical practice, we excluded articles that compared the discriminatory and predictive abilities of continuous Framingham equations including different risk factors or prospectively examined the continuous Framingham equations in large epidemiological study populations. We also excluded articles that examined the accuracy of non-Framingham-based risk tools, used a gold standard other than the continuous Framingham model, or that reported only the difference in accuracy among various provider groups.
To determine the feasibility of risk tools in clinical practice, we included articles that provided information on the availability of the risk factor information required for risk calculation, subjective preference for one risk calculator over another, or subjective ease of use of the various risk calculators.
Two of us independently reviewed the results of the literature and web searches (MP, SS) to determine article and website inclusion. We then abstracted relevant information from included articles and websites into tables for analysis (CM, MP, SS). Disagreements were resolved by discussion among team members.
We categorized the risk tools into 2 main groups: 1) risk charts (usually printed); and 2) electronic calculators, including computer programs for personal digital assistants (handheld PDAs), spreadsheet programs designed to run on personal computers, and web-based risk calculators. We then reviewed each tool to determine the required input and to characterize its output.
For studies reporting on the accuracy and feasibility of various risk calculators, we abstracted information that we felt would impact the quality of the accuracy estimates reported and their applicability to clinical practice. Specifically, we abstracted information on the identity of the risk scorer, whether they were blinded to the gold standard risk assessment, what patient population was used for risk assessment, whether all necessary patient data were available for the risk calculation, and what reference cutpoint was used to distinguish high versus low CHD risk. We made no attempt to combine these factors into an overall quality score.
Our medline search identified 1,306 articles on risk assessment for coronary heart disease and our final Internet search, conducted on April 28, 2002, identified 3,690 websites. After review of abstracts and potentially relevant articles, we included 8 articles describing Framingham-based risk calculation tools and 7 articles providing information on the accuracy and feasibility of the tools. Two independent reviewers additionally reviewed the 100 websites rated most relevant to our search by the google search engine, including 10 sites described in this report. We did not include websites with required member log-in (N = 2), nonfunctional links (N = 3), no CHD risk calculator (N = 28), non-Framingham-based calculators (N = 7), calculators including nontraditional risk factors (N = 2), calculators with unspecified risk equations (N = 5), or calculators with undefined outcomes (N = 3). Forty of the 100 sites were repeat references.
Table 1 provides a representative, but not exhaustive, sample of available tools. Tools have a variety of formats including risk charts (simple tables or wall charts) and electronic calculators, which are available as stand-alone or web-based applications for personal computers, or as stand-alone applications for personal digital assistants. All tools require information on age, gender, total cholesterol, systolic blood pressure, and smoking status for risk calculation; most also include diabetes, assessed as a yes/no answer, and high-density lipoprotein (HDL) cholesterol. Some tools using older versions of the Framingham equations also prompt input on the presence of left ventricular hypertrophy (LVH) on electrocardiogram, although lack of this information does not preclude risk calculation.
The output of the risk tools we reviewed is diverse. CHD events are defined alternately as a composite of myocardial infarction (nonfatal or fatal) and sudden death or as new-onset stable angina, unstable angina (called “coronary insufficiency” in the Framingham study), myocardial infarction, and sudden death. Some tools (e.g., Sheffield tables, Joint British charts, and Joint European charts) estimate the risk of CHD events alone, while others (e.g., New Zealand tables) give risks for CHD events and for stroke. One tool (Birmingham Heartlands Calculator) also included peripheral vascular disease as an outcome.
The presentation of CHD risk (see Fig. 1) is generally in numeric or graphic terms, with few tools including written explanation of the results. Some tools (e.g., New Zealand tables) give a point estimate of risk, whereas others provide a range of risks or simply state whether a predefined treatment threshold to initiate therapy had been exceeded (e.g., Sheffield tables). Most tools provide either a comparison to the risk of an individual of the same age or gender who has no risk factors or to an individual with “average” risk factors. Many also provide a qualitative description, such as high or low risk. A minority provide treatment advice or links to evidence-based treatment guidelines.
Several different risk charts are available in print form or from the Internet. The charts (or tables) generally fall into 2 types: 1 type assigns points to various levels of each risk factor and then assigns a specific risk for the total score obtained after summing the individual scores for each risk factor (e.g., Categorical Framingham tables). The second type arrays information in various combinations of columns and rows either to allow a specific risk to be read from the chart (e.g., New Zealand tables) or to reach a treatment decision given a predefined threshold for treatment (e.g., Sheffield tables). The main advantage of tables and charts is that they do not require a computer for use. They can be downloaded, printed, or photocopied and used in any setting. The main downsides are that they may be difficult or time consuming to use at first and that they are not as accurate or precise as some of the spreadsheet or web-based calculators described below.
Currently, few risk tools are available for handheld computers or PDAs (e.g., Stat Cardiac Risk, the National Cholesterol Education Program Palm Calculator, FramPlus, and Heart-to-Heart). Based on the updated Framingham risk equations, these programs use categorical classification of risk factors to estimate the 10-year risk of CHD.20 Because they use ranges, they are slightly less precise than some of the spreadsheet calculators that use exact values. On the positive side, they are portable and very easy and fast to use and can be shared with other PDA users by simply “beaming” the program via the infrared port.
Spreadsheet-based calculators make the Framingham equations available in a computer program such as Microsoft Excel (Microsoft Corporation, Redmond, WA). They require that the spreadsheet program be installed on each computer that is to be used for calculating risk. One commercial product, the BMJ CardioRisk Manager, adds the capability of producing more sophisticated reports (including a letter to send results to the patient) and can archive results. It also includes a “slider bar” to allow patients and providers to see the projected effect of treatment on CHD outcomes. The expected effect of treatment is demonstrated by recalculating risk using posttreatment risk factor levels rather than by applying the best evidence about expected risk reduction to baseline calculated risk. This may be misleading because changes in risk levels with treatment do not produce the same degree of risk reduction as would be predicted from observational studies. Another calculator, the Birmingham Heartlands Calculator, does estimate the effect of treatment, by applying evidence about expected risk reduction.
Several web-based risk calculators are available. They require that the user have Internet access, but no local software is needed other than a web browser. They can only be used effectively in practice settings that have continuous access to the Internet; establishing a dial-up connection each time the program is used is impractical. Web-based calculators generally use the full Framingham equation. Results can be printed from the browser to be placed in the medical record. Additionally, a few tools (the risk calculator from the University of Edinburgh (www.cardiacrisk.org.uk) and the Heart-to- Heart tool (www.med-decisions.com) offer the option to print individualized evidence-based treatment advice for patients.
We found 6 studies that compared the relative accuracy of various risk prediction charts or tables with full Framingham risk equations (Table 2).23,31–35 Because electronic calculators use the full Framingham equation or tally scores from charts or tables, we found no studies separately examining these tools. In the studies we identified, risk assessors calculated CHD risk from data obtained from patient charts, physical examinations, and laboratory assessments; the standard for comparison was the full Framingham equation. In 2 studies,33,34 risk assessors were practicing clinicians with no prior knowledge of the results of the full Framingham calculation. In the remaining 4 studies, the risk assessors were computer operators, medical students, or other observers.23,31,32,35 We could not tell whether these risk assessors had prior knowledge of the Framingham equations or the risk calculation tools or whether they received any special training in their use.
Table 3 gives reported sensitivity and specificity values for the most commonly used risk assessment tools from the 6 studies. Although all studies used full Framingham equations as the reference standard, different cutpoints were sometimes used to define high-risk status or thresholds for treatment. We include the results for the most common cutpoints here. In general, the tools displayed good to excellent sensitivity and specificity for detection of patients with increased CHD risk. Only the Canadian tool had poor accuracy in predicting a risk of greater than 3% per year; it performed much better at a reference standard cutpoint of 1.5% per year (sensitivity 95%−98%).23,31 We make no comparisons of sensitivity and specificity findings across studies due to the varying numbers of indeterminate assessments, different reference standard cutpoints, and diverse study populations.
The proportion of insufficient data available to complete the Framingham calculations varied from 5% to 49% across studies, including 11%το 49% of cases in the 1 study that relied on randomly selected patient charts for risk calculations.33 When data were missing, none of the study authors used mean risk factor values to estimate risk. The most common reason for inability to assess patient risk was missing HDL cholesterol values. Thus, risk assessments that do not require HDL values (Joint European charts) were completed more often than those that rely on HDL values (Joint British tables, New Zealand tables).
McManus and colleagues33 examined the reliability of the risk calculations of general practitioners and practice nurses. They found κ values ranging from 0.47 to 0.58, suggesting moderate reliability. In the same study, however, risk assessments were inappropriately completed for 40% of patients with known coronary heart disease, even though such patients can be classified as high-risk based on disease history alone.
We found 1 additional Scottish study that compared the calculations from 3 risk assessment tools (New Zealand table, old Sheffield table, and Joint British chart) with each other, rather than with full Framingham equation estimates, and provided information about the feasibility of using these tools in clinical practice.36 In this study, a self-nominated general practitioner and nurse from each of 37 general practices completed risk assessments on a set of 12 case histories that reflected varying levels of CHD risk. Doctors and nurses preferred New Zealand tables and Joint British charts over the Sheffield tables and found them easier to use. Doctors generally scored case histories with similar risks using the 3 different risk tools, while accuracy among nurses was significantly poorer with the Sheffield table compared to the 2 other tools.
Policy-making bodies increasingly agree that the most efficient and effective clinical CHD prevention requires a global assessment of CHD risk.5,13 Fortunately, a variety of user-friendly tools based on the Framingham equation are available to help clinicians perform CHD risk assessment for patients with no known history of cardiovascular disease. Our review suggests that, in general, the categorical charts and tables derived from the Framingham equation are accurate and feasible for use in clinical settings and can be used in lieu of the continuous Framingham calculators when necessitated by the clinical environment. This supports findings by chart developers who report similar discriminatory ability between their charts and the full Framingham equations.20 Some features of the computer or PDA-based tools, however, may make them a better choice for providers with access to such devices.
In deciding among available tools, providers may wish to choose tools that provide risk information in a format that can be used with current guidelines for risk reduction (see Table 4). For instance, to allow risk-based decision making about lipid-lowering therapy, providers need a tool that allows stratification of risk into <10%, 10%−20%, and >20%.1,37 All of the spreadsheets, PDA, and web-based calculators have this capability because all use the continuous Framingham equations or the original Framingham categorical charts. Many of the risk charts also have this capability; the notable exception is the Modified Sheffield table, which uses only 15% and 30% cutoffs. To adhere to evidence-based guidelines on aspirin use, providers need a tool with finer gradations of risk because the risk/benefit ratio for aspirin use transitions from helpful to harmful at a 10-year risk of CHD events between 3% to 5% and 10%.2,5 This again reduces the number of useful risk charts, but still allows many acceptable options. At present, it is unclear how providers should address risk calculation in patients with diabetes. The National Cholesterol Education Program and the American Heart Association currently recommend that physicians treat patients with diabetes as though they have a risk for subsequent CHD events that is equivalent to that in patients with known CHD.1,13 In accordance with this, they have recommended that their Framingham risk calculators be used only in patients without diabetes. At present, however, we are unaware of direct evidence that suggests this strategy is more effective than relying on calculated risk assessment, and many calculators continue to request input of diabetes status for risk calculations.
In choosing which risk tool to use, providers should consider their practice environment and who will be performing the risk assessments. Providers who have access to a computer with an available spreadsheet program or dedicated high-speed Internet access line should consider spreadsheet and web-based programs for risk calculation. These tools allow calculation of fine gradations of risk, frequently provide comparisons to individuals with low risk (e.g., BMJ Cardiorisk Manager, Birmingham Heartlands Calculator, National Cholesterol Education Program Risk Calculator, RiskCalculator from the Center for Cardiovascular Sciences at the University of Edinburgh, Healing Hearts Risk Calculator, Medical-decisions.com calculator, and American Heart Association Calculator), and, in some cases, provide targeted advice on treatment and allow exploration of the effects of treatment on calculated risk (e.g. BMJ Cardiorisk Manager, Birmingham Heartlands Calculator, Heart-to-Heart Calculator). Additionally, at least one of these tools (Heart-to-Heart Calculator) is targeted to patients and can be used independently of the clinician visit. For providers who do not have access to these tools, current PDA tools and risk charts offer an acceptable option.
Some providers may find that a combination of products is most useful, particularly if the outcome of interest varies according to patient concerns. Most tools provide information on the combined risk of stable and unstable angina, myocardial infarction, and CHD death. Some tools, however, report only the risk of myocardial infarction and CHD death; these tools will produce smaller numeric estimates of risk than tools that also include angina. The current NCEP risk calculator, for example, uses a set of newly revised Framingham equations that only predict the risk of myocardial infarction and CHD death. To our knowledge, these equations have not been published in the peer-reviewed literature. Other tools allow calculation of all CVD events by adding stroke outcomes (e.g., New Zealand Risk Table, British Cardiac Risk Assessor, BMJ Cardiorisk Manager, Risk Calculator from the Center for Cardiovascular Sciences at the University of Edinburgh) or by allowing independent calculation of the risk of stroke and peripheral vascular disease (e.g., Birmingham Heartlands Calculator).
In addition to choosing which type of risk tool to use, providers must ensure that they have sufficient information to complete the risk assessment. Some information, such as age, smoking status, and presence or absence of diabetes, can be obtained by interview at the time the risk calculation is performed. Other information, such as blood pressure, cholesterol levels, and presence or absence of left ventricular hypertrophy on electrocardiogram must be obtained prior to risk calculation.
Our review identified several limitations among the available Framingham tools. First, existing tools do not predict risk beyond 12 years. This is a limitation imposed by the published data available from the Framingham Heart Study. Although Framingham investigators have published data on the lifetime risk of developing coronary heart disease,38 they have not incorporated lifetime risk into tools for clinical risk estimation. Presentation of lifetime risks may have different effects on perceived threat and motivation to undertake risk-reducing behavior for some patients, particularly younger ones, who are making longer-term prevention decisions,38–41 although to date this has not been empirically studied. Second, none of the tools specify how electrocardiographic LVH is to be defined, although available evidence suggests that LVH with repolarization abnormality (strain pattern) provides the best predictive ability, and LVH by voltage criteria alone is not associated with clearly increased risk.42 Third, none of the tools provide confidence intervals around risk estimates. Their absence may convey a false sense of precision. Finally, most tools do not provide accurate information about the benefits and adverse effects of risk-lowering interventions, which may limit their clinical utility.
Aside from the limitations of the tools, we acknowledge the limitations of the Framingham equations themselves. Although the Framingham equations predict the degree of risk well in white and African-American men and women between the ages of 30 and 65 in the United States, they predict the degree of risk less well in non-U.S. populations, certain U.S. ethnic groups (Japanese men, Hispanic men, and Native-American women), men and women younger than age 30 or older than age 65, and diabetic persons.25,27,28 One approach to the Framingham equations' limits is to recalibrate the tool for use in designated target populations.28 At present, we are not aware of any Framingham-based risk calculation tools that have attempted to do this.
The current Framingham equations have additionally attempted to balance accuracy and feasibility43 and hence have limited the number of risk factors required for risk estimation. They do not include the following established and potential risk factors, which may be of interest: blood glucose level, hemoglobin A1C, triglycerides, lipoprotein A, small dense low-density lipoprotein particles, homocysteine, c-reactive protein, microalbuminuria, coagulation factors, weight or body mass index, physical activity, and family history of premature cardiovascular disease. The effect of adding additional risk factors to risk calculation tools has been little studied.
As of April 2003, our searches of the medical literature also show that the effect of risk calculators on clinical practice and outcomes has not been well studied. Two studies6,44 suggest that providing physicians with computerized risk calculators has had little impact on CHD risk. These studies, however, provided no link to evidence-based guidelines and had important methodological limitations including high attrition rates44 and use in populations who already have existing CHD.6 A third study, in which researchers alternately wrote patient risk scores on the front of patient charts or not, also suggests the limited effects of providing physicians with only risk estimates.45 Whether calculating and communicating global CHD risk to patients affects their willingness or ability to change their lifestyle and use preventive medications, such as aspirin, antihypertensive drugs, or cholesterol-lowering medication, has not been well studied. Although a recent pilot study46 testing the combined effects of a self-guided workbook and physician visit on global CHD risk reported that 68% of users planned to make a variety of interventions on their risk as a result of using the book, traditional CHD risk appraisal has had only modest impact on actual patient behavior in the areas of diet and exercise.47–49 One recent study has shown reductions in CHD risk, body mass index, and cholesterol levels at 5 years follow-up in intervention groups that received CHD risk appraisal with or without physician consultation,50 but conclusions were limited by high attrition rates and poor participation in follow-up consultations throughout most of the study. Further research is still needed.
Research should also determine whether the inclusion of newer risk factors for CHD (i.e., lipoprotein a, homocysteine, micro-albuminuria, or c-reactive protein), or noninvasive measures of atherosclerosis, such as electron-beam computerized tomography (EBCT) or carotid Doppler ultrasound, improves risk assessment and leads to better use of CHD risk-reducing treatments. Some have suggested that these novel risk factors may be best used to modify the pretest probability estimate from the Framingham risk score, particularly for those with intermediate risk.43