|Home | About | Journals | Submit | Contact Us | Français|
The recently developed and internally validated Health ABC HF model uses nine routinely available clinical variables to determine incident heart failure risk. In this study, we sought to externally validate the Health ABC HF model.
Observed 5-year incidence of heart failure, defined as first hospitalization for new onset heart failure, was compared with 5-year risk estimates derived from the Health ABC HF model among participants without heart failure at baseline in the Cardiovascular Health Study. During follow-up, 400 of 5335 (7.5%) participants developed heart failure over 5 years vs. 364 (6.8%) predicted by the Health ABC HF model (predicted to observed ratio, 0.90). Observed vs. predicted 5-year heart failure probabilities were 3.2% vs. 2.8%, 9.0% vs. 7.0%, 15.9% vs. 13.7%, and 24.6% vs. 30.8% for the <5%, 5–10%, 10–20%, and >20% 5-year risk categories, respectively. The Hosmer-Lemeshow χ2 was 14.72 (d.f.=10; P=0.14) and the C index was 0.74 (95% CI, 0.72–0.76). Calibration and discrimination demonstrated adequate performance across sex and race overall; however risk was underestimated in white men, especially in the 5–10% risk category. Model performance was optimal when participants with normal left ventricular function at baseline were assessed separately. Performance was consistent across age groups. Analyses with death as a competing risk yielded similar results.
The Health ABC HF model adequately predicted 5-year heart failure risk in a large community-based study, providing support for the external validity of the model. This tool may be used to identify individuals to target heart failure prevention efforts.
Heart failure prevalence continues to rise globally, with the majority of affected individuals being elderly.1–3 Unless effective prevention strategies are implemented, this trend is likely to worsen as the proportion of elderly individuals increases in the population.4 However, in order to implement cost-effective prevention strategies, risk stratification for incident heart failure at the population level is important. Such risk prediction schemes have to be applicable across major demographic groups. Moreover, to enhance community dissemination, these prediction schemes should preferably be based on parameters that are widely available and obtained at no or minimal cost.
We recently developed and internally validated the Health ABC Heart Failure (HF) model,5 which utilizes nine routinely available clinical variables to provide 5-year incident heart failure risk estimates in older adults. The model performed well in the overall cohort and in the four gender and race subgroups. Although internal validation by bootstrapping is a valid statistical method,6 it does not obviate the need for external validation, because bootstrapping cannot overcome the limitations related to the specific characteristics of the derivation population, which may be biased in some respects. Specifically, the Health ABC HF model was derived from data on community-dwelling, well-functioning elderly participants of the Health, Aging and Body Composition (Health ABC). To be enrolled in the study, participants had to be able to walk 1/4 mile without difficulty or climb 10 stairs without resting. Exclusion criteria included difficulties with activities of daily living, obvious cognitive impairment, or inability to communicate. Thus, it is possible that the Health ABC Study enrolled individuals who were healthier than the average “real-life” elderly person, raising concerns about the model performance in other populations.
In this study, we investigated the performance of the Health ABC HF model in the Cardiovascular Health Study (CHS), a large community-based cohort study designed to assess cardiovascular disease epidemiology.7
The design, rationale, and details of CHS have been previously published.7 Eligible participants (non-institutionalized persons aged 65 to 100 years expected to remain in the defined geographic area for ≥3 years) were recruited from Medicare eligibility lists and examined at the four field centers in Forsyth County, NC; Sacramento County, CA; Allegheny County, PA; and Washington County, MD. An original cohort of 5201 persons was recruited in 1989–90, and a second cohort of 687 African-Americans was recruited in 1992–93, giving a total of 5888 participants (2495 men and 3393 women). Interviews and questionnaires were used to obtain information on medical history and medications. Physical examination and tests including electrocardiogram and echocardiogram were performed. Blood chemistries were assessed after a 12-hour overnight fast.
The Health ABC HF model was developed for 5-year heart failure risk prediction. Therefore, the model performance was evaluated for events during the first 5 years of follow-up in CHS participants without heart failure at baseline.8 Figure 1 presents the flow diagram for selection of CHS participants for the current analysis.
Classification of prevalent coronary heart disease was based on self-report of coronary revascularization, myocardial infarction, or angina pectoris.8 To align definitions with those used in the Health ABC Study, “definite” coronary heart disease was defined as history of coronary artery bypass graft surgery, percutaneous coronary intervention, myocardial infarction, or angina accompanied by antianginal medication use (calcium channel blockers, beta blockers, or nitrates). “Probable” coronary heart disease was defined as a history of angina without use of antianginal medications (or missing data on medications) and missing or negative information about coronary revascularization or myocardial infarction. Smoking was classified as current, past, or never. Electrocardiographic left ventricular hypertrophy was classified using the Minnesota code 3.1 criteria.9 Heart rate was recorded from the electrocardiogram. Blood pressure was calculated as the average of two sitting measurements. The core laboratory at the University of Vermont, Colchester, VT, performed analyses of fasting serum chemistry.
The design for the echocardiographic study in CHS has been published previously.10 Two-dimensional echocardiography was performed at baseline for the original cohort (n=5201); for the second cohort, consisting of 687 African-American individuals, echocardiography was performed at the year 2 visit. Global left ventricular systolic function could be assessed qualitatively in 99% of the cohort and was classified as normal, borderline, or impaired, corresponding to a left ventricular ejection fraction (LVEF) of ≥0.55, 0.45 to 0.54, and <0.45, respectively.
Methods used to assess cardiovascular events including heart failure in the CHS have been reported previously.11 Briefly, in CHS, potential clinical events were identified through (1) the regular surveillance process (every six months), which included clinic visits and surveillance calls by the field centers; during this process, participants were asked to provide information on all hospitalizations and outpatient end point diagnoses since the last CHS contact; (2) participant-initiated reports, in which participants or proxies contacted the local site to report an event; and (3) secondary sources of events, including unreported earlier hospitalizations or end point diagnoses during review of medical records for a reported event, and Medicare hospitalization data.
The CHS Events Committee adjudicated heart failure by reviewing all pertinent data, including history, physical examination, chest radiography report, and medication use.12 A heart failure event was confirmed if, in addition to a physician diagnosis, there was (1) documentation in the medical record of a constellation of symptoms (such as shortness of breath, fatigue, orthopnea, paroxysmal nocturnal dyspnea) and physical signs (such as edema, pulmonary rales, gallop rhythm, displaced left ventricular apical impulse); (2) supporting clinical findings such as evidence of pulmonary edema on chest X-ray; or (3) a record of medical therapy for heart failure, including diuretics, digitalis, angiotensin-converting enzyme inhibitors, or beta-blockers. All potential incident cardiovascular events in CHS were adjudicated by the events committee. To be consistent with the definition in the derivation cohort (the Health ABC Study), incident heart failure in the current analysis was defined as adjudicated first hospitalization for heart failure.
Left ventricular ejection fraction at the time of incident heart failure was not assessed systematically in CHS. Therefore, data on LVEF post heart failure development are based on reviews of the available echocardiographic reports by the CHS investigators.13
In the Health ABC HF model, heart failure risk prediction for each participant is based on the prognostic index (PI) of the model, i.e. the weighted sum of the nine covariates (age, history of coronary heart disease, smoking status, systolic blood pressure, heart rate, serum glucose, creatinine, and albumin levels, and left ventricular hypertrophy by electrocardiogram)5:
During internal validation,5 we estimated that predictions should be moderated by a linear factor of 0.95 (“shrinkage factor”) to obtain more realistic risk estimates.14 Thus, we used the moderated prognostic index (PImod) for all subsequent calculations:
The estimated heart failure risk P over 5 years becomes then:
where S0 is the event-free Kaplan-Meier estimate at 5 years (S0= 0.9658), as observed in the Health ABC Study, for a reference person with zero sum of covariates (PI=0).
We assessed calibration by comparing observed versus predicted probabilities of incident heart failure. In the total cohort, we compared probabilities across deciles of risk, obtained the Hosmer-Lemeshow goodness-of fit χ2 (H-L χ2), and plotted smoothed calibration plots; we also compared probabilities across categories of 5-year predicted risk (<5%, 5–10%, 10–20%, >20%) and calculated the predicted to observed ratio. In subgroup analyses, we only considered calibration across categories of risk because of small numbers of events within deciles of risk. Discrimination was evaluated using the Harrell’s C index for survival models,14 equivalent in interpretation to the area under the receiver-operator characteristic curve.
In subgroup analyses, we evaluated model performance in sex and race subgroups. Because CHS included participants 65–100 years old at baseline, whereas the model was derived in a population of age 70–79 years, we also examined performance in age groups by 5-year increments. Due to small number of participants and events in those ≥80 years old, the corresponding groups were combined into a single category. In the derivation cohort (Health ABC Study), data on baseline left ventricular systolic function were not available, raising the possibility that individuals with asymptomatic systolic dysfunction (stage B heart failure) may have been inadvertently included in the analysis. Although the model performance is still valid from a clinical perspective, as it is designed to assess the risk of clinical (stage C) heart failure irrespectively of baseline asymptomatic stage A or B, to further clarify this issue we examined model performance in the subset of individuals with normal left ventricular systolic function at baseline.
Mortality in the derivation and validation cohorts was higher than heart failure incidence, raising concerns about the validity of absolute risk estimates.15, 16 Therefore, we repeated the validation analyses using the competing-risks extension of the Cox proportional hazards model, as proposed by Fine and Gray,15 to estimate the baseline cumulative subhazard function of incident heart failure (i.e. the cumulative hazard of heart failure considering death as competing event) in the Health ABC Study. We then obtained predictions for cumulative heart failure incidence at 5 years in the CHS using the baseline cumulative subhazard function and the prognostic index of the Health ABC HF model.16 Further details for this analysis are provided in the online Supplemental Material.
To evaluate for possible bias introduced by the omission of participants (5.0%) with incomplete data on model covariates, we repeated the process in 5 imputed datasets including all participants who were free of prevalent heart failure at baseline. For imputation of missing values we used imputation by chained equations as described by van Buuren.17 All statistical analyses were carried out in STATA 10.1.
There were 5613 CHS participants without prevalent heart failure at baseline. Of these, 278 (5.0%) had incomplete data on Health ABC model variables and were excluded from the main analysis, leaving 5335 participants in the validation cohort (Figure 1). Mean age was 72.7±5.5 years; 57.6% were women and 84.7% were white (96.2% of non-white participants were black). The characteristics of the derivation (Health ABC) and validation (CHS) cohorts are presented in Table 1. There were 400/5335 (7.5%) incident heart failure events at 5 years, corresponding to 16.2 events per 1000 person-years (95% CI, 14.7–17.9), and 651/5335 (12.2%) deaths, corresponding to 2.6% (95% CI, 2.4–2.8%) annual mortality. There was no loss to follow-up during the 5-year period.
Data on left ventricular systolic function at baseline were available in 4720 of 5335 (88.5%) participants in the validation cohort; of these, 4363 (92.4%) had normal systolic function, 235 (5.0%) had borderline systolic function, and 122 (2.6%) had impaired systolic function. The 357 participants with diminished left ventricular systolic function contributed 62 (15.5%) of the total 400 incident heart failure cases.
Data on LVEF at the time of incident heart failure were available in 228 of the 400 (57.0%) heart failure cases. The average LVEF at the time of incident heart failure in those cases was 43.4±16.1% (median, 43%; interquartile range, 30%–55%); 109 of 228 (57.3%) cases presented with a LVEF ≥45%.
Table 2 presents the observed 5-year heart failure incidence in comparison to the Health ABC HF model predicted probability across categories of predicted risk. The total number of expected events was 364; 36 cases less than the observed 400 (predicted 5-year probability 6.8% vs. observed 7.5%; ratio, 0.90). Observed 5-yearheart failure incidence was within the predicted 5-year probability range in all risk categories. Figure 2 shows the calibration of the model across deciles of predicted risk. The H-L χ2 did not reach significance (χ2=14.72; d.f.=10; P=0.14), indicating adequate model fit. The C index was 0.74 (95% CI, 0.72–0.76), comparable to the moderated estimate (C=0.72) obtained during development of the Health ABC HF model.5
Among participants with normal left ventricular systolic function (n=4363), there were 275 (6.3%) incident heart failure events at 5 years vs. 271.4 (6.2%) predicted with the model (predicted to observed ratio, 0.99). Observed vs. predicted 5-year heart failure probabilities were 3.5 vs. 3.1%, 7.7% vs. 7.8%, 14.9% vs. 15.0%, and 28.0% vs. 33.3% for the <5%, 5–10%, 10–20%, and >20% 5-year risk categories, respectively. The H-L χ2 was 8.02 (d.f.=10; P=0.63) and the C index was 0.73 (95% CI, 0.70–0.76).
Table 3 and Figure 3 summarize the predictions of the Health ABC HF model in the sex- and race-based subgroups. Overall risk was underestimated in white males (predicted vs. observed 5-year heart failure probability, 7.7% vs. 9.6%; ratio, 0.80); this was primarily due to underestimation of risk in the 5–10% risk category (Figure 3). Predicted 5-year heart failure probability was concordant with the observed heart failure incidence in the remaining subgroups (Table 3, Figure 3). Black men and women represented small subgroups with only 26 and 40 events, respectively. Therefore, assessment of calibration in these subgroups should be interpreted with caution. Discrimination (C index) was adequate in the sex- and race-based subgroups (Table 3). Figure 4 summarizes the age-specific performance of the model; observed and predicted heart failure probabilities were concordant across age groups.
When predictions were obtained using the 5-year cumulative subhazard estimate for incident heart failure (with death as a competing event) as the basis of predictions, results were very similar to those obtained using the Kaplan-Meier estimate (standard Cox approach) as the basis for predictions. The model predicted a total of 365 events in the validation cohort (predicted to observed ratio, 0.91). Observed vs. predicted 5-year heart failure probabilities were 3.2 vs. 2.8%, 8.9% vs. 7.0%, 15.9% vs. 13.7%, and 24.7% vs. 30.8% for the <5%, 5–10%, 10–20%, and >20% 5-year risk categories, respectively. The corresponding H-L χ2 was 14.42 (d.f.=10; P=0.15). The calibration plot for this analysis is provided in the online Supplemental Material.
In analyses performed on 5 imputed datasets including all participants without heart failure at baseline (n=5613), the increased number of participants and events led to higher values of the H-L statistic, suggesting relatively more marginal model fit. However, predictions were not materially affected from a clinical perspective. The predicted to observed ratio was 0.91–0.92 across imputed datasets and the C index was 0.74. The performance summaries for these datasets are provided in the online Supplemental Material.
In this study, we examined the external validity of the Health ABC HF model in the CHS cohort. The model showed clinically adequate properties in this independent cohort and performed well in sex- and gender-based subgroups. These data support the potential use of this model to determine heart failure risk in the elderly and possibly identify individuals who would benefit the most from preventive interventions.
The Health ABC HF model was developed in a well-functioning, well-characterized, elderly population and showed good performance on internal validation. However, internal validation takes into account only sampling variability; thus, external validation is necessary in order to assess generalizability of a model.18, 19 Moreover, baseline echocardiographic assessment was not performed in the Health ABC Study, raising the possibility that individuals with asymptomatic systolic dysfunction (stage B heart failure) may have been inadvertently included in the analysis. The CHS was designed to assess the development and progression of cardiovascular disease in the elderly and was ideally suited for external validation of the Health ABC HF model. The enrollment criteria provided an opportunity for a broader group of elderly persons to be studied. In contrast to the Health ABC Study, all subjects in the original CHS cohort had a baseline echocardiogram performed. The performance of the model in this independent cohort supports heart failure risk prediction with this tool. Moreover, model performance remained stable when assessed in the subset of individuals with normal left ventricular systolic function, alleviating concerns about the validity of the original results due to potential influence of inclusion of participants with baseline asymptomatic ventricular dysfunction, i.e. in transition from stage B to stage C heart failure. It is important to note, however, that although assessment of ventricular function is not recommended for screening purposes, impaired left ventricular systolic function increases risk for heart failure and, when known, probably adds to risk stratification. Thus, although assessment of ventricular function is not a cost-effective screening procedure in the general population,20, 21 targeted assessment of ventricular function in individuals at intermediate or higher risk for heart failure might be worth investigating in prospective studies.
Literature continues to evolve elucidating the differences between men and women with respect to risk factors and outcomes related to cardiovascular diseases.22, 23 The majority of previous literature on heart failure risk factors was derived from white populations, including the Framingham Heart Failure Risk Score.24 We have previously demonstrated that blacks have a higher accumulation of heart failure risk factors, a higher prevalence of heart failure, and worse outcomes once heart failure develops.25. Even for risk factors that are predictive of heart failure in both white and blacks, there are significant differences in the population attributable risk for these risk factors -- a prime example being electrocardiographic left ventricular hypertrophy --between the two races.25 These differences derive in turn from differences in the prevalence and relative contribution of the various risk factors and may directly affect the performance of a predictive model in different racial and sex groups. Based on these important differences in race- and gender-based subgroups, it is important for any risk prediction scheme to be tested in these specific populations. Without a wide applicability of screening tools in the general population, the practical applicability may be significantly limited. The Health ABC HF model was the first attempt to systematically stratify heart failure risk in these major demographic subgroups.5 In this external validation study, the model had adequate performance in white women; however, risk was underestimated among white men in the intermediate risk category. The cause of this deviation is not obvious and may represent a more prominent role of non-traditional risk factors, necessitating assessment of other risk markers for further risk classification in the intermediate risk male population. On the other hand, we did not observe systematic effects of race on model performance; however, the overall number of participants and events was low in black men and black women and therefore these results need to be cautiously interpreted.
Considering the increasing proportion of elderly individuals in the population, the already worsening heart failure epidemic is likely to accentuate.3, 26 This will have major clinical, quality of life, and economic consequences. For many other common, lethal, and costly diseases, like breast or colon cancer and coronary heart disease, there are specific tools to either risk stratify individuals or perform screening tests to detect high risk individuals.27–29 Heart failure is an exception where there are no targeted risk stratification tools.30 This is partly rooted in the belief that treating individual risk factors like hypertension or diabetes would reduce the risk for heart failure. Although largely true, such an approach misses many details and opportunities. For example, there is a growing literature that treating blood pressure with different agents leads to a differential reduction in cardiovascular risk,31 and that for other risk factors, like diabetes, intensive control may have no impact on certain clinical outcomes.32 Treatment of individual risk factors is a disease- and not a patient-based approach. Many subjects have multiple risk factors that act in concert to determine an individual’s risk. Therefore, patients may need differential control of risk factors if multiple risk factors coexist, e.g. hypertension control in individuals with both hypertension and diabetes is different than those with hypertension alone.33 Moreover, evidence suggests that among agents used to control a risk factor, e.g. hypertension, there may be differential preventive effects regarding development of heart failure. For example, in the ALLHAT trial,34 there was a significantly higher risk of new onset heart failure with doxazosin-based therapy. Similarly, there were quantitative differences in incident heart failure between chlorthalidone-based versus lisinopril- or amlodipine-based therapy.34 These data suggest that careful therapeutic consideration needs to be given when attempting to treat risk factors for heart failure in order to achieve maximum benefit. Also, the Health ABC model could serve as a tool to identify individuals who might require intensified control of individual risk factors based on the overall heart failure risk profile; however, this concept is speculative and needs further study. It is also likely, though not proven, that non-pharmacologic life style interventions might reduce heart failure risk in high-risk individuals. The Health ABC HF model therefore puts the risk for incident heart failure development in a comprehensive perspective and may potentially be used both for individual risk assessment and for intervention evaluation in research settings.
Both the Health ABC Study and CHS enrolled older adults. However, as the risk factor profile for cardiovascular diseases, including obesity and metabolic syndrome increases in the society,35 it is possible that a higher proportion of younger individuals may develop heart failure.36 How the Health ABC HF model will perform for risk assessment in younger individuals needs further study. Neither the Health ABC Study nor CHS had left ventricular function systematically assessed after heart failure development and thus the differential properties of the model for prediction of heart failure with reduced vs. preserved left ventricular systolic function need further investigation. Prevalence of risk factors and their impact on developing disease may vary by race; for example, risk for coronary heart disease in subjects of Asian descent tends to be lower at the same levels of risk factors like body mass index or cholesterol levels.37 The performance of the model in individuals not of white or black race needs further study. In addition, risk calculation is based on accurate identification of prevalent risk factors in a population; although variability in risk factor ascertainment is less of a concern for physical examination (e.g. blood pressure), laboratory (e.g. serum creatinine), and behavioral (e.g. smoking) parameters, ascertainment of coronary heart disease may be challenging. Despite the vigorous study design of CHS,8 documentation of coronary imaging or revascularization is not always present, and history of myocardial infarction, and even more so angina, can be subject to interpretation. This can adversely affect the model estimates as presence of coronary heart disease may be overdiagnosed (e.g. subjective chest pain and use of beta-blockers or calcium blocker for other indications like hypertension) or underdiagnosed (e.g. recall bias or suboptimal therapy). Also, it is important to note that the Health ABC HF model is based on inpatient incident heart failure and, therefore, provides only an estimate of the risk of hospitalization for incident heart failure among the elderly. Finally, in the development of the Health ABC HF prediction model, we specifically focused on commonly available clinical variables that may be suitable for screening purposes. Imaging modalities and biomarkers may also help classify risk further, especially in the intermediate risk group. However, the cost-effectiveness of these more expensive tests needs further study.
In conclusion, the Health ABC HF model demonstrated adequate performance for heart failure risk assessment in a large, prospective cohort. Whether determination of heart failure risk in the community using this tool and subsequent lifestyle or other intervention will help reduce heart failure incidence needs further study. Considering the epidemiologic trends in both, the societal demographics and the cardiovascular risk factor profile, such efforts are essential.
Research Support: The research reported in this article was supported by contract numbers N01-HC-85079 through N01-HC-85086, N01-HC-35129, N01 HC-15103, N01 HC-55222, N01-HC-75150, N01-HC-45133, grant number U01 HL080295 from the National Heart, Lung, and Blood Institute, with additional contribution from the National Institute of Neurological Disorders and Stroke. A full list of the Cardiovascular Health Study investigators and institutions can be found at http://www.chs-nhlbi.org/pi.htm. This project was also partially funded by an Emory University Heart and Vascular Board grant entitled ‘Novel Risk Markers and Prognosis Determination in Heart Failure’ and Public Health Service Grant UL1 RR025008 from the Clinical and Translational Science Award program, National Institutes of Health, National Center for Research Resources.
Conflict of Interest Disclosures
Andreas Kalogeropoulos: None
Bruce M. Psaty: None
Ramachandran S. Vasan: None
Vasiliki Georgiopoulou: None
Andrew L. Smith: None
Nicholas L. Smith: None
Stephen B. Kritchevsky: None
Peter W.F. Wilson: None
Anne B. Newman: None
Tamara B. Harris: None
Javed Butler: None