We previously demonstrated absence of association between peer-review derived percentile ranking and raw citation impact in a large cohort of NHLBI cardiovascular R01 grants, but we did not consider pre-grant investigator publication productivity. We also did not normalize citation counts for scientific field, type of paper, and year of publication.
Determine whether measures of investigator prior productivity predict a grant’s subsequent scientific impact as measured by normalized citation metrics.
Methods and Results
We identified 1492 investigator-initiated de novo NHLBI R01 grant applications funded between 2001 and 2008 and linked the publications from these grants to their “InCites™” (Thompson Reuters) citation record. InCites™ provides a normalized citation count for each publication stratifying by year of publication, type of publication, and field of science. The co-primary endpoints for this analysis were the normalized citation impact per million dollars allocated and the number of publications per grant that have normalized citation rate in the top decile per million dollars allocated (“top-10% papers”). Prior productivity measures included the number of NHLBI-supported publications each principal investigator published in the 5 years before grant review and the corresponding prior normalized citation impact score. After accounting for potential confounders, there was no association between peer-review percentile ranking and bibliometric endpoints (all adjusted P > 0.5). However, prior productivity was predictive (P<0.0001).
Even after normalizing citation counts, we confirmed a lack of association between peer-review grant percentile ranking and grant citation impact. However, prior investigator publication productivity was predictive of grant-specific citation impact.
bibliometrics; National Heart; Lung; Blood Institute (U.S.); research impact
Precision medicine; Tailored interventions; Personalized medicine; Mobile health; Health informatics; Pharmacogenetics; Cohort studies; Behavioral risk factors; Environmental risk factors
Funding decisions for cardiovascular R01 grant applications at NHLBI largely hinge on percentile rankings. It is not known whether this approach enables the highest impact science.
To conduct an observational analysis of percentile rankings and bibliometric outcomes for a contemporary set of funded NHLBI cardiovascular R01 grants.
Methods and results
We identified 1492 investigator-initiated de novo R01 grant applications that were funded between 2001 and 2008, and followed their progress for linked publications and citations to those publications. Our co-primary endpoints were citations received per million dollars of funding, citations obtained within 2-years of publication, and 2-year citations for each grant’s maximally cited paper. In 7654 grant-years of funding that generated $3004 million of total NIH awards, the portfolio yielded 16,793 publications that appeared between 2001 and 2012 (median per grant 8, 25th and 75th percentiles 4 and 14, range 0 – 123), which received 2,224,255 citations (median per grant 1048, 25th and 75th percentiles 492 and 1,932, range 0 – 16,295). We found no association between percentile ranking and citation metrics; the absence of association persisted even after accounting for calendar time, grant duration, number of grants acknowledged per paper, number of authors per paper, early investigator status, human versus non-human focus, and institutional funding. An exploratory machine-learning analysis suggested that grants with the very best percentile rankings did yield more maximally cited papers.
In a large cohort of NHLBI-funded cardiovascular R01 grants, we were unable to find a monotonic association between better percentile ranking and higher scientific impact as assessed by citation metrics.
Research funding; bibliometrics; scientific impact; NHLBI
We sought to determine if treadmill exercise time may be of value as an initial prognostic screening tool in ambulatory patients with impaired systolic function referred for cardiopulmonary exercise testing.
Methods and Results
We studied 2,231 adult systolic heart failure patients (27% women) who underwent cardiopulmonary stress testing using a modified Naughton protocol. We assessed the value of treadmill exercise time for prediction of all-cause death and a composite of death or UNOS status 1 heart transplantation. During a mean follow up of 5 years, 742 (33%) patients died. There were 249 (11%) UNOS status 1 heart transplants. Treadmill exercise time was predictive of death and the composite outcome in both women and men, even after accounting for peak oxygen consumption and other clinical covariates (adjusted hazard ratio of lowest versus high sex-specific quartile for prediction of death 1.70, 95% CI 1.05–2.75, P=0.03, and for prediction of the composite outcome 1.75, 95% CI 1.15–2.66, P=0.009). For a one minute change in exercise time there was a 7% increased hazard of death (e.g. comparing 480 to 540 seconds HR 1.07, 95% CI 1.02–1.12, P=0.004).
Since cardiopulmonary stress testing is not available in every hospital, treadmill exercise time using a modified Naughton protocol may be of value as an initial prognostic screening tool.
Heart failure; exercise; sex; prognosis
Rapid publication of clinical trials is essential in order for the findings to yield maximal benefits for public health and scientific progress. Factors affecting the speed of publication of the main results of government-funded trials have not been well characterized.
We analyzed 244 extramural randomized clinical trials of cardiovascular interventions that were supported by the National Heart, Lung, and Blood Institute (NHLBI). We selected trials for which data collection had been completed between January 1, 2000, and December 31, 2011. Our primary outcome measure was the time between completion of the trial and publication of the main results in a peer-reviewed journal.
As of March 31, 2012, the main results of 156 trials (64%) had been published (Kaplan–Meier median time to publication, 25 months, with 57% published within 30 months). Trials that focused on clinical events were published more rapidly than those that focused on surrogate measures (median, 9 months vs. 31 months; P<0.001). The only independent predictors of more rapid publication were a focus on clinical events rather than surrogate end points (adjusted publication rate ratio, 2.11; 95% confidence interval, 1.26 to 3.53; P = 0.004) and higher costs of conducting the trial, up to a threshold of approximately $5 million (P<0.001). The 37 trials that focused on clinical events and cost at least $5 million accounted for 67% of the funds spent on clinical trials but received 82% of the citations. After adjustment of the analysis for a focus on clinical events and for cost, trial results that were classified as positive were published more quickly than those classified as negative.
Results of less than two thirds of NHLBI-funded randomized clinical trials of cardiovascular interventions were published within 30 months after completion of the trial. Trials that focused on clinical events were published more quickly than those that focused on surrogate end points. (Funded by the National Heart, Lung, and Blood Institute.)
translational research; research funding
Both abnormal microvolt T-wave alternans (MTWA) and low peak VO2 predict poor outcome in heart failure. However, their independent predictive properties have not been assessed in large scale cohorts.
We performed an observational prospective cohort study of 303 consecutive patients referred for metabolic stress testing. All had an ejection fraction fl40% and were considered candidates for transplantation. Patients with defibrillators did not have MTWA collected by our exercise laboratory and therefore were not included in the analysis. The primary endpoint was a composite of all-cause death or UNOS 1 transplantation.
During 2.8 years there were 34 deaths and 17 transplantations. Patients with abnormal MTWA had a higher event rate (31/136, 23%, vs. 20/167, 12%, unadjusted HR 1.90, 95% CI 1.90-3.33, P=.03). The association remained significant after adjustment for 3 clinical variables (1.89, 95% CI 1.05-3.39, P=.03). After adding peak VO2 to the model the association was no longer significant (adjusted HR 1.18, 95% CI 0.64-2.17, p=.60). After accounting for peak VO2 and 28 other confounders in a matched propensity analysis, MTWA was not predictive (propensity-matched HR 0.79, 95% CI 0.37-1.66, P=.53).
We confirm the association of abnormal MTWA with poor outcome amongst patients with impaired left ventricular systolic function. However, this association is markedly attenuated after accounting for peak VO2.
Heart failure survival models are typically constructed using Cox-proportional hazards regression. Regression modeling suffers from a number of limitations, including bias introduced by commonly used variable selection methods. We illustrate the value of an intuitive, robust approach to variable selection, random survival forests (RSF), in a large clinical cohort. RSF is a potentially powerful extension of Classification and Regression Trees (CART), with lower variance and bias.
Methods and Results
We studied 2231 adult systolic heart failure patients who underwent cardiopulmonary stress testing. During a mean follow-up of 5 years, 742 patients died. Thirty-nine demographic, cardiac and noncardiac co-morbidity, and stress testing variables were analyzed as potential predictors of all-cause mortality. A RSF of 2000 trees was constructed, with each tree constructed on a bootstrap sample from the original cohort. The most predictive variables were defined as those near the tree trunks (averaged over the forest). The RSF identified peak VO2, serum BUN, and treadmill exercise time as the three most important predictors of survival. The RSF predicted survival similarly to a conventional Cox-proportional hazards model (out-of-bag C-index of 0.705 for RSF vs 0.698 for Cox-proportional hazards model).
A random survival forests model in a cohort of heart failure patients performed as well as a traditional Cox-proportional hazard model, and may serve as a more intuitive approach for clinicians to identify important risk factors for all-cause mortality.
Heart failure; prognosis; statistical modeling; survival analyses
In 2012, the National Cancer Institute (NCI) engaged the scientific community to provide a vision for cancer epidemiology in the 21st century. Eight overarching thematic recommendations, with proposed corresponding actions for consideration by funding agencies, professional societies, and the research community emerged from the collective intellectual discourse. The themes are (i) extending the reach of epidemiology beyond discovery and etiologic research to include multilevel analysis, intervention evaluation, implementation, and outcomes research; (ii) transforming the practice of epidemiology by moving towards more access and sharing of protocols, data, metadata, and specimens to foster collaboration, to ensure reproducibility and replication, and accelerate translation; (iii) expanding cohort studies to collect exposure, clinical and other information across the life course and examining multiple health-related endpoints; (iv) developing and validating reliable methods and technologies to quantify exposures and outcomes on a massive scale, and to assess concomitantly the role of multiple factors in complex diseases; (v) integrating “big data” science into the practice of epidemiology; (vi) expanding knowledge integration to drive research, policy and practice; (vii) transforming training of 21st century epidemiologists to address interdisciplinary and translational research; and (viii) optimizing the use of resources and infrastructure for epidemiologic studies. These recommendations can transform cancer epidemiology and the field of epidemiology in general, by enhancing transparency, interdisciplinary collaboration, and strategic applications of new technologies. They should lay a strong scientific foundation for accelerated translation of scientific discoveries into individual and population health benefits.
big data; clinical trials; cohort studies; epidemiology; genomics; medicine; public health; technologies; training; translational research
Simultaneous contribution of hundreds of electrocardiographic biomarkers to prediction of long-term mortality in post-menopausal women with clinically normal resting electrocardiograms (ECGs) is unknown.
Methods and Results
We analyzed ECGs and all-cause mortality in 33,144 women enrolled in Women’s Health Initiative trials, who were without baseline cardiovascular disease or cancer, and had normal ECGs by Minnesota and Novacode criteria. Four hundred and seventy seven ECG biomarkers, encompassing global and individual ECG findings, were measured using computer algorithms. During a median follow-up of 8.1 years (range for survivors 0.5–11.2 years), 1,229 women died. For analyses cohort was randomly split into derivation (n=22,096, deaths=819) and validation (n=11,048, deaths=410) subsets. ECG biomarkers, demographic, and clinical characteristics were simultaneously analyzed using both traditional Cox regression and Random Survival Forest (RSF), a novel algorithmic machine-learning approach. Regression modeling failed to converge. RSF variable selection yielded 20 variables that were independently predictive of long-term mortality, 14 of which were ECG biomarkers related to autonomic tone, atrial conduction, and ventricular depolarization and repolarization.
We identified 14 ECG biomarkers from amongst hundreds that were associated with long-term prognosis using a novel random forest variable selection methodology. These were related to autonomic tone, atrial conduction, ventricular depolarization, and ventricular repolarization. Quantitative ECG biomarkers have prognostic importance, and may be markers of subclinical disease in apparently healthy post-menopausal women.
Electrocardiography; epidemiology; women; prognosis
Implantable cardioverter defibrillators (ICDs) are increasingly used for primary prevention followingrandomized controlled trials (RCTs) demonstrating that they reduce the risk of death in patients with left ventricular systolic dysfunction (LVSD). The extent to which the clinical characteristics and long-term outcomes of unselected, community-based patients with LVSD undergoing primary prevention ICD implantation in a real-world setting compare with those enrolled in the RCTs is not well characterized. The Longitudinal Study of ICDs is being conducted to address these questions.
Methods and Results
The study cohort includes consecutive patients undergoing primary prevention ICD placement between 1/1/2006 and 12/31/2009 in seven health plans. Baseline clinical characteristics were acquired from the NCDRICD Registry. Longitudinal data collection is underway and will include hospitalization, mortality, and resource utilization from the Virtual Data Warehouse. Data regarding ICD therapies will be obtained through chart abstraction and adjudicated by a panel of experts in device therapy. Compared with the populations of primary prevention ICD therapy RCTs, the cohort (n=2,621) is on average significantly older (by 2.5-6.5 years); more often female, more often from racial and ethnic minority groups, and has a significantly higher burden of coexisting conditions. The cohort is similar, however, to a national population undergoing primary prevention ICD placement.
Patients undergoing primary prevention ICD implantation in the Longitudinal Study of ICDs differ from those enrolled in the RCTs that established the efficacy of ICDs. Understanding a broad range of health outcomes, including ICD therapies, in this cohort will provide patients, clinicians, and policy-makers with contemporary data to inform decision-making.
arrhythmia; electrophysiology; epidemiology
Comparative effectiveness research (CER) aims to provide decision-makers the evidence needed to evaluate the benefits and harms of alternative clinical management strategies. CER has become a national priority, with considerable new research funding allocated. Cardiovascular disease is a priority area for CER. This workshop report provides an overview of CER methods, with an emphasis on practical clinical trials and observational treatment comparisons. The report also details recommendations to the National Heart Lung and Blood Institute for a new framework for evidence development to foster cardiovascular CER, and specific studies to address eight clinical issues identified by the Institute of Medicine as high priorities for cardiovascular CER.
comparative effectiveness; research methods; clinical trials
In the current issue of Population Health Metrics, two reports paint a bleak picture of American public health. Both physical inactivity and obesity remain highly prevalent; yet, it is not clear that increased physical activity will reduce the burden of obesity. There continue to be widespread disparities in life expectancy across United States counties. These reports appear against a backdrop of debate regarding how we should allocate our scarce resources for improving health: should we focus more on improving access to high-quality medical care, or should we instead focus on more and better public health interventions? While optimal solutions remain obscure, a look at prior successes suggests that ultimately they will come from the conduct and implementation of rigorous science, and in particular event-driven trials.
Public health; Population science; Obesity; Physical activity; Life-expectancy; Randomized trials
Over the past 60 years, revolutionary discoveries made by epidemiologists have contributed to marked declines in cardiovascular disease morbidity and mortality. Now, in an era of increasingly constrained resources, researchers in cardiovascular epidemiology face a number of challenges that call for novel, paradigm-shifting approaches. In this paper, the authors pose to the community 4 critical questions: 1) How can we avoid wasting resources on studies that provide little incremental knowledge? 2) How can we assure that we direct our resources as economically as possible towards innovative science? 3) How can we be nimble, responding quickly to new opportunities? 4) How can we identify prospectively the most meritorious research questions? Senior program staff at the National Heart, Lung, and Blood Institute invite the epidemiology community to join them in an ongoing Web-based blog conversation so that together we might develop novel approaches that will facilitate the next generation of high-impact discoveries.
cardiovascular diseases; epidemiology; National Institutes of Health (U.S.); research
Atrial fibrillation (AF) after cardiac surgery is a common marker of poor outcomes. Quantitative electrocardiographic (ECG) measurements may be valuable predictors of postoperative AF.
We evaluated clinical and ECG predictors of postoperative AF in 13,356 patients who underwent cardiac surgery in sinus rhythm.
4,724 patients (35%) developed postoperative AF. P-wave amplitude in lead aVR and V1 were the strongest ECG predictors. A less negative P-wave amplitude in lead aVR was associated with increased risk for postoperative AF (OR 1.46, CI 1.32–1.61), as was a more positive or a more negative P-wave amplitude in lead V1 (OR 1.25, CI 1.16–1.36) after adjusting for clinical and procedural predictors of postoperative AF. Reclassification analysis showed a 7% discrimination improvement (p<0.0001).
P-wave amplitude in lead aVR and lead V1 are powerful predictors of postoperative AF and in combination with other clinical predictors can guide application of prophylactic interventions.
editorials; exercise; risk, low; Framingham Risk Score; cardiovascular diseases
Chronic kidney disease (CKD) increases cardiovascular risk and mortality. However, traditional cardiovascular risk factors do not adequately account for the substantial increase in mortality observed in CKD. The aim of this study was to examine the relative contributions of novel cardiovascular risk factors to the risk between CKD and mortality. The study population included 4,680 consecutive new patients from a tertiary care preventive cardiology program from 1996 to 2005. Estimated glomerular filtration rate was calculated using the Modification of Diet in Renal Disease (MDRD) method. Baseline levels of traditional (low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, hypertension, triglycerides, total cholesterol, and fasting glucose) and emerging (apolipoproteins A-I and B, lipoprotein[a], fibrinogen, homocysteine, and high-sensitivity C-reactive protein) risk factors were examined. All-cause mortality was obtained from the Social Security Death Index. There were 278 deaths over a median follow-up period of 22 months. CKD (estimated glomerular filtration rate ≤60 ml/min/1.73 m2) was strongly associated with mortality after adjusting for traditional cardiovascular risk factors (hazard ratio 2.31, 95% confidence interval 1.77 to 3.11, p <0.001) and with the addition of propensity score (hazard ratio 2.33, 95% confidence interval 1.75 to 3.10, p <0.001). Of all the traditional and emerging risk factors monitored, only the addition of homocysteine and fibrinogen significantly attenuated the association between CKD and mortality (adjusted hazard ratio 1.73, 95% confidence interval 1.23 to 2.34, p <0.001), explaining 38% of the attributable mortality risk from CKD. A significant interaction (p = 0.004) between homocysteine and estimated glomerular filtration rate was observed whereby the annual mortality rate in subjects with CKD with homocysteine <10 μmol/L (the bottom tertile) was similar to those with normal renal function (1% per year), whereas homocysteine levels ≥12.5 μmol/L (the top tertile) were associated with a sevenfold greater mortality risk. In conclusion, homocysteine and fibrinogen levels explain nearly 40% of the attributable mortality risk from CKD.
The increasing availability of personal genomic tests has led to discussions about the validity and utility of such tests and the balance of benefits and harms. A multidisciplinary workshop was convened by the National Institutes of Health and the Centers for Disease Control and Prevention to review the scientific foundation for using personal genomics in risk assessment and disease prevention and to develop recommendations for targeted research. The clinical validity and utility of personal genomics is a moving target with rapidly developing discoveries but little translation research to close the gap between discoveries and health impact. Workshop participants made recommendations in five domains: (1) developing and applying scientific standards for assessing personal genomic tests; (2) developing and applying a multidisciplinary research agenda, including observational studies and clinical trials to fill knowledge gaps in clinical validity and utility; (3) enhancing credible knowledge synthesis and information dissemination to clinicians and consumers; (4) linking scientific findings to evidence-based recommendations for use of personal genomics; and (5) assessing how the concept of personal utility can affect health benefits, costs, and risks by developing appropriate metrics for evaluation. To fulfill the promise of personal genomics, a rigorous multidisciplinary research agenda is needed.
behavioral sciences; epidemiologic methods; evidence-based medicine; genetics; genetic testing; genomics; medicine; public health
In July of 2008, the National Heart, Lung, and Blood Institute convened experts in noninvasive cardiovascular imaging, outcomes research, statistics, and clinical trials to develop recommendations for future randomized controlled trials of the use of imaging in: 1) screening the asymptomatic patient for coronary artery disease; 2) assessment of patients with stable angina; 3) identification of acute coronary syndromes in the emergency room; and 4) assessment of heart failure patients with chronic coronary artery disease with reduced left ventricular ejection fraction. This study highlights several possible trial designs for each clinical situation.
cardiovascular imaging; chest pain diagnosis; clinical trials
cardiovascular imaging; chest pain diagnosis; clinical trials