Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Am Coll Cardiol. Author manuscript; available in PMC 2013 August 14.
Published in final edited form as:
PMCID: PMC3416944

Future Directions for Cardiovascular Disease Comparative Effectiveness Research

Report of a Workshop Sponsored by the National Heart, Lung, and Blood Institute


Comparative effectiveness research (CER) aims to provide decision-makers the evidence needed to evaluate the benefits and harms of alternative clinical management strategies. CER has become a national priority, with considerable new research funding allocated. Cardiovascular disease is a priority area for CER. This workshop report provides an overview of CER methods, with an emphasis on practical clinical trials and observational treatment comparisons. The report also details recommendations to the National Heart Lung and Blood Institute for a new framework for evidence development to foster cardiovascular CER, and specific studies to address eight clinical issues identified by the Institute of Medicine as high priorities for cardiovascular CER.

Keywords: comparative effectiveness, research methods, clinical trials

Comparative Effectiveness Research (CER) has recently emerged as a national priority, spurred by health care reform and economic stimulus legislation. Congress appropriated 1.1 billion dollars for CER as part of the American Recovery and Reinvestment Act of 2009, and is anticipated to enable additional annual spending of 500 million dollars as part of the newly established Patient-Centered Outcomes Research Institute (PCORI). According to Congressional legislation, PCORI will give priority for project management to the National Institutes of Health and the Agency for Health Research and Quality. The National Heart, Lung, and Blood Institute (NHLBI) may therefore have new opportunities to advance CER related to cardiovascular disease, which remains the leading cause of death and disability in the United States today.

The NHLBI sponsors workshops to solicit input and recommendations on important topics, so on July 13 and 14, 2010 the Division of Cardiovascular Sciences convened a Workshop on CER in Cardiovascular Disease. The Workshop brought together 25 outside experts from a variety of disciplines (clinical trials, epidemiology, biostatistics, health services research, and clinical medicine) to discuss a range of future opportunities that NHLBI could consider in CER as it relates to cardiovascular disease and the specific priorities for CER in cardiovascular disease that were identified by the Institute of Medicine. The discussions at the Workshop therefore represent the opinions and recommendations of the participants, and are not necessarily the policy or priorities of NHLBI.

This report summarizes the deliberations and recommendations to the NHLBI of this Workshop: “Future Directions for Cardiovascular Disease Comparative Effectiveness Research.” The report is divided into several sections, including 1) an overview of CER data sources and methods, 2) a proposed framework for CER at the NHLBI (Figure 1), and 3) possible approaches to eight priority CER topics identified by the Institute of Medicine in the areas of cardiovascular and peripheral vascular disease.

Figure 1
Conceptual framework for comparative effectiveness research (CER)

Overview of CER

Comparative effectiveness research has been defined as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers and policy makers in making informed decisions that will improve health care at both the individual and population levels” (1). CER can focus on care at the patient level or the system level, but regardless of its scope, CER is intended to provide information that can facilitate medical decision making and improve health outcomes (2).

There is great public value in rigorous studies comparing alternative strategies for diagnosis and treatment, as demonstrated by many landmark NHLBI-sponsored clinical trials (313). Nevertheless, comparative effectiveness studies that would facilitate health care decisions are not performed as often as they are needed, and gaps persist between the production of scientific evidence and the needs of consumers and health care providers for evidence on CER. Since there are limited resources to support biomedical research, it is necessary to prioritize key clinical questions that can be answered with comparative effectiveness studies, while expanding as much as possible the pool of investigators capable of performing CER.

Study Designs for CER

A variety of methods are used in CER, including randomized trials, observational studies, simulations and models, systematic reviews, meta-analyses, and collaborative pooling of individual patient data from multiple studies. Randomized trials can be used to compare a management strategy (preventative, diagnostic, therapeutic) with the best alternative strategy. Analyses of data from clinical registries, electronic health records, and administrative databases can address CER questions where randomization may be difficult. While observational studies utilizing existing data may be simpler and less expensive to conduct, they are more susceptible to bias introduced by selection of patients for alternative treatments. Statistical approaches to analysis of observational data can be used to minimize these biases, but these methods need to be further developed. Decision models and simulations may also be very useful methods in CER. The Workshop participants explored the unique opportunities, strengths, and limitations of these methods for use in CER, as summarized below.

Randomized Trials

Randomized clinical trials (RCTs) may be applied in CER to compare treatments (e.g., use of a drug, device, procedure, or a behavioral intervention), clinical evaluation strategies (e.g., biomarkers, imaging), health care delivery methods (e.g., disease management programs, specialist vs. generalist care), and policy interventions (e.g., copayments, formulary restrictions, regionalization of procedures). The strengths of the RCT include its use of a prospective protocol, standardized data collection, complete follow-up and random treatment assignment, each of which enhances its “internal validity.” A RCT is particularly appropriate when a high level of evidence is required to change practice, such as when optimal management is controversial, the effect of the treatment on outcomes is modest, or the intervention is costly. Although RCTs often involve highly selected patients and atypical practice settings, which can limit generalizability, RCTs can be designed to enroll more representative patient populations and performed in more typical practice settings.

There are many practical barriers to conducting large, representative clinical trials in the United States. The focus on delivering care efficiently and uniformly may not readily accommodate clinical research in routine clinical practice. Building the infrastructure to perform multicenter clinical trials can be difficult, time-consuming, and costly. Thus, exploring strategies to promote more efficient clinical trials is important, such as adopting “large practical trial designs” or fostering the conduct of a range of CER trials by existing investigator or site teams. Use of real world settings, such as private practice groups, community health centers, and integrated health care systems, may enhance recruitment of representative patients in CER trials.

Monetary barriers can impede the conduct of clinical trials, since the cost of research data collection, clinic visits and tests, and follow-up is high. Thus, identifying ways to streamline data collection while maintaining accuracy and validity over the length of follow-up would enhance the ability to conduct CER trials. Health insurers may balk at covering the costs of clinic visits and tests for patients in a clinical trial, which increases the cost to the research budget and may lead patients to drop out of the study. Furthermore, in some instances financial incentives in health care may not be aligned for randomized trials; for example, trials that randomize patients to procedures compared with medical therapy may result in foregone procedural fees. The duration of follow-up in clinical trials is also constrained by cost, which limits collection of long-term data on comparative efficacy and safety.

Barriers for CER trials also include lack of time in busy practices to enroll participants in research studies and low levels of academic recognition for site investigators who are just one member of a large trial team. While Institutional Review Boards are needed to protect participants in clinical trials, the multiplicity of jurisdictions across multi-site trials complicates conduct of the trial. Centralized or simplified institutional review process for multi-center studies should be explored, particularly comparative trials of existing, approved interventions.

Clinical Registries

A clinical registry is the prospective collection of standardized data on the clinical characteristics and outcomes of patients defined by a particular disease, diagnosis, procedure or exposure. Clinical registries share some features with randomized trials, such as standardized data collection, but unlike clinical trials do not dictate patient treatment by either random assignment or a strict protocol. Clinical registries can have broader inclusion criteria than clinical trials, and therefore better represent the diversity of patients, providers, and practice settings found in contemporary clinical care. Clinical registries occupy a middle ground between the formal structure of a randomized trial and the collection of relatively unstructured data from medical charts, electronic health records, or claims data.

Clinical registries have evolved from small case series to national (or international) collaborations that enroll thousands to millions of patients. Standardized data definitions and data collection methods are key features of a high quality clinical registry, as they ensure comparable detail about each patient enrolled. Documenting outcomes is also essential if clinical registries are to be used in CER – particularly outcomes that occur late after a single episode of care. Longer follow-up can be obtained actively by contacting patients (as is done in clinical trials) or passively by linking registry data to electronic health records, claims data, or state and national mortality files.

Clinical registries have been used to define contemporary practice patterns, document disparities in care, and assess the safety of cardiovascular drugs, devices and procedures in clinical practice. Clinical registries can also be analyzed to compare alternative treatments, but these studies require advanced biostatistical methods to reduce the biases introduced by nonrandomized patient selection for treatment.

Clinical registries may also be linked with randomized clinical trials, as when a registry prospectively collects data on patients screened for entry into a trial, or on patients eligible for a trial who decline to be randomized (14). Randomized trials can also capitalize on an ongoing clinical registry, which can be used to identify eligible patients and capture clinical data. Hybrid registry trial designs may be particularly powerful tools for CER, as they permit efficient patient enrollment and provide information of the generalizability of trial results.

The resources required to establish and maintain clinical registries have been provided by a variety of mechanisms. Professional societies have sponsored notable clinical registries often supported by hospital-paid fees for participation, such as the American Heart Association’s Get with the Guidelines Programs (in cardiovascular disease, heart failure, and stroke), the Society of Thoracic Surgeon’s National Cardiothoracic Surgery databases (covering coronary bypass surgery, valve surgery, thoracic surgery and congenital heart surgery), and the American College of Cardiology’s National Cardiovascular Data Registries (including percutaneous coronary intervention, implantable defibrillators and carotid stenting, as well as diseases such as congenital cardiovascular disease, and acute coronary syndromes). State governments have also sponsored clinical registries, such as the New York State Registry for cardiac surgery and percutaneous coronary intervention. Clinical registries have also been established using federal research funding (e.g., the NHLBI sponsored Dynamic Registry). Industry has sponsored clinical registries on particular drugs or devices (e.g., stent registries), as well as specific diseases (e.g., the National Registry of Myocardial Infarction (NRMI), the Acute Decompensated Heart Failure National (ADHERE) Registry, and the Reduction of Atherothrombosis for Continued Health (REACH) Registry).

Health System Data

Electronic health records and the administrative records of insurers or integrated health plans are valuable sources of observational data for CER. These “found data” are generated in the routine practice of medical care for billing purposes, public reporting, or clinical care, and are not produced primarily for research purposes. For instance, the diagnoses and procedures during hospitalization are recorded using the nomenclature of the International Classification of Diseases, Clinical Modification (ICD-9-CM), and reported to public authorities and health insurers. These administrative data are not subject to the standardization and quality control applied to clinical trial or clinical registry data, although these data are usually recorded by trained medical records abstractors and required for provider reimbursement. Some integrated health care organizations have additional sources of clinical data, such as drug prescriptions, outpatient claims, and computerized laboratory results. Linkage of several of these data sources can provide a very detailed picture of medical care provided to a representative, relatively unselected population of patients.

The advent of fully electronic health records offers the possibility of capturing greatly detailed clinical information about individual patients, such as symptoms, vital signs, and results of imaging studies. There are formidable technical challenges in extracting specific data elements from electronic health records, since clinical notes are typically entered as free text rather than using a controlled vocabulary. Advances in medical informatics, such as studies on natural language processing, will likely facilitate use of electronic health records for research purposes. Nevertheless, use of electronic records for research purposes does not overcome the well-recognized fundamental limitations of retrospective chart review studies – namely, that key data may not have been recorded at all, and that the data that were recorded are unlikely to be standardized or quality controlled.

Analysis of Observational Data

A weakness of all observational CER studies (including analyses of clinical registries, electronic health records, and administrative data) is the absence of randomized assignment of treatments. In contrast to a randomized trial, clinicians and patients represented in observational databases select treatments for a variety of reasons, which may not be recorded in the chart. Treatment selection can lead to differences in patient prognosis between treatment groups, so their subsequent clinical outcomes may differ, even in the absence of a treatment effect. Methods for addressing selection biases, whether due to known or unknown factors, are evolving. One simple step is to restrict the study patient population to newly treated patients and to patients eligible for either treatment; these restrictions narrow any pre-treatment differences between patients receiving alternative therapies (15). Modeling the selection of treatment by using a propensity score or a disease risk score can balance treatment groups on large numbers of measured clinical covariates (16, 17). Marginal structural models with inverse weighting by propensity for treatment have been used to estimate the effect on outcomes of treatments that vary over time, such as use of prescription drugs (18). Each of these statistical methods relies on adjusting for clinical characteristics that were recorded in the data, and therefore may not adjust fully for clinical factors that were not recorded (e.g., patient frailty or socioeconomic status) or that are difficult to capture (e.g., degree of social support). Instrumental variable methods and multi-level analyses have been used in an attempt to adjust for unmeasured confounders (1921). These approaches identify variables that are strongly related to the likelihood of receiving a specific treatment, but do not directly affect clinical outcomes (e.g., inclusion of specific drugs in a formulary, distance to a referral hospital). The Workshop participants recognized the need for further advances in statistical methods in order to conduct high quality CER that minimizes confounding and residual selection bias in observational treatment comparisons.

Systematic Reviews and Decision Models

Systematic reviews and meta-analyses of existing effectiveness and safety data are an important tool for CER. Systematic reviews can identify evidence gaps, including a lack of evidence or unclear evidence for an important clinical question, which suggest a new trial is warranted (22), and when there is “too much evidence,” which suggests further trials may be unnecessary (23). Network meta-analysis (mixed treatment comparison meta-analysis) may be used to compare interventions even when direct “head-to-head” trials are not available (24). This new statistical approach is promising, but is still being examined for its validity as a method to compare treatments.

The potential for treatment efficacy to vary significantly according to patient characteristics (e.g., age, sex, diabetes, or genetic markers) is an important dimension of CER, as it provides evidence to “personalize” treatment decisions. Very large sample sizes are required to investigate potential variation in treatment effects across populations, which can be accomplished by pooling individual patient-level data from several trials. Further research could address the practical and methodologic challenges of collaborative studies that pool data from multiple studies of cardiovascular treatments.

Decision models and simulation studies are also valuable methods to foster CER. These techniques start with the best available evidence on a clinical question, and typically rely on the results of systematic reviews, meta-analyses, and pooling studies to provide the needed data. Models and simulation studies are particularly well suited to identify evidence gaps and the value of information by pinpointing which clinical factors have the greatest impact on the clinical effectiveness and cost-effectiveness of treatment alternatives. Studies can subsequently be designed to address the most important evidence gaps, which will be particularly valuable in meeting the CER objective of “comparing benefits and harms of alternative methods [of care].”

Proposed Framework for CER

The Workshop participants proposed a new framework for evidence development (Figure 1) to foster cardiovascular CER. This framework identifies gaps between the evidence needed for practice and the research available to support it. A “portfolio analysis” of the current state of knowledge, by using clinical guidelines, evidence reviews and decision models to identify key evidence gaps, could focus CER on key questions through a variety of research designs, including observational studies, clinical registries and randomized trials. This cycle of portfolio analysis, identification of key evidence gaps, and research addressing these gaps could repeat over time (Table 1).

Table 1
General Recommendations to Foster Comparative Effectiveness Research

The proposed framework (Figure 1) involves a multiple stakeholder process to identify questions for CER, consistent with recent changes on the political as well as scientific landscape. For example, the creation of the new Patient-Centered Outcomes Research Institute suggests that the model for setting research priorities may be changing from an investigator and industry driven process to one with broader patient and clinician input.

The proposed framework implies that knowledge gaps could be identified through systematic reviews of existing evidence, meta-analyses and decision analytic models. These types of studies could be efficient and timely opportunities to focus on important comparative effectiveness studies (Table 1).

The proposed framework also implies that interactions among potential funders and stakeholders, such as the Agency for Healthcare Research and Quality, the Centers for Medicare and Medicaid Services, and the Centers for Disease Control and Prevention could help foster CER focused on cardiovascular disease. The Cochrane Collaboration is an international organization that conducts systematic reviews and promotes methodologic development, and could assist in assessing evidence gaps and areas of focus for research. The professional societies (e.g., the American College of Cardiology, the American Heart Association, and the Society of Thoracic Surgeons) now operate clinical registries that may be well suited to cardiovascular CER, and also develop clinical guidelines, performance measures, and appropriate use criteria that could potentially translate CER findings into practice.

Expand the Scope of Discovery

CER focuses on developing scientific knowledge that will be useful during the course of patient care. The creativity and innovation that characterize investigator-initiated research have served medical science well. However, since the research questions of CER focus on addressing the needs of practitioners and patients, the scope of discovery may need to expand to include not only specific topics of interest to individual investigators and experts, but also topics driven by clinical evidence gaps.

The implications of developing a CER portfolio that focuses on knowledge gaps are far reaching. The evolution of a research question from model to meta-analysis to mega-trial to implementation underscores the contributions of methodologic research as well as research on specific clinical questions. While some clinical questions may be addressed in large randomized clinical trials, other pivotal questions may not be amenable to randomized studies, and therefore require alternative methods.

Create a Culture of Research

The broad scope of CER implies similar breadth in thinking about the research enterprise, such that the development of new knowledge about optimal practice becomes an intrinsic part of the health care system. Since CER aims to address the needs of patients and providers, health systems, providers and patients should therefore embrace the need to perform CER and recognize its value. A culture of research may be fostered when it is recognized that there is uncertainty regarding what constitutes optimal care, so that alternative forms of management may be reasonable and acceptable – the concept of clinical equipoise. While clinical systems and caregivers face challenges incorporating research as part of daily practice, they should recognize that improving the quality of care through discovery and learning from experience should be part of their professional and institutional obligations. Integrating research into routine care might be further fostered by focusing on quality improvement, aligning incentives within health systems, and, in the larger health care enterprise, by establishing policies such as coverage with evidence development and pay for performance.

The collection of observational data as part of a prospective clinical registry is an example of how research can be incorporated into daily practice. Clinical registries cover only specific clinical populations, however, and a broader knowledge base could advance CER. Fully interoperable electronic health records, with promotion of standardized clinical terminology, would facilitate CER, particularly if electronic health records, clinical registries, image repositories, clinical trial data and longitudinal claims can be linked to create study cohorts. To this end, further development of health informatics and its application to cardiovascular disease is an opportunity to advance CER.

Nurture the National CER Workforce

Individuals from many distinct backgrounds are needed to meet the challenges of CER, ranging from skilled investigators to inquisitive clinicians, knowledgeable patients, and practical methodologists. Thus, “team science” is integral to the success of CER. Paralleling the need for more diverse research partners is the opportunity to collaborate with nontraditional research venues and partners, including those usually focused only on care delivery or education, rather than scholarly research.

As CER is relatively new field, there is a great need for more investigators who are well-trained in its methods. Training, mentoring, and professional development programs aimed specifically at expanding the pool of investigators skilled in the methods of cardiovascular CER are possible ways to cultivate this field. Furthermore, nontraditional stakeholders may be valuable contributors to the development of the portfolio of CER studies.


The Workshop addressed two broad questions: 1) how might the NHLBI foster CER related to cardiovascular disease in general, and 2) how might the NHLBI respond to the specific CER priorities in the area of cardiovascular disease identified by the Institute of Medicine (IOM). In this section, we summarize the Workshop participants’ recommendations on the second broad question—what types of studies could potentially address the IOM priorities (1)? The Workshop participants were charged to identify examples of one or two study ideas for each IOM area.

NHLBI convenes working groups of experts to provide recommendations and input on specific topic areas, which the Institute then carefully reviews. The suggestions of this Workshop represent a list of important areas for investigator-initiated and/or Institute-initiated projects. The Institute carefully considers these and other recommendations as it sets its priorities and attempts to maintain a balanced portfolio across its entire mission; no NHLBI funding commitment is made or implied by inclusion of the topics in the report of this Workshop. The Workshop recognized that the NHLBI is particularly able to organize CER studies free of conflicts of interest related to specific drugs, devices, or management strategies, and has great experience in conducting comparative studies of alternative management strategies to treat cardiovascular disease.

The IOM priority areas discussed at the Workshop include:

  • Compare the effectiveness of treatment strategies for atrial fibrillation including surgery, catheter ablation, and pharmacologic treatment;
  • Compare the effectiveness of anticoagulant therapies (e.g., low-intensity warfarin, aspirin, injectable anticoagulants) for patients undergoing procedures;
  • Compare the effectiveness of treatment strategies for vascular claudication (e.g., medical optimization, smoking cessation, exercise, catheter-based treatment, open surgical bypass);
  • Compare the effectiveness of aggressive medical management and percutaneous coronary interventions in treating stable coronary disease for patients of different ages and with different comorbidities;
  • Compare the effectiveness of innovative treatment strategies (e.g., cardiac resynchronization, remote physiologic monitoring, pharmacologic treatment, novel agents such as CRF-2 receptors) for congestive heart failure;
  • Compare the effectiveness of different treatment strategies (e.g., modifying target levels for glucose, lipid, or blood pressure) in reducing cardiovascular complications in newly diagnosed adolescents and adults with Type 2 diabetes;
  • Compare the effectiveness of traditional risk stratification for coronary heart disease and noninvasive imaging (using coronary artery calcium, carotid intima media thickness, and other approaches) on outcomes;
  • Compare the effectiveness of computed tomography (CT) angiography and conventional angiography in assessing coronary stenosis in patients at moderate pretest risk of coronary artery disease.

Atrial Fibrillation

Atrial fibrillation is a highly prevalent condition associated with increased cardiovascular mortality and a high risk of stroke. The NHLBI has previously supported comparative treatment trials for atrial fibrillation, including the completed AFFIRM trial (Atrial Fibrillation Follow-Up Investigation of Rhythm Management) (4) and the ongoing CABANA trial (Catheter Ablation Versus Antiarrhythmic Drug Therapy for Atrial Fibrillation Trial, NCT00911508). The NHLBI is also funding a clinical registry of atrial fibrillation in two large integrated health plans to address specific comparative effectiveness research issues, which expands on a previous atrial fibrillation registry (25) and epidemiologic studies of atrial fibrillation. Despite these and other studies of atrial fibrillation, key knowledge gaps remain, including 1) the generalizability of trials to the larger population of patients with atrial fibrillation, 2) whether the treatment approach ought to vary depending on the sub-type of atrial fibrillation, and 3) the effect of treatments on expanded outcomes, such as stroke, dementia, heart failure, quality of life, health care utilization, and cost-effectiveness.

The Workshop participants recommend that one approach to fill these gaps could be a robust clinical registry with broader representation of patients with atrial fibrillation, with a particular focus on patients with new onset disease, to better define risks for adverse outcomes in relation to patient characteristics, sub-type of atrial fibrillation, biomarkers and treatment. In conjunction with the registry, a comprehensive decision model of atrial fibrillation management, as outlined in Figure 2, could be used to identify priority areas for additional CER studies, including clinical trials. There is also the potential to build upon ongoing studies of other conditions and treatments by adding atrial fibrillation as a secondary outcome measure.

Figure 2
Evidence and data portfolio management for AF

Anticoagulant Therapies

Anticoagulant and antithrombotic therapies are increasingly used to prevent thromboembolism in patients with atrial fibrillation, deep vein thrombosis, or pulmonary embolism. Management of these therapies at the time of invasive procedures and surgery poses a difficult problem in balancing the bleeding risk due to treatment and the thromboembolic risk due to the underlying disease (26). Furthermore, immobilization after surgery promotes venous thromboembolism, which is particularly increased by hip or knee joint replacement procedures. Management decisions have been further complicated by the introduction of several novel anticoagulant drugs that lack specific antidotes and whose anticoagulant intensity cannot be reliably assessed by laboratory tests. The NHLBI is currently supporting the BRIDGE trial (Effectiveness of Bridging Anticoagulation for Surgery, NCT00786474), and the GIFT trial (Genetics Informatics Trial of Warfarin to Prevent DVT, NCT01006733), which tests low intensity warfarin properties following orthopedic surgery.

The Workshop participants felt that a large “real-world” registry of patients undergoing specific surgical procedures would be of benefit to document risk factors for bleeding, cardiac events and thromboembolism. This registry could be used to assess the effect of different treatments on those outcomes, particularly among patients under-represented in randomized trials (Web Figure 1). Assessment of adverse orthopedic outcomes such as joint hemorrhage, peri-prosthetic infection and repeat procedures, would provide critical information needed to balance risks and benefits of anticoagulant treatment. The Registry could capitalize on substantial practice variations to perform observational treatment comparisons, and assess the effects of treatment on cost, quality of life, and cost-effectiveness.

Peripheral Artery Disease

Lower extremity peripheral artery disease is a common, costly condition that is associated with high morbidity and mortality. The pathophysiology of exertional limb claudication is analogous to that of exertional angina pectoris, but claudication has not received as much attention as angina in either clinical investigation or the development of new drugs and devices. There have been relatively few CER studies of claudication treatments, apart from the ongoing NHLBI sponsored CLEVER trial (Claudication: Exercise Versus Endoluminal Revascularization, NCT00132743). CLEVER compares exercise therapy with endovascular stenting for treatment of a documented lesion in a specific proximal aortoiliac site that is amenable to stent therapy (27). CLEVER does not, however, address the larger question of whether a strategy of mechanical limb revascularization (using any combination of endovascular techniques and surgery) leads to better clinical outcomes than the alternative strategy of optimal medical management (including exercise, lifestyle modification and drug therapy). A large practical randomized trial comparing these distinct approaches to claudication could address this priority area (Web Figure 2). This trial could enroll relatively unselected patients with claudication, whether new onset or after prior revascularization, without requiring prior angiography or any specific anatomic findings. The major outcomes of this trial could be patient functional capacity and quality of life, with secondary outcomes of major clinical complications, cardiovascular risk factor control, medical care utilization and cost, and cost-effectiveness.

Severe limb ischemia, including acute or chronic critical limb ischemia syndromes, represents another major clinical manifestation of peripheral artery disease, in which tissue necrosis is threatened due to reduced resting blood flow. There are newer therapeutic options for severe limb ischemia, but few reliable data on their long-term clinical outcomes or comparative effectiveness. Severe limb ischemia is particularly suitable for the process of priority setting outlined in Figure 1: performing a systematic review of evidence, modeling to identify critical parameters, and collecting observational data on epidemiology, treatment patterns, and determinants of clinically important outcomes. Establishment of a clinical registry of patients with severe limb ischemia could foster CER on this topic by identifying patient, provider, and treatment predictors of outcome. Such a registry could provide the basis for initiating subsequent, targeted clinical trials of evaluation and management strategies for severe limb ischemia.

Stable Ischemic Heart Disease

Coronary artery disease is well recognized as a major health problem in the United States, and has been the subject of numerous clinical investigations. The NHLBI has sponsored pivotal CER clinical trials, including CASS (28) (the Coronary Artery Surgery Study) and BARI 2D (7) (Bypass Angioplasty Revascularization Investigation Two, Diabetes, NCT00006305)to compare coronary revascularization with medical therapy among patients with ischemic heart disease. The NHLBI has also sponsored trials comparing bypass surgery with coronary angioplasty (BARI (29) and EAST (9)). Despite extensive investigation in this field, numerous knowledge gaps persist about optimal management of patients with stable ischemic heart disease. In particular, prior clinical trials have required knowledge of the coronary anatomy prior to randomization, but the decision to perform an invasive coronary angiogram has often been tantamount to the decision to perform coronary revascularization. The NHLBI just announced funding for ISCHEMIA (International Study of Comparative Health Effectiveness with Medical and Invasive Approaches), a trial that will randomize patients with stable coronary disease and objective evidence of myocardial ischemia to an invasive strategy of early coronary angiography or a conservative strategy of initial optimal medical therapy, with angiography reserved for development of refractory symptoms or a clinical event (Web Figure 3), in order to address a need for further evidence on this important decision point.

In addition to a large practical trial of coronary revascularization, many questions remain about how to define and deliver optimal medical management for patients with stable ischemic heart disease. In particular, it has been challenging to promote drug adherence and behavior change (diet, exercise, smoking cessation) in the setting of a busy outpatient practice. One approach to consider is to apply insights from behavioral economics to investigate the effect of economic incentives to clinicians (structuring of payment) or patients (copayments for drugs or visits, the costs of improving exercise and diet) on clinical outcomes.

As a third consideration, formal analysis of evidence gaps and opportunities in stable ischemic heart disease using the processes of evidence review, model building, and analysis of clinical registries (Figure 1) would allow the identification of additional opportunities for CER in this area.

Heart Failure

Heart failure continues to be the most common reason for hospital admission among Americans 65 years of age and older, and the prevalence of heart failure has continued to rise, even though other forms of heart disease have been declining. The NHLBI has sponsored numerous CER investigations in heart failure, including SOLVD (30) (Studies of Left Ventricular Dysfunction), SCD-HeFT (11) (Sudden Cardiac Death in Heart Failure, NCT00000609) and, more recently, STICH (12) (Surgical Treatment of Ischemic Heart Failure, NCT00023595). Many other studies of the effects of drugs and devices on clinical outcome in patients with heart failure have been sponsored by industry.

A hallmark of heart failure is that many patients are frequently re-hospitalized to treat exacerbations of the disease, at great expense to the system and considerable distress to patients and their families. There are major gaps in knowledge about how to address this problem, so the Workshop participants proposed that one study to consider is enhanced disease management and transition of care with monitoring (remote or biomarker) as a means of improving clinical outcomes. A large practical trial (Figure 3) could enroll unselected patients with heart failure at the time of hospital discharge, and randomize them either to usual care or to enhanced disease management with tailored therapy guided by remote monitoring (e.g., weight, heart rate, blood pressure, biomarkers). An associated registry of patients with heart failure could be established in conjunction with the randomized trial to collect additional data on the full spectrum of patients with heart failure, and thereby assess the generalizability of the trial results.

Figure 3
Proposed cluster randomized trial for the management of heart failure

The proposed study could use a cluster randomized design (31), in which clinical sites rather than individuals are randomized. The cluster randomized design is well suited to evaluation of interventions that target behavior or processes of care. Cluster randomized trials are particularly useful when it is difficult to conceal the nature of the intervention from the clinic staff, and to mask patients to the intervention. The novel research designs that may be required for CER entail unique challenges (32). Despite such challenges, the Workshop participants encouraged use of such novel research designs, as they provide several advantages, including simpler patient recruitment and “real life” data directly applicable to clinical practice.


The incidence of diabetes continues to rise, driven in large part by the epidemic of obesity in the United States. Most patients with diabetes die of heart disease, yet the optimal approach to prevention and treatment of cardiovascular disease in patients with diabetes is not well established. Despite the recent publications of the results of the ACCORD (33), ADVANCE (34), VADT (35), long-term follow-up from UKPDS and STENO-2 (36), and the ongoing Look AHEAD trial (NCT00017953), the comparative effectiveness of tight glycemic control versus more liberal control of diabetes, of lifestyle management versus early drug treatment, of different initial drug treatments, and of different systems of care are all uncertain in patients with newly diagnosed diabetes. The Workshop participants felt that the IOM priority could potentially be addressed by the following two trials.

The first trial could compare two strategies for initial treatment in patients with newly diagnosed Type 2 diabetes: intensive lifestyle intervention versus immediate metformin (Web Figure 4). A third arm of this trial could include both interventions. A second trial could focus on the comparative effectiveness on cardiovascular outcomes of using different classes of glucose lowering drugs as second agents in patients who fail metformin monotherapy (Web Figure 5). Metformin therapy is the current evidence-based standard for initial pharmacotherapy, but multiple medications are usually needed to maintain durable hemoglobin A1C control. In this trial, participants would be randomized to second stage treatment with incretin mimetics versus a thiazolidinedione versus a secretagogue, with insulin withheld for third line use.

Potential outcomes in each study could include non-invasive measures of atherosclerosis, (e.g., coronary artery calcium), progression of diabetes, quality of life, and satisfaction with treatment. In light of unexpectedly higher rate of total mortality in the more intensively treated group in ACCORD, any early differences in subclinical disease markers or surrogate outcomes ought to be confirmed by subsequent assessments of clinical outcomes. Clinical cardiovascular events (cardiac death, myocardial infarction, and stroke) could be potential outcomes as well, although the group recognized that potentially low event rates might require large sample sizes and long follow-up. Depending on the outcome and participant characteristics, this simple trial could range from as few as 1,000–2,000 to as many as 40,000–50,000 participants.

A registry could also be useful for post-marketing surveillance of cardiovascular disease events among patients prescribed drugs to treat diabetes.

Risk Stratification for Coronary Artery Disease

The current paradigm for prevention of coronary disease in asymptomatic adults is based on individual risk assessment using a standard risk predictor (such as the Framingham Risk Score), and intervening with drug therapy among the individuals identified as being at highest cardiovascular risk (37). The optimal management for the patient at intermediate risk in this paradigm remains uncertain. Additional risk stratification of individuals at intermediate cardiac risk, with drug treatment of individuals with “positive markers,” may substantially improve outcomes. There are many candidate risk markers that could be applied at the population level, but coronary calcification on computed tomography (CT) is quite promising, as it provides high levels of incremental information and risk reclassification. Consequently, the Workshop participants considered a large trial of coronary calcium screening among individuals found to be at intermediate risk on conventional evaluation (Web Figure 6), with individuals found to have elevated coronary calcium receiving intensive drug therapy and lifestyle modification, and individuals without elevated coronary calcium scores undergoing lifestyle modifications alone. The trial could follow patients for up to five years, with the primary outcome of major cardiac events. Secondary outcomes could include quality of life, adherence to drug and lifestyle management, cost, and cost-effectiveness.

The current paradigm of individual risk assessment includes pharmacologic therapy for those with elevated Framingham risk scores, but most of the high risk group will not have a cardiovascular event in the subsequent ten years. Additional testing of individuals with high Framingham risk scores might identify a subgroup that would not benefit sufficiently to justify lifelong pharmacologic therapy. Optimal management of these individuals might be improved by imaging to document the severity of the underlying disease process. The Workshop participants also suggest that a comprehensive evidence review and decision modeling of the application of imaging the disease substrate (e.g. by CT coronary angiography, or coronary calcium measurement) could help to further stratify risk and guide therapy. This investigation would fit within the framework outlined in Figure 1, and assist in determining the potential value of larger CER studies to address this question.

CT Coronary Angiography

Coronary angiography can now be performed non-invasively with recent generation CT scanners, and in several case series CT angiography has demonstrated high sensitivity and good specificity when compared with invasive coronary angiography as a reference standard (38, 39). These data suggest that CT angiography may be very useful in the evaluation of patients with symptoms of coronary disease. The effect of CT angiography on clinical outcomes is uncertain, however, since visualization of coronary obstructions may well lead to unnecessary or inappropriate coronary revascularization, the value of detecting incidental non-cardiac findings is unknown, and the ionizing radiation from CT scanning may lead to adverse events. Consequently, outcome-based studies of the comparative effectiveness of CT coronary angiography and alternative diagnostic strategies would address an important gap in the evidence. The NHLBI is currently funding the PROMISE trial (Prospective Multicenter Imaging Study for the Evaluation of Chest Pain; NCT 01174550), which is randomizing symptomatic patients suspected of having coronary artery disease to either usual stress testing (functional) or CT angiography (anatomic).

Another study to address this IOM priority could be a large clinical trial of patients with symptoms suggestive of coronary disease without high risk features, in which patients would be randomized to either invasive coronary angiography or to CT coronary angiography after a stress test that had either inconclusive or “not high risk” results (Web Figure 7). Patients in the invasive angiography arm of the study would receive coronary revascularization according to current usual care, whereas in the CT angiography arm of the study only patients with specific anatomic findings (left main disease, severe three-vessel disease) would be recommended to receive coronary revascularization (and invasive angiography if needed to further define coronary anatomy). The primary endpoint of this trial study would be major cardiovascular events (cardiac death, myocardial infarction, stroke), for which the non-inferiority of CT angiography would be tested. Secondary endpoints could include quality of life, cost, and cost-effectiveness, for which superiority of CT angiography would be tested. The Workshop participants suggest that any research infrastructure created for the proposed trial could be used as an “advanced cardiovascular imaging network” to conduct efficiently other CER studies of imaging, as has been undertaken by the CAIN (Canadian Atherosclerosis Imaging Network), and the MITNEC (Medical Imaging Trials Network of Canada) networks.

The Workshop also identified a clinical registry of CT coronary angiography procedures, ideally as an extension of the ongoing National Cardiovascular Disease Registries sponsored by the American College of Cardiology, as a further opportunity to promote CER on this topic.


The recent recognition of the importance of comparative effectiveness research places increasing emphasis on studies that directly inform and improve patient care. The Workshop on Cardiovascular Comparative Effectiveness Research was designed to propose approaches for the NHLBI to consider, utilizing both an overall framework for CER and specific study designs as examples.

The approach to actionable research outlined by workshop participants implies a cycle of research and its application (Figure 1). The four important linked steps in this cycle are: 1) the prospective articulation of research questions based on identifying gaps in knowledge about optimal patient care, incorporating input of stakeholders, including patients, 2) the development of evidence by a variety of research methods to address key evidence gaps, 3) application of the evidence in practice guidelines and standards of care, and 4) determination whether quality of care and patient outcomes are improved.

Several important points follow logically from this vision, beginning with consideration of a new research paradigm. Following that, relationships can be created to identify important research questions, and infrastructure developed to perform studies, such as large scale registries and CER trials, which can foster translation of research results into practice. The training and development of CER researchers, and the creation of a culture of learning health care systems, are important steps in furthering CER. Finally, clear identification of evidence gaps and key questions will help guide the performance of high quality research.

The Institutes of Medicine has begun this process by identifying several research priorities in cardiovascular disease. This Workshop has advanced this process by proposing research ideas to address each of the eight IOM priority areas. These range from systematic reviews, to secondary analyses of existing data, to registries, to large scale clinical trials; each could add important evidence needed to improve patient care. The Workshop participants hope that these recommendations will provide valuable information to investigators and funding agencies as they seek to advance the nation’s commitment to cardiovascular comparative effectiveness research.

Workshop Participants

Supplementary Material


The authors would like to acknowledge the NHLBI staff, Drs. Denise Bonds and Jean Olson, as well as student volunteer, Mr. Jared Lucas, for their efforts in the planning and conduct of this workshop.


comparative effectiveness research
computed tomography
Institute of Medicine
National Heart Lung and Blood Institute
randomized controlled trial


Dr. Selby joined the Patient-Centered Outcomes Research Institute (PCORI) in July 2009. The opinions expressed in this paper do not necessarily reflect those of PCORI.


Co-authors Nakela Cook, Barbara Wells, and Michael Lauer are full-time employees of the NHLBI. The views expressed in this manuscript do not necessarily represent the views of the NHLBI, National Institutes of Health, or any other government entity.

This article was co-authored by Dr. Selby while he was employed by Kaiser Permanente. In July 2011, Dr. Selby became Executive Director of the Patient Centered Outcomes Research Institute (“PCORI”). This article does not necessarily represent the views or position of PCORI.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.


1. IOM (Institute of Medicine) Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.
2. Concato J, Peduzzi P, Huang G, O’Leary TJ, Kupersmith J. Comparative effectiveness research: What kind of studies do we need? J Investig Med. 2010;58:764–769. [PubMed]
3. The ACCORD Study Group. Long-term effects of intensive glucose lowering on cardiovascular outcomes. N Engl J Med. 2011;364:818–828. [PMC free article] [PubMed]
4. The Atrial Fibrillation Follow-up Investigation of Rhythm Management (AFFIRM) Investigators. A comparison of rate control and rhythm control in patients with atrial fibrillation. N Engl J Med. 2002;347:1825–1833. [PubMed]
5. The ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group. Major outcomes in high-risk hypertensive patients randomized to angiotensin-converting enzyme inhibitor or calcium channel blocker vs diuretic. The Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) JAMA. 2002;288:2981–2997. [PubMed]
6. The BARI Investigators. The final 10-year follow-up results from the BARI randomized trial. J Am Coll Cardiol. 2007;49:1600–1606. [PubMed]
7. The BARI 2D Study Group. A randomized trial of therapies for type 2 diabetes and coronary artery disease. N Engl J Med. 2009;360:2503–2515. [PMC free article] [PubMed]
8. Alderman EL, Bourassa MG, Cohen LS, et al. Ten-year follow-up of survival and myocardial infarction in the randomized Coronary Artery Surgery Study. Circulation. 1990;82:1629–1646. [PubMed]
9. King SB, Kosinski AS, Guyton RA, Lembo NJ, Weintraub WS. Eight-year mortality in the Emory Angioplasty versus Surgery Trial (EAST) J Am Coll Cardiol. 2000;35:1116–1121. [PubMed]
10. Hochman JS, Lamas GA, Buller CE, et al. Coronary intervention for persistent occlusion after myocardial infarction. N Engl J Med. 2006;355:2395–2407. [PMC free article] [PubMed]
11. Bardy GH, Lee KL, Mark DB, et al. Amiodarone or an implantable cardioverter-defibrillator for congestive heart failure. N Engl J Med. 2005;352:225–237. [PubMed]
12. Velazquez EJ, Lee KL, Deja MA, et al. Coronary-artery bypass surgery in patients with left ventricular dysfunction. N Engl J Med. 2011;364:1607–1616. [PMC free article] [PubMed]
13. Manson JE, Hsia J, Johnson KC, et al. Estrogen plus progestin and the risk of coronary heart disease. N Engl J Med. 2003;349:523–534. [PubMed]
14. Myers WO, Blackstone EH, Davis K, Foster ED, Kaiser GC. CASS Registry. Long term surgical survival. J Am Coll Cardiol. 1999;33:488–498. [PubMed]
15. Ray WA. Evaluating medication effects outside of clinical trials: New-user designs. Am J Epidemiol. 2003;158:915–920. [PubMed]
16. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41–55.
17. Glynn RJ, Schneeweiss S, Stürmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98:253–259. [PMC free article] [PubMed]
18. Robins JM, Hernán MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology. 2000;11:550–560. [PubMed]
19. Newhouse JP, McClellan M. Econometrics in outcomes research: The use of instrumental variables. Annu Rev Public Health. 1998;19:17–34. [PubMed]
20. Selby JV, Fireman BH, Lundstrom RJ, et al. Variation among hospitals in coronary angiography practices and outcomes after myocardial infarction in a large health maintenance organization. N Engl J Med. 1996;335:1888–1896. [PubMed]
21. Johnston SC, Henneman T, McCulloch CE, van der Lann M. Modeling treatment effects on binary outcomes with grouped-treatment variables and individual covariates. Am J Epidemiol. 2002;156:753–760. [PubMed]
22. Dickersin K. To reform U.S. health care, start with systematic reviews. Science. 2010;329:516–517. [PubMed]
23. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: Treatments for myocardial infarction. JAMA. 1992;268:240–248. [PubMed]
24. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: Combining direct and indirect evidence. BMJ. 2005;331:897–900. [PMC free article] [PubMed]
25. Go A, Hylek EM, Phillips KA, et al. Prevalence of diagnosed atrial fibrillation in adults: National implications for rhythm management and stroke prevention: The AnTicoagulation and Risk factors In Atrial fibrillation (ATRIA) study. JAMA. 2001;285:2370–2375. [PubMed]
26. Douketis JD, Berger PB, Dunn AS, et al. The perioperative management of antithrombotic therapy: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2008;133:299–339. [PubMed]
27. Murphy TP, Hirsch AT, Ricotta JJ, et al. The CLaudication: Exercise Vs. Endoluminal Revascularization (CLEVER) study: Rationale and methods. J Vasc Surg. 2008;47:1356–1363. [PMC free article] [PubMed]
28. CASS Principal Investigators and Their Associates. Coronary Artery Surgery Study (CASS): A randomized trial of coronary artery bypass surgery. Survival data. Circulation. 1983;68:939–950. [PubMed]
29. The Bypass Angioplasty Revascularization Investigation (BARI) Investigators. Comparison of coronary bypass surgery with angioplasty in patients with multivessel disease. N Engl J Med. 1996;335:217–225. [PubMed]
30. Effect of enalapril on survival in patients with reduced left ventricular ejection fractions and congestive heart failure. The SOLVD Investigators. N Engl J Med. 1991;325:293–302. [PubMed]
31. Eldridge S, Ashby D, Bennett C, Wakelin MFG. Internal and external validity of cluster randomised trials: Systematic review of recent trials. BMJ. 2008;336:876–880. [PMC free article] [PubMed]
32. Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D, et al. Rethinking randomized clinical trials for comparative effectiveness research: The need for transformational change. Ann Intern Med. 2009;151:206–209. [PubMed]
33. The Action to Control Cardiovascular Risk in Diabetes Study Group. Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med. 2008;358:2545–2559. [PMC free article] [PubMed]
34. The ADVANCE Collaborative Group. Intensive blood glucose control and vascular outcomes in patients with type 2 diabetes. N Engl J Med. 2008;358:2560–2572. [PubMed]
35. Duckworth W, Abraira C, Moritz T, et al. Glucose control and vascular complications in veterans with type 2 diabetes. N Engl J Med. 2009;360:129–139. [PubMed]
36. Gæde P, Lund-Andersen H, Parving HH, Pedersen O. Effect of a multifactorial intervention on mortality in type 2 diabetes. N Engl J Med. 2008;358:580–5919. [PubMed]
37. Greenland P, Alpert JS, Beller GA, et al. 2010 ACCF/AHA guideline for assessment of cardiovascular risk in asymptomatic adults: A report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2010;56:e50–103. [PubMed]
38. Mowatt G, Cummins E, Waugh N, et al. Systematic review of the clinical effectiveness and cost-effectiveness of 64-slice or higher computed tomography angiography as an alternative to invasive coronary angiography in the investigation of coronary artery disease. Health Technol Assess. 2008;12:iii–iv. ix–143. [PubMed]
39. Mark DB, Berman DS, Budoff MJ, et al. ACCF/ACR/AHA/NASCI/SAIP/SCAI/SCCT 2010 expert consensus document on coronary computed tomographic angiography: A report of the American College of Cardiology Foundation Task Force on Expert Consensus Documents. J Am Coll Cardiol. 2010;55:2663–2699. [PubMed]