Search tips
Search criteria 


Logo of ajrccmIssue Featuring ArticlePublisher's Version of ArticleSubmissionsAmerican Thoracic SocietyAmerican Thoracic SocietyAmerican Journal of Respiratory and Critical Care Medicine
Am J Respir Crit Care Med. 2011 October 1; 184(7): 848–856.
Published online 2011 October 1. doi:  10.1164/rccm.201104-0634WS
PMCID: PMC3265273

Comparative Effectiveness Research in Lung Diseases and Sleep Disorders

Recommendations from the National Heart, Lung, and Blood Institute Workshop
Tracy A. Lieu,1,2 David Au,3 Jerry A. Krishnan,4 Marc Moss,5 Harry Selker,6 Andrea Harabin,7 Virginia Taggart,corresponding author7 and Alfred Connors8
1Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts
2Division of General Pediatrics, Children's Hospital Boston, Boston, Massachusetts
3Health Services Research and Development, VA Puget Sound Health Care System and the Division of Pulmonary and Critical Care Medicine, University of Washington and Department of Veterans Affairs, Seattle, Washington
4Asthma and COPD Center, Department of Medicine and Department of Health Studies, University of Chicago, Chicago, Illinois
5Department of Medicine, University of Colorado School of Medicine, Denver, Colorado
6Tufts Medical Center Institute for Clinical Research and Health Policy Studies, Tufts University Clinical and Translational Science Institute, Boston, Massachusetts
7Division of Lung Diseases, National Heart, Lung and Blood Institute, National Institutes of Health, Bethesda, Maryland
8Case Western Reserve University School of Medicine at MetroHealth Medical Center, Cleveland, Ohio
and the Comparative Effectiveness Research in Lung Diseases Workshop Panel*


The Division of Lung Diseases of the National Heart, Lung, and Blood Institute (NHLBI) held a workshop to develop recommendations on topics, methodologies, and resources for comparative effectiveness research (CER) that will guide clinical decision making about available treatment options for lung diseases and sleep disorders. A multidisciplinary group of experts with experience in efficacy, effectiveness, implementation, and economic research identified (a) what types of studies the domain of CER in lung diseases and sleep disorders should include, (b) the criteria and process for setting priorities, and (c) current resources for and barriers to CER in lung diseases. Key recommendations were to (1) increase efforts to engage stakeholders in developing CER questions and study designs; (2) invest in further development of databases and other infrastructure, including efficient methods for data sharing; (3) make full use of a broad range of study designs; (4) increase the appropriate use of observational designs and the support of methodologic research; (5) ensure that committees that review CER grant applications include persons with appropriate perspective and expertise; and (6) further develop the workforce for CER by supporting training opportunities that focus on the methodologic and practical skills needed.

Keywords: randomized controlled trials, observational studies, implementation, study designs, methodology

Recent congressionally mandated federal efforts to establish a “robust comparative effectiveness enterprise” have generated definitions, priorities, and organizational mechanisms for conducting comparative effectiveness research (CER) (13). The National Institute of Health's commitment to CER is reflected in its history of comparative clinical trials that have profoundly influenced clinical practice as well as its current involvement in shaping the CER enterprise (4).

The NIH currently uses the Federal Coordinating Council definition of CER (5, 6): “…the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in ‘real world settings’” (5). Operationalizing this definition to shape a specific research agenda poses numerous challenges, including defining where CER would best fill the gaps in clinical evidence for particular lung diseases or sleep disorders, understanding how to use innovative study designs, obtaining large sample sizes and data sets to examine subgroup variations in responses, using electronic health records and integrated data systems to increase efficiency, and identifying the role for observational studies.

The Division of Lung Diseases (DLD), National Heart, Lung, and Blood Institute (NHLBI) convened a workshop in September 2010 to discuss the role of CER in DLD-supported research and to make recommendations to help assure that CER research in lung diseases and sleep disorders meets the challenges and opportunities noted above. Participants included clinical investigators in asthma, chronic obstructive pulmonary disease (COPD), acute respiratory distress syndrome (ARDS), pediatric lung diseases, and sleep disorders, as well as methodologists with expertise relevant to CER.

Before the workshop, participants completed a survey asking about priority topics for CER lung diseases overall and in their respective fields, the barriers to conducting CER, and the most important steps to advance CER in their fields. Results were used to provide a framework for the meeting agenda. At the workshop, participants were divided into subgroups to design a mock CER intervention trial and a mock CER observational study. This activity provided a practical consideration of the themes that emerged from the morning's more general discussion of CER in lung disease. Group discussion of the mock trials and the key issues and concerns that emerged in their development led to the specific recommendations to advance CER in lung diseases presented in this report.

The Domain of CER in Lung Diseases

Efficacy Versus Effectiveness

The efficacy of a treatment refers to its ability to improve patient outcomes in a population likely to respond to the treatment, with strict adherence to the treatment protocol, in specialized centers and with highly motivated providers. An efficacy experiment, most often a randomized controlled trial (RCT), answers the question, “Can this intervention achieve a beneficial effect under highly favorable circumstances?” Generally, efficacy research precedes effectiveness research. The effectiveness of a treatment refers to its ability to improve patient outcomes in more real-world settings, for example, with a more heterogeneous population exhibiting the normal spectrum of disease, comorbid illness, ethnicity, and/or age, with average compliance with treatment protocols and medications, and/or in a wider spectrum of practice settings. Effectiveness research, even with large study populations in community settings, often includes randomized controlled study designs.

What Is CER?

CER is within the continuum of effectiveness research. CER is characterized by the study of two or more interventions using a variety of study designs, which allows comparative evaluation of the effectiveness of these interventions in real-world settings, for diverse patient populations and subgroups, assessing a full range of outcomes to address questions important to patients, caregivers, and other key stakeholders. Examples of such outcomes include clinical events, measures of disease control, symptom status, activity limitations, work or school absenteeism or lost productivity (presenteeism), and disease impact or burden. There are a number of independent measures of these outcomes that are not limited to quality-of-life questionnaires. Not all effectiveness research is CER. For example, an RCT comparing a medication or other therapy to placebo, even if conducted in large populations under real-life conditions, would be an effectiveness study but would not meet criteria for CER.

The NIH working definition of comparative effectiveness research is as follows: CER is the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in “real-world” settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.

  • To provide this information, comparative effectiveness research must assess a comprehensive array of health-related outcomes for diverse patient populations and subgroups.
  • Defined interventions compared may include medications, procedures, medical and assistive devices and technologies, diagnostic testing, behavioral change, and delivery system strategies.
  • This research necessitates the development, expansion, and use of a variety of data sources and methods to assess comparative effectiveness and actively disseminate the results (5).

The workshop participants discussed this definition and identified the following core elements to emphasize for CER in lung diseases and sleep disorders: (1) comparison of interventions or implementation strategies in real-world settings, (2) assessment of a comprehensive array of health-related outcomes, (3) studies with diverse patient populations and attention to important subgroups, (4) use of a variety of observational or experimental methods.

Implementation research—the scientific study of how to promote the adoption of research findings (7)—should be supported to assure that effective interventions are adopted into practice without needless delays. For example, some implementation studies have evaluated interventions directed toward providers to improve clinical practices, such as audit and feedback, local opinion leaders, and educational outreach visits (810). Implementation research may also evaluate the effects of changes in health care systems or financing on translating research into practice. Workshop participants also recommended that costs should be considered using rigorous economic evaluation methodologies.

The workshop participants were concerned that the NIH emphasizes comparison of interventions of proven efficacy. However, there will be many CER questions about lung diseases and sleep disorders that may be best answered by comparing a proven intervention to usual care, and workshop participants felt the NIH definition should be interpreted to include this possible design as long as there is appropriate equipoise between the proven intervention and the active comparator of usual care. To include diverse patient populations, a number of different approaches may be needed, depending on the specific research questions addressed. The NIH definition's recommendation to use a variety of data sources and methods suggests support of, for example, innovative use of practice-based research networks, registries and pooled data from electronic medical records, and appropriate use of prospective and retrospective observational study designs.

CER Study Designs

CER can be conducted using interventional or observational study designs (Table 1). Arguably, randomized studies provide the strongest evidence about the effect of a treatment because they are less prone to bias than observational designs. They are not always feasible or affordable and usually require several years to produce results. Pragmatic trials (also called practical trials, which are randomized trials designed for typical, real-world settings and may include relaxed inclusion/exclusion criteria, relaxed simple and/or flexible protocols, and outcome measures of relevance to patients, physicians, and payers) and group-randomized trials (trials in which the subjects are assigned to intervention or control in groups, or clusters, defined by a common features) (11) are increasingly used approaches to increasing the size and efficiency of randomized designs (12, 13). Pragmatic trials achieve better generalizability than efficacy trials, at some cost to internal validity (12). Group-randomized trials are especially useful when individual-level randomization is not possible, and they are well suited to studies involving health care systems or multisite networks of providers, schools, or communities. When randomization is not feasible, novel approaches to the design and analyses of interventions, such as interrupted time series analysis, can be used in the setting of intervention studies.


Observational study designs are an important component of CER, and workshop participants endorsed increasing their use for lung diseases and sleep disorders. Observational designs can take advantage of natural experiments or existing variation in practice. They can be conducted retrospectively or prospectively. They often use existing computerized data; additional information can be collected via medical record review or surveys. Natural experiments may occur when there are changes in the availability of an intervention. In one example, Medicaid policies in one state were changed to limit access to psychotropic medications, enabling researchers to evaluate effects on health outcomes using interrupted time-series analysis (14). Conversely, a new treatment, such as pneumococcal conjugate vaccination of children, may become widely adopted at a specific point in time, affording investigators the chance to evaluate its effects (15).

Observational research is preferred when interventional studies are not feasible (for example, when randomization is not acceptable due to lack of equipoise) or when key outcomes are delayed and long-term follow-up would be impractical. Observational studies are more likely to allow timely comparisons in diverse populations and in real-world settings than randomized trials. In addition, they can contribute critical information to characterize actual practice in real-life settings and are very useful before the design of large RCTs to identify study questions that are likely to produce results that will change care. At times, they have helped create equipoise that allows RCTs to be performed (16).

An important limitation of observational study designs is the limited ability to infer causation. There are striking examples wherein multiple observational studies suggested one treatment but subsequent randomized studies showed opposite results (e.g., hormone replacement therapy for menopausal women). A related limitation is confounding by indication, in which treatment decisions are associated with patient characteristics that influence outcomes. Methods used to address confounding include propensity scores, instrumental variables, restriction, interrupted time-series designs, and regression discontinuity designs (1723). However, propensity scores do not control for unidentified confounders, and instrumental variable analyses may have limitations in this regard.

The workshop participants agreed that grant applications for CER in lung diseases might choose among a variety of methods and should provide a clear justification why the method chosen is best for the question being addressed. Applicants may also propose to apply different methodological approaches to evaluate whether results are consistent across them. Workshop participants also endorsed further methodological research to identify optimal existing strategies and to develop new approaches to common issues, such as adjustment for confounding in observational CER.

Criteria and Process for Setting Priorities Among Topics for CER

Criteria will be needed to set priorities for CER in lung diseases and sleep disorders. Workshop participants noted that priorities need to be set within three general areas: (1) identification of specific lung diseases and sleep disorders and research questions within them, (2) methods for CER, and (3) infrastructure for CER. Workshop participants endorsed the criteria for setting priorities defined by the Federal Coordinating Council on CER and listed in the NIH definition of CER and added specific comments for consideration in lung disease CER (Table 2).


CER is intended to address the expressed needs of end users, including patients (e.g., advocacy groups, community organizations), clinicians (e.g., professional organizations), health care providers (hospitals, health systems), and health care payers/policy makers (including insurers and the Centers for Medicare and Medicaid Services and the Veterans Administration). Examples of areas in which end-user involvement can enhance the CER enterprise, for both investigator-initiated and institute-prioritized research, include the prioritization of research questions, providing advice about the study design used to address the research question (e.g., selection of comparators, feasibility of conducting randomized or controlled clinical trials, quasi-experimental, or observational studies), and development of partnerships for data collection and for dissemination/implementation activities once study results are available. An active collaboration between those who develop, disseminate, and use the evidence generated from CER offers the greatest opportunity to provide actionable information to patients, clinicians, and other stakeholders, and therefore the basis to more rapidly translate CER findings into clinical practice.

Workshop experts recommended use of structured and transparent approaches to selecting priorities in CER. Various systematic methods for eliciting input from stakeholders have been developed, such as Analytic Hierarchy Process (24). Many methods have been applied primarily in business settings, but they could be adapted to apply to health research. Systematic approaches to setting research priorities offer an opportunity to understand the relative preferences of different stakeholder groups, as well as the criteria used by groups in prioritization. For example, the COPD Outcomes-based Network for Clinical Effectiveness and Research Translation (CONCERT) researchers have developed a collaboration with a diverse group of stakeholders (patient advocacy groups, professional organizations representing physicians, nurses, respiratory therapists, and social workers, health plans/insurers, quality improvement organizations, and the Joint Commission) to develop and prioritize a research agenda for CER in COPD (25, 26) CONCERT has also engaged stakeholders to inform the planning and design of CER studies responsive to this research agenda (e.g., selection of populations, interventions, comparators, outcomes), providing the opportunity to develop an evidence base that directly meets the expressed needs of end users.

Barriers to CER in Lung Diseases and Resources to Overcome Barriers

Presently, there are several general impediments and some barriers specific to lung diseases and sleep disorders for the accomplishment of CER. The group identified these barriers, considered how they affect the ability to perform CER, and discussed potential strategies to overcome these obstacles.

Identification of Patients and Therapies for Studies

Many pulmonary disorders and some sleep disorders meet the criterion of a rare disease, defined by the NIH as a disorder that affects fewer than 200,000 people in the United States, which will present a challenge for CER. For example, cystic fibrosis (CF) is a chronic and debilitating disorder that often results in premature morbidity and mortality, but it only affects an estimated 30,000 individuals. In addition, regional variability exists in the prevalence of some pulmonary disorders, including interstitial lung diseases (2729). The low prevalence and geographical heterogeneity of these diseases may hamper the ability to enroll these patients into pragmatic clinical trials.

The development of large national registries of patients with specific pulmonary diseases and sleep disorders might assist in the identification of patients for enrollment into clinical trials. Registries have generally been used for relatively rare diseases, such as CF. The information in the CF registry created more than 40 years ago by the CF Foundation allows caregivers and researchers to identify new health trends, design clinical trials for potential therapies, and identify patients for future trials. Few such registries currently exist for lung diseases and sleep disorders. To facilitate CER, it is necessary to develop more disease-specific registries, including developing methodologies and approaches for capturing key representative data on common complex pulmonary and sleep disorders.

Identification of large patient populations for CER in more common lung disorders can also be difficult, and systematic methods to aid recruitment are important. Clinical trial networks can help in identification of appropriate patients. The DLD has established a number of disease-specific networks that created the necessary infrastructure of clinical and associated support centers to enable the rapid development and conduct of multiple clinical protocols to evaluate the efficacy of promising diagnostic and therapeutic approaches. These clinical networks, which currently study asthma, interstitial lung disease, ARDS, and COPD, may be properly positioned to conduct CER if efforts are made to expand the patient pools and interventions outside of academic settings.

More recently, a large network of academic institutions supported by NIH Clinical and Translational Science Awards have led the development of a Web site for patients who are interested in participating in clinical research. The goal of this program, called ResearchMatch (30), is to link suitable patients with researchers who are looking for study participants. The NIH registry is another resource to help connect patients with appropriate studies.

Another barrier to the performance of CER is the lack of proven therapies for many pulmonary and sleep diseases. For example, there is a paucity of proven medical therapies for ARDS and interstitial lung diseases. Large pragmatic trials that compare different therapies may not yet be practical for certain lung and sleep diseases. The identification of treatments with proven efficacy with smaller efficacy trials may be necessary before trials of the size and magnitude necessary for CER should be performed.

Electronic Data

Administrative data sets, including claims data files kept by health plans and government payers, are a potentially powerful tool for CER (31). Clinical information obtained during routine clinical care provides a unique opportunity to examine the relative effectiveness of treatments. The information derived from such records are coded data (International Classification of Diseases-9 codes), including diagnostic and procedure codes, dispensing records for medications, and free text entries. Newer technology, such as natural language processing, will be needed to take full advantage of the variety of different types of data available, such as free text that occurs in electronic medical records.

The Agency for Healthcare Research and Quality (AHRQ) has funded several large projects to build and expand the nation's clinical electronic infrastructure to generate prospective, patient-centered outcomes for CER on therapeutics and tests. The infrastructure could be scaled up to include other organizations and data systems, with the goal of sustainable support for CER. The development of new methods and data governance approaches needed to enhance the national infrastructure for CER will be spearheaded by the Electronic Data Methods Forum, a new initiative led for AHRQ by AcademyHealth. Descriptions of these AHRQ-sponsored projects are available on the program Web sites (32, 33).

Effectiveness studies that compare different mechanisms of health care delivery may also benefit directly from clinically derived data that can identify populations for study and/or allow assessment of the delivery and effectiveness of care. However, for investigators to make valid inferences from studies using such administrative data, they need to understand the validity of all the data elements to be used, including variables on exposures, outcomes, and covariates.

The quality of administrative data depends on limitations and changes in the specificity of the criteria for diagnoses and on temporal changes related to financial reimbursement and regional coding practices. Internation Classification of Diseases-9 codes require additional investigation to evaluate and improve the reliability and validity of these measures in heterogeneous clinical settings even for common conditions, such as COPD, asthma, sleep apnea, and acute lung injury. Furthermore, laboratory values require a reference standard to be meaningful, because results are known to vary across laboratories and time. Defining common reference standards is important for other elements of the medical record, including medications in which these elements may be described by name or coded according to other common standards, such as National Drug Classification or Systematized Nomenclature of Medicine codes. Harmonization of data records may need to occur not only across administrative data sets but also across electronic medical records (EMRs) from different settings. This will require transforming native data from disparate EMRs into data that have common meaning across information platforms. This issue is being addressed using standardized data dictionaries, such as those developed by the Health Maintenance Organization Research Network (34), the Mini-Sentinel Initiative sponsored by the Food and Drug Administration (35), and the COPD DataHub, CONCERT (26). For example, research in ontologies also may improve data integration efforts. Recent informatics research for sleep disorders research has led to the development of ontologic-driven data-mapping procedures that have provided an efficient approach for integrating data across databases and institutions (R. Mueller and colleagues, unpublished results).

A related concept is that certain CER questions may be best addressed by merging data from existing administrative or clinical databases (including EMRs) with data from prospectively collected research databases. Research databases often better standardize the definitions of data elements than do administrative or clinical databases. However, the formats of such databases may vary greatly and the coding systems may not easily harmonize between databases. Continued work on the development of standardized coding systems and ontological frameworks may facilitate the harmonization of data across many sources.

Although there are a number of questions regarding the validity and reliability of administrative data, another major barrier is access to the data. To understand the comparative effectiveness of care, up-to-date near real-time data are essential and becoming more common. Integrating data from multiple sources, including administrative data sets and EMRs, for CER purposes would significantly improve the ability to compare treatment and diagnostic strategies across multiple systems of health care. Because issues of privacy are determined by local institutional review boards (IRBs), there is significant heterogeneity in the willingness to permit sharing of data. There is currently no national standard for sharing data between health care providers, health care organizations, academic and nonacademic institutions, and federal entities. In research that relies on the sharing and pooling of data for individual use or reuse between such organizations, the process can be exceedingly inefficient and time consuming, with approvals required from multiple individuals within each organization, including IRB officials and security and privacy compliance officers. Despite having previously shared data between organizations, the process often must be replicated for each additional investigation, adding to the inefficiency and delaying research progress. Given the need to be able to perform comparative effectiveness research in an efficient and timely fashion, a national emphasis is needed to reduce this barrier. Emphasis needs to be placed on developing national standards to facilitate the sharing of data by agencies such as Centers for Medicare and Medicaid Services (CMS), NIH, and Department of Veterans Affairs and among academic and nonacademic partners. Such data sharing could involve deidentified or limited data sets in which personal identifiers such as names, birth dates, and dates of medical services are removed or transformed to protect patient privacy. These standards need to emphasize the relative benefit of CER in relationship to risk of disclosure associated with the use of clinically derived administrative data. Innovative strategies include distributed data network approaches wherein identifiable data elements remain under the control of the respective institutions and their disclosure or data sharing policies (36, 37). With appropriate governance arrangements, permissions for disclosure, access, and use of these data can be controlled by the originating health care system.

Funding Mechanisms

Investigator-initiated funding for CER is proportionately and in absolute terms far less than it is for other biomedical research. The need for CER to inform the nation's health care enterprise has motivated the use of targeted contracts, task orders, and other government-specified work. AHRQ has developed centers and networks, including Evidence-Based Practice Centers, Centers for Education and Research on Therapeutics, and the Developing Evidence to Inform Decisions about Effectiveness Network (38). These types of structures are important. In addition, the workshop participants emphasized the need for investigator-initiated peer-reviewed CER research via NIH mechanisms. Without funding to support innovative, peer-reviewed CER studies, the scientific quality of CER could suffer.

CER will benefit from multidisciplinary teams. These are not common and often do not include expertise in implementation or improvement research. In addition to the contract-based work done at AHRQ CER centers, programs and centers should be funded that assemble multiple disciplines and conduct investigator-initiated studies. Networks of investigators that have been established based on clinical trials, clinical and translational science awards (CTSA), or linkages of electronic data sets will be valuable resources. Approaches should be developed to encourage collaboration from investigators currently outside these networks.

Stakeholder Contributions

The ultimate impact of CER will depend on its implementation of treatment and strategies in real-world settings by the health care industry. Partnerships with stakeholders are needed, both to conduct research and to translate CER findings into practice. Although AHRQ, NIH, and the new Patient-Centered Outcomes Research Institute presumably will be the major funders of CER, other stakeholders will be needed to provide other resources crucial for CER implementation, including access to health care systems, providers, and patients; electronic data; and medical records. For example, health plans and provider groups might be able to provide access to claims and electronic medical records, pharmaceutical companies might conduct and provide clinical trials using novel designs, and information technology vendors or consultants could provide and develop data collection and interface methods to support CER. In addition, these partners could also provide settings and opportunities for training CER researchers. Potential barriers to these collaborations include economic interests that could be affected by the results and implementation of CER, cultural and political differences among the research community and stakeholders, organizational hurdles including legal concerns about patient privacy, and the lack of channels for communication and collaboration.

The National CTSA Consortium sponsored a CER forum in December 2010 at the NIH so that representatives from federal agencies, industry, foundations, and other stakeholders could discuss such high-impact CER topics as infrastructure, education/training/workforce development, methods development, community and practice engagement, and health information technology. The meeting outcomes included identifying challenges, opportunities, and next steps, including a recommendation that priority should be given to funding studies that have explicit plans to work with other stakeholders in the health care community. More information can be found at

Engagement of Community Participants

The CER reports by the Institute of Medicine (2) and Federal Coordinating Council for Comparative Effectiveness Research (5) emphasized the need for community input when choosing relevant questions and for recruiting sites, patients, and physicians into studies. These efforts also should address disparities in health and health care among communities. Engagement of the community has not been the tradition in academic medical research. This is an important part of CER and is mandated for work done at the AHRQ CER centers. Not only will such engagement enhance the relevance of questions addressed by CER but also it will engage members of the community in the research and will educate the public so it can maximally benefit from the findings of CER. Community engagement is a substantially new approach, there is relatively little experience in this area, and it is time consuming. Although expertise is sparse, it is growing through the efforts of Centers for Disease Control (CDC), NIH-supported community-based participatory research grants, and the Community Engagement Components of the NIH CTSAs. Support for developing the procedures and practices for engaging the public in CER is needed to further leverage these and encourage research in a wide variety of settings.

Needs for Training

The competencies needed by a researcher in CER include some that are distinct. Currently there is a critical need to strengthen a national cadre of investigators with the skills to conduct CER (39). Research training is a central role of CTSAs and they are involved in CER-specific education and career development. Investigators who conduct CER benefit from training in specific fields, especially epidemiology and biostatistics. More specifically, a recent report of the CTSA Consortium CER Workforce Development workgroup identified special skills and education relevant to CER, including research ethics and logistics related to the complex issues involved in community-engaged research, pragmatic clinical trials, biomedical informatics, electronic health records research, large database research, practice-based network research, and decision analysis/cognitive sciences, health economics/cost-effectiveness, and health services research (39). CER training and career development should be supported by training grants. Investigators who conduct CER may have Master in Public Health degrees, PhDs, or similar training.

Research institutions as well as investigators may view CER as representing a large paradigm shift. Research-intensive academic centers may be most familiar with traditional efficacy studies and randomized trials, and may not understand or value the role CER has in the spectrum of clinical research. IRBs may be uncomfortable with the flexibility needed in pragmatic clinical trials or with alternative approaches to seeking informed consent from patients in real-world settings. Thus, programs to orient institutions and IRBs to CER could help accelerate support for CER. Furthermore, in multicenter studies, the need for IRB approval at multiple sites may sometimes create long delays. One solution is to arrange for IRBs to be able to cede to one another, an approach used in some projects led by members of the Health Maintenance Organization Research Network.

Summary and Recommendations

This NHLBI Expert Workshop highlighted areas that need strengthening to optimize the potential of CER in pulmonary diseases, sleep, and critical care. Key recommendations for agencies and investigators are:

  1. Promote interest in CER among the scientific community, NIH research programs, and existing disease research networks or interest groups.
  2. Increase efforts to engage stakeholders, including patients, providers, and payers, as participants in the development of CER questions and study designs.
  3. Invest in the further development of databases and other infrastructure to enable CER. This includes establishing efficient methods for data sharing while still protecting patient privacy.
  4. Make full use of a broad range of study designs, selecting the design that best addresses the need for timeliness and rigor. Designs include but are not limited to cluster randomized trials, pragmatic clinical trials, quasi-experimental studies, and other observational designs.
  5. Increase the appropriate use of observational studies, both to conduct CER and to identify areas where CER is needed.
  6. Encourage investigators to collect information on a range of outcomes that are important to varying stakeholders, including clinical outcomes, patient-reported outcomes, and economic outcomes.
  7. Ensure that committees that review CER grant applications consist of reviewers who have appropriate perspective and training. This includes methodologic expertise in the broad range of study designs described above.
  8. Further develop the workforce for CER by supporting training opportunities that focus on the appropriate methodologic and practical skills.

The workshop participants generated a lengthy list of potential CER questions for lung diseases and sleep disorders, but agreed that it was premature to prioritize among them. Rather, Table 3 presents illustrative examples of CER research topics, which are not intended to be all-inclusive. CER has a unique and potentially powerful role in the spectrum of research to enhance health. It is early in its evolution as a research field. Focused efforts to enhance the quality and use of CER in lung diseases are likely to yield rich downstream benefits by improving clinical practice and health care systems and ultimately improving patient health.



Additional participants included Dr. James Kiley (NHLBI) and Dr. Susan Shurin (NHLBI). The authors thank Kaitlyn Harrington for her skilled assistance with coordinating input from the many coauthors on the drafts of this paper. They also thank Tracey Woodard, Andrea Moreira, and Theresa Nativio for their excellent logistical support.

*Members of the Comparative Effectiveness Research in Lung Diseases Workshop Panel:

Andrea Apter, M.D., M.A., M.Sc., University of Pennsylvania, Philadelphia, PA; Carol Blaisdell, M.D., NHLBI, Bethesda, MD; Thomas F. Boat, M.D., University of Cincinnati College of Medicine, Cincinnati, OH; Noreen Clark, Ph.D., Center for Managing Chronic Disease, University of Michigan, Ann Arbor, MI; Dennis Drotar, Ph.D., Children's Hospital Medical Center, Cincinnati, OH; Daniel Lewin, Ph.D., D.ABSM, NHLBI, Bethesda, MD; Bruce C. Marshall, M.D., Cystic Fibrosis Foundation, Bethesda, MD; Antonello Punturieri, M.D., Ph.D., NHLBI, Bethesda, MD; Susan Redline, M.D., M.P.H., Brigham and Women's Hospital, Boston, MA; Shelby D. Reed, Ph.D., Duke University, Durham, NC; Michael Schatz, M.D., M.S., Kaiser Permanente Medical Center, San Diego, CA; Robert Smith, Ph.D., NHLBI, Bethesda, MD; Stephen B. Soumerai, Sc.D., Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA; B. Taylor Thompson, M.D., Massachusetts General Hospital, Boston, MA.


Supported by the Division of Lung Diseases, National Heart, Lung, and Blood Institute, National Institutes of Health.

D.A. was funded by HSR&D, VA Puget Sound Health Care System. The views expressed in this manuscript are those of the authors and do not necessarily represent the opinions of the Department of Veterans Affairs.

Originally Published in Press as DOI: 10.1164/rccm.201104-0634WS on June 23, 2011

Author Disclosure: None of the authors has a financial relationship with a commercial entity that has an interest in the subject of this manuscript.


1. Iglehart JK. Prioritizing comparative-effectiveness research–IOM recommendations. N Engl J Med 2009;361:325–328. [PubMed]
2. Institute of Medicine (US) Committee on Comparative Effectiveness Research Prioritization. Initial national priorities for comparative effectiveness research. Washington, D.C.: National Academies Press; 2009.
3. Clancy C, Collins FS. Patient-Centered Outcomes Research Institute: the intersection of science and health care. Sci Transl Med 2010;2:37cm18. [PubMed]
4. Lauer MS, Collins FS. Using science to improve the nation's health system: NIH's commitment to comparative effectiveness research. JAMA 2010;303:2182–2183. [PubMed]
5. Federal Coordinating Council for Comparative Effectiveness Research (US) Report to the President and the Congress. Washington, DC: US Dept. of Health and Human Services; 2009.
6. Selker HP, Strom BL, Ford DE, Meltzer DO, Pauker SG, Pincus HA, Rich EC, Tompkins C, Whitlock EP. White paper on CTSA consortium role in facilitating comparative effectiveness research: September 23, 2009 CTSA consortium strategic goal committee on comparative effectiveness research. Clin Transl Sci 2010;3:29–37. [PubMed]
7. Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME–PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice. BMC Health Serv Res 2003;3:22. [PMC free article] [PubMed]
8. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006;2:CD000259. [PubMed]
9. O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard-Jensen J, Kristoffersen DT, Forsetlund L, Bainbridge D, Freemantle N, Davis DA, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2007;4:CD000409. [PubMed]
10. Flodgren G, Deane K, Dickinson HO, Kirk S, Alberti H, Beyer FR, Brown JG, Penney TL, Summerbell CD, Eccles MP. Interventions to change the behaviour of health professionals and the organisation of care to promote weight reduction in overweight and obese people. Cochrane Database Syst Rev 2010;3:CD000984. [PubMed]
11. Tunis SR, Benner J, McClellan M. Comparative effectiveness research: policy context, methods development and research infrastructure. Stat Med 2010;29:1963–1976. [PubMed]
12. Ware JH, Hamel MB. Pragmatic trials–guides to better patient care? N Engl J Med 2011;364:1685–1687. [PubMed]
13. Murray DM. Design and analysis of group-randomized trials. New York: Oxford University Press; 1998.
14. Soumerai SB, McLaughlin TJ, Ross-Degnan D, Casteris CS, Bollini P. Effects of a limit on Medicaid drug-reimbursement benefits on the use of psychotropic agents and acute mental health services by patients with schizophrenia. N Engl J Med 1994;331:650–655. [PubMed]
15. Grijalva CG, Nuorti JP, Arbogast PG, Martin SW, Edwards KM, Griffin MR. Decline in pneumonia admissions after routine childhood immunisation with pneumococcal conjugate vaccine in the USA: a time-series analysis. Lancet 2007;369:1179–1186. [PubMed]
16. Connors AF, Jr, Speroff T, Dawson NV, Thomas C, Harrell FE, Jr, Wagner D, Desbiens N, Goldman L, Wu AW, Califf RM, et al. The effectiveness of right heart catheterization in the initial care of critically ill patients. SUPPORT Investigators. JAMA 1996;276:889–897. [PubMed]
17. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin; 2001.
18. Klungel OH, Martens EP, Psaty BM, Grobbee DE, Sullivan SD, Stricker BH, Leufkens HG, de Boer A. Methods to assess intended effects of drug treatment in observational studies are reviewed. J Clin Epidemiol 2004;57:1223–1231. [PubMed]
19. Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol 2006;98:253–259. [PMC free article] [PubMed]
20. Brookhart MA, Rassen JA, Schneeweiss S. Instrumental variable methods in comparative safety and effectiveness research. Pharmacoepidemiol Drug Saf 2010;19:537–554. [PMC free article] [PubMed]
21. Martens EP, Pestman WR, de Boer A, Belitser SV, Klungel OH. Instrumental variables: application and limitations. Epidemiology 2006;17:260–267. [PubMed]
22. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther 2002;27:299–309. [PubMed]
23. Psaty BM, Siscovick DS. Minimizing bias due to confounding by indication in comparative effectiveness research: the importance of restriction. JAMA 2010;304:897–898. [PubMed]
24. Saaty TL. Decision making for leaders: the analytical hierarchy process for decisions in a complex world. Belmont, CA: Lifetime Learning Publications; 1982.
25. Krishnan JA. Con: comparative effectiveness research: more than dollars and cents. Am J Respir Crit Care Med 2011;183:975–976. [PubMed]
26. CONCERT [Internet]. [Accessed November 14, 2010] Available from:
27. Olson AL, Swigris JJ, Lezotte DC, Norris JM, Wilson CG, Brown KK. Mortality from pulmonary fibrosis increased in the United States from 1992 to 2003. Am J Respir Crit Care Med 2007;176:277–284. [PubMed]
28. Soriano JB, Miravitlles M, Borderias L, Duran-Tauleria E, Garcia Rio F, Martinez J, Montemayor T, Munoz L, Pineiro L, Sanchez G, et al. (Geographical variations in the prevalence of COPD in Spain: relationship to smoking, death rates and other determining factors). Arch Bronconeumol 2010;46:522–530. [PubMed]
29. Morris RD, Munasinghe RL. Geographic variability in hospital admission rates for respiratory disease among the elderly in the United States. Chest 1994;106:1172–1181. [PubMed]
30. National Institutes of Health NCfRR. ResearchMatch [accessed March 8, 2011] Available from:
31. Krishnan JA, Schatz M, Apter AJ. A call for action: comparative effectiveness research in asthma. J Allergy Clin Immunol 2011;127:123–127. [PMC free article] [PubMed]
32. Quality AfHRa. Recovery Act Awards: Evidence Generation Awards. Fact Sheet; 2010 [accessed March 8, 2011]. Available from:
33. Quality AfHRa. Recovery Act Awards: Data Infrastructure Awards. Fact Sheet; 2010 [accessed March 8, 2011].
34. Holmes JH, Brown J, Hennessy S, Lane K, Langer RD, Lazarus R, O'Connor PJ, Strom BL, Platt R. Developing a distributed research network to conduct population-based studies and safety surveillance [abstract]. AMIA Annu Symp Proc 2008;6:973. [PubMed]
35. Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network–improving the evidence of medical-product safety. N Engl J Med 2009;361:645–647. [PubMed]
36. Brown JS, Holmes JH, Shah K, Hall K, Lazarus R, Platt R. Distributed health data networks: a practical and preferred approach to multi-institutional evaluations of comparative effectiveness, safety, and quality of care. Med Care 2010; 48:S45–S51. [PubMed]
37. Maro JC, Platt R, Holmes JH, Strom BL, Hennessy S, Lazarus R, Brown JS. Design of a national distributed health data network. Ann Intern Med 2009;151:341–344. [PubMed]
38. DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) Network [Internet] Quality AfHRa. 2011; [accessed March 12, 2011]. Available from:
39. Kroenke K, Kapoor W, Helfand M, Meltzer DO, McDonald MA, Selker H. Training and Career Development for Comparative Effectiveness Research Workforce Development: CTSA Consortium Strategic Goal Committee on Comparative Effectiveness Research Workgroup on Workforce Development. Clin Transl Sci 2010;3:258–262. [PMC free article] [PubMed]
40. Stutchfield P, Whitaker R, Russell I. Antenatal betamethasone and incidence of neonatal respiratory distress after elective caesarean section: pragmatic randomised trial. BMJ 2005;331:662. [PMC free article] [PubMed]
41. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002;325:941. [PMC free article] [PubMed]
42. Donahue JG, Weiss ST, Livingston JM, Goetsch MA, Greineder DK, Platt R. Inhaled steroids and the risk of hospitalization for asthma. JAMA 1997;277:887–891. [PubMed]

Articles from American Journal of Respiratory and Critical Care Medicine are provided here courtesy of American Thoracic Society