Search tips
Search criteria 


Logo of jcerLink to Publisher's site
J Comp Eff Res. 2015 September; 4(5): 465–471.
PMCID: PMC5549719

Stakeholder assessment of comparative effectiveness research needs for Medicaid populations


Patients, providers and policy-makers rely heavily on comparative effectiveness research (CER) when making complex, real-world medical decisions. In particular, Medicaid providers and policy-makers face unique challenges in decision-making because their program cares for traditionally underserved populations, especially children, pregnant women and people with mental illness. Because these patient populations have generally been underrepresented in research discussions, CER questions for these groups may be understudied. To address this problem, the Agency for Healthcare Research and Quality commissioned our team to work with Medicaid Medical Directors and other stakeholders to identify relevant CER questions. Through an iterative process of topic identification and refinement, we developed relevant, feasible and actionable questions based on issues affecting Medicaid programs nationwide. We describe challenges and limitations and provide recommendations for future stakeholder engagement.

Keywords: : CER, comparative effectiveness research, Medicaid, stakeholder, underserved populations

By providing information on the relative benefits of real-world therapeutic options, comparative effectiveness research (CER) can help patients, clinicians and policy-makers make more informed decisions about medical care. Medicaid provides health coverage for low-income Americans, with combined federal and state funding. While some elements of the program are guided by federal mandates, state Medicaid officials have control of a wide range of coverage decisions, but also often face budgetary and other challenges as they work to make evidence-based choices to care for the populations they serve. New CER that addresses the real-world choices most relevant to Medicaid populations represents an important priority.

Several prior initiatives have developed broad-based lists of potential CER topics. In 2005, the Agency for Healthcare Research and Quality (AHRQ) published a list of priority conditions for CER, incorporating input from a broad range of stakeholders and ultimately prioritized by a steering committee from the Department of Health and Human Services [1].

Over the subsequent years these priorities became the focus for the AHRQ-supported Effective Health Care (EHC) program, as well as AHRQ's DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) and DEcIDE-2 Networks. In 2009, a committee convened by the Institute of Medicine (IOM) created a list of 100 priority topics for CER focused on improving the quality of healthcare at the request of Congress to help guide research efforts support by the American Recovery and Reinvestment Act of 2009. The IOM described this work as reflecting: “the insights of health professionals, consumer advocates, policy analysts and others. The committee received 1268 unique topic suggestions, which it narrowed to 100. The list reflects a range of clinical categories, populations to be studied, categories of interventions and research methodologies” [2].

Although this work applies to healthcare in the USA broadly, there is still a pressing need for research targeting the unique needs of the Medicaid population. Medicaid provides coverage for groups that may be underrepresented in other patient populations, including large numbers of children, pregnant women and people with mental illness. CER that addresses the specific choices confronting Medicaid programs can inform the decisions of Medicaid officials. Recognizing this need, AHRQ commissioned our team to seek input from Medicaid Medical Directors (MMDs) and to engage with a wide range of additional stakeholders to identify relevant, feasible and actionable CER topics that would address the needs of the Medicaid population.

Stakeholder engagement

The work described here was supported by an AHRQ task order, which specified the stages in which the process was to be conducted. In order to identify topics most relevant to the Medicaid population, the process began with areas of interest and concern identified by state MMDs and then incorporated patient, provider, policy-maker and other stakeholder input [3]. The process included six stages (Table 1), beginning with the MMDs identifying general topic areas for potential research. Through group and individual discussions these initial insights were developed into a large group of possible research avenues. The potential topics were further refined during interviews and discussions with governmental, clinical and patient advocacy stakeholders who provided perspectives on the clinical areas identified by the MMDs. A web-based meeting was used to review the research ideas developed and determine the final list of CER topics.

Table 1.
Process for identification of priority topics for comparative effectiveness research.

Topic identification with MMDs

The engagement process began in November 2012 at the fall meeting of the MMD Learning Network (MMDLN) in Washington DC with a 90-min workshop-style session moderated by the principal investigator from Brigham and Women's Hospital (BWH). MMDs from approximately 25 states participated in the workshop in person, with several more joining by teleconference. The workshop introduced the topic identification and stakeholder engagement process, reviewed the history of the DEcIDE program and consisted mainly of a guided discussion to elicit broad topic areas for CER that the directors felt would be most helpful to their programs. The MMDs participated enthusiastically during the session and generated an extensive list of potential CER topics relevant to the needs of the Medicaid population.

Following the MMDLN workshop, the BWH team reviewed the list of topics and categorized them into six major clusters of needs for CER from the perspective of Medicaid programs. Table 2 shows the six topic clusters, with details on some of the specific elements within each cluster mentioned most often by MMDs.

Table 2.
Six major clusters of need for comparative effectiveness research.

Individual interviews with MMDs

To explore these six-topic clusters more deeply, the BWH team conducted individual telephone interviews with MMDs from November 2012 to February 2013. Then the team divided the topics list into two general groupings: clinical topics and systems topics (Box 1).

Box 1. 

Priority topic areas identified by Medicaid Medical Directors.

Clinical topics

  • Prenatal testing
  • Autism
  • Substance abuse
  • Specialty drugs
  • Use of diagnostic imaging
  • Pediatric medicine
  • Long term care

Systems topics

  • Re-admission rates
  • Birth outcomes
  • Patient adherence
  • Behavioral health
  • Medical homes
  • Managed care

Specialty drugs included treatments for rheumatoid arthritis, multiple sclerosis and hepatitis C, as well as intravenous immunoglobulin and long-acting antipsychotic medications.

Stakeholder group engagement

During the following 5 months, the BWH team solicited input on these priority Medicaid topics from stakeholder groups in order to explore in-depth the specific potential research questions contained in each of the clusters. Before project initiation, the BWH team proposed a list of 33 stakeholder groups that represented patients (eight), providers (17) and policy-makers (eight). Detailed interviews were conducted with 23 of these representatives from stakeholder groups (five patients, 13 providers, five policy-makers) to discuss the topic areas and begin to identify more specific and actionable research questions (see Supplementary Table 1). Seven additional interviews were conducted with MMD and provider stakeholders specifically suggested by AHRQ.

Stakeholder input clarified the topics with greatest gaps in knowledge and greatest suitability for CER. These discussions were intended to narrow the list to the areas of greatest opportunity for Medicaid programs, where reducing gaps in knowledge about comparative effectiveness would have the greatest positive impact. The BWH team used these contributions to create a tentative list of 13 research areas.

Second meeting with MMDs to prioritize topics

Using these 13 research areas as a base, the team conducted a web-based conference in July 2013 with approximately 20 members of the MMDLN to discuss and prioritize. Participants were polled on the various topics within each grouping and asked to select the most important. The participating MMDs ranked the most highly (with polling %) for the clinical topics: substance abuse (69%), autism (50%), specialty drugs (44%), imaging studies (38%); and for the systems topics: re-admission rates (75%), medical homes (69%) and behavioral health (56%). A total of 25 possible research questions were generated during these sessions. Through subsequent internal review as well as consultation with AHRQ, ten questions were selected for more in-depth development. Decisions about which topics to select were driven by the feasibility of performing CER for that question (e.g., whether relevant data sources existed) and priorities within AHRQ. These topics areas became the focus of the stakeholder web conference during which CER topics were selected.

Stakeholder input

In September 2013 we conducted a day-long web conference based on these ten key questions to elicit further input and feedback from stakeholders. Stakeholder participants were asked to consider the relevance, feasibility and actionability of each research question and topic. For a topic to be considered relevant it had to address MMD and stakeholder priorities, fill gaps in current research and fit within the goals of CER/patient-centered outcomes research (PCOR). For a topic to be feasible, the data and methods needs and timeline must be reasonable. Last, a topic should be actionable; it must provide data that MMDs and other stakeholders can use in decision-making. A total of 23 people participated in the web-based workshop and discussed the specific CER questions.

Priority finalization with AHRQ

After the stakeholder meeting, the BWH team reviewed and organized the MMD and stakeholder feedback and worked with AHRQ to develop the final list of ten potential CER topics and questions (Table 3). The BWH team then reviewed the literature to identify existing research and ongoing studies relevant to these clinical areas. This information was applied in a PICOTS (population, intervention, comparator, outcome, timing, setting) framework so that each CER topic identified could be outlined clearly to guide future research (see Supplementary Data) [4,5].

Table 3.
Final ten comparative effectiveness research topics for Medicaid populations.

Limitations & challenges

Stakeholder specialization vs topic breadth

This process successfully incorporated input from a broad range of stakeholders to produce a list of relevant CER questions for Medicaid populations. However, the team encountered challenges during the process with resulting limitations that must be kept in mind when assessing the list of topics. While input from MMDs on broad healthcare topics was generally straightforward and consistent, incorporating clinically specialized stakeholders into broad discussions was more challenging. While MMDs are concerned with a diverse range of areas – including prenatal care and birth outcomes, hepatitis C treatments, antipsychotic medication use and many more – clinically specialized stakeholder groups have a much narrower focus of interest. This type of stakeholder group is likely to have strong experience and knowledge regarding one specific area, but less investment in or knowledge of others. Therefore, bringing stakeholders with specific interests together for broad group discussion did not reliably lead to consensus on research priorities.

Challenges for objective ranking

Many advocacy groups focus on one healthcare issue and advocate for the issue passionately, and accordingly may not value research that does not directly address that issue of interest. Given this focus on a single segment of care or therapeutic issue, clinically specific stakeholders provided valuable insights in specific topics, but without an unbiased scoring system, were sometimes unable to compare the importance of different issues objectively. As other stakeholder engagement research groups have found, the lack of an objective rating or polling system can create difficulties in reaching consensus in group discussions [6,7]. It is important to note that this project was not designed as a pure epidemiologic study. Our process was guided by a predetermined set of steps specified in the initial task order from AHRQ, which also had input at multiple stages to narrow down the list of topics based on their priorities.

Uneven knowledge of stakeholders in any given area

Even when bringing specialized groups together around a given healthcare topic, we found challenges in bridging knowledge gaps between stakeholders. Patients, providers and researchers may all focus on one disease area, but their experiences, priorities and knowledge will vary. For instance, providers may have a strong grasp of the clinical issues in a disease area, but not understand the research challenges involved. This grouping of varied perspectives is one of the main strengths of stakeholder engagement, but in this project the range of stakeholder experience was so broad that it complicated the process of reaching consensus.

Limitations of web-based meetings

The original project plan envisioned a one-day in-person stakeholder meeting conducted at the AHRQ conference center. Due to travel restrictions which affected many agencies and organizations, the decision was made to convert the 1-day meeting to a web conference format. We found the format itself to be effective: many participants had experience with web-based meetings, and there did not seem to be decreased interaction or discussion among attendees due to the format. However, retaining the single-day format may not be the optimal structure for presentation and discussion-generating sessions. Attendance varied for individual sessions, reflecting participants joining from different time zones, and attendance decreased later in the day, probably due to participant fatigue from many hours in web conference session. Once the in-person format was discarded, it is possible that consistent participant energy through all sessions would have been easier to maintain if the meeting was divided into shorter sessions over a series of different days.

Strengths of approach

Despite some challenges, we found this iterative model of input, consolidation and discussion with our stakeholder groups effective. First, the project benefited from focused and consistent input from the MMD Learning Network: the MMDs worked well together as a team during the two MMD meetings and many generously contributed time for additional individual interviews. Eager participation and input from a broad range of stakeholder groups allowed the team to gather a broad representation of experiences from researchers, providers, payers, patients and caregivers. During the stakeholder interviews and web conferences, 38 stakeholders worked together to refine research questions. Despite differences in clinical expertise, research knowledge and disease experience, the stakeholders collaborated to generate ideas, refine topics and develop research questions.

Although some challenges of web conferences were noted above, the format was generally effective and efficient. Using virtual meetings reduced travel time and cost, allowing the BWH team to consult with MMDs and stakeholders more often and gather more input. It also offered participants multiple ways to participate actively through traditional voice discussion, typed question and comment submission and online polling. This allowed all participants to voice their ideas and concerns, even if they were hesitant to voice them to the whole group. In particular, the MMDs found online polling useful for collaboration as the team worked to prioritize topics.

Recommendations for future stakeholder engagement in CER

Timing of engagement

Throughout this process, we found that thoughtful engagement timing was crucial. To address our concerns with the efficacy of including clinically specialized stakeholder groups in broad discussions, we recommend that future groups involve MMDs and generalist stakeholder groups more heavily in the initial and final phases of topic generation. Because MMDs and generalists deal with a broad range of diseases and healthcare system issues, they are well equipped to identify cross-cutting issues and are especially helpful during general brainstorming discussions. Clinically specialized groups bring expertise in their particular topic areas and may be most effectively included in interviews or small group discussion to refine specific research questions, rather than in the more general discussions.

Objective ranking of topics

To combat the challenges in balancing priorities of advocacy groups, we recommend that future research teams develop methods for stakeholders to rate topics objectively. This would allow groups to develop consensus more effectively and move discussions forward more efficiently. It would also bolster the credibility of the topic selection process and ensure that all stakeholder perspectives are incorporated in decision-making.

Web conference tools

We highly recommend online commenting, survey and polling tools, which are often built into web conference software. We found that these tools encourage participation, especially from those stakeholders who may be uncomfortable sharing their opinions in large groups. Additionally, we recommend that future group organize meetings using a multiday format and spreading sessions over several days. Fatigue for moderators, organizers and participants in multiple sessions could be lessened by avoiding day-long meetings.

Linking to funding

Lastly, future stakeholder engagement in research priority development could benefit from a linkage of CER priority lists to research funding announcements. This would have a threefold positive effect. First, it would underline the long-term importance of the project and encourage active engagement from all stakeholder groups. Second, the inclusion of the priority list by funding organizations would ensure that this list is disseminated broadly within the research community. Third, funding opportunities linked to the priority list would guarantee that researchers have the financial tools to pursue the stakeholder-identified topics.


By combining state MMD and stakeholder group input through a six-step engagement process, we created a list of CER priorities based on the concerns most relevant to Medicaid programs nationwide. Despite differences between the state programs, there was general agreement on priorities for Medicaid populations. The MMD's priorities are strongly influenced by the new and challenging decisions and choices which Medicaid programs need to make. Areas with high cost, increasing incidence and/or proliferating treatments tended to have the greatest need for improved information on comparative effectiveness. Potentially owing to the growing body of information compiled over the last several years regarding pharmacologic CER, nonpharmacological interventions and health services issues were identified as areas of particular importance for new CER across the states.

Future perspective

In addition to the identification of ten areas for which new CER could meet the needs of MMDs and Medicaid populations, this project highlighted the evolution from earlier lists of CER topics to our list, suggesting trends in the priorities for Medicaid that are likely to affect future research needs. Specifically, Medicaid officials are increasingly interested in research that evaluates delivery system approaches to care, such as community-based screening or treatment programs or managed care interventions that may cut across a range of conditions. The MMDs are especially interested in understanding whether these systems-based interventions actually have a meaningful impact on the quality of medical care. While the classical CER framework comparing two specific treatments or testing options will remain important for patients, clinicians and other decision-makers, future CER will need to assess broader-based interventions that impact many elements of the delivery system simultaneously.

Leveraging the perspectives and expertise of these important stakeholder groups allowed the team to develop a list of relevant, feasible and actionable research questions. Besides yielding this list, the six-step process of topic identification and refinement with stakeholders generated insights into best practices for stakeholder engagement and recommendations for future work. We believe that these tools could be used to identify and refine topics in fields beyond CER research.

Executive summary


  • Increased stakeholder engagement in research prioritization is critical to guide relevant, feasible and actionable comparative effectiveness research. However, there remain many unanswered questions about optimal models of stakeholder engagement.
  • Medicaid providers and policy-makers face unique challenges in clinical decision-making because the program cares for traditionally underserved populations.

Stakeholder engagement

  • In order to develop a list of priority comparative effectiveness research topics for Medicaid populations, our team gathered input from Medicaid Medical Directors and stakeholder groups through one initial in-person conference, 38 individual stakeholder interviews and three web conferences.

Limitations & challenges

  • For future research teams looking to develop research priority lists, we recommend thoughtful timing of engagement to include stakeholder expertise most effectively, use of web conference tools to encourage participation and consensus, objective rating systems to balance diverse perspectives and promote consensus, and linkage of a final list to funding announcements to promote dissemination and future research.

Supplementary Material


Financial & competing interests disclosure

S Schneeweiss is principal investigator of the Harvard–Brigham Drug Safety and Risk Management Research Center funded by the US FDA. His work is partially funded by grants/contracts from the Patient-Centered Outcomes Research Institute, the FDA, and the National Heart, Lung, and Blood Institute. He is also a consultant to WHISCON, LLC and Aetion, Inc., a software manufacturer of which he also owns shares, and is principal investigator of investigator-initiated grants to the Brigham and Women's Hospital from Novartis and Boehringer-Ingelheim, unrelated to the topic of this review. MA Fischer is a clinical consultant for the Alosa Foundation, an educational non-profit and has research support through his institution from CVS-Caremark and Otsuka America. HHSA290200500161 TO3 was the source of funding. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.


Papers of special note have been highlighted as: • of interest

1. Chalkidou K, Whicher D, Kary W, Tunis S. Comparative effectiveness research priorities: identifying critical gaps in evidence for clinical and health policy decision making. Int. J. Technol. Assess. Health Care. 2009;25(3):241–248. [PubMed]
2. 2009. IOM Report Recommends 100 Initial Priorities for Research to Determine Which Health Care Approaches Work Best.
3. Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 2012;1(2):181–194. [PMC free article] [PubMed]
4. Riva JJ, Malik KM, Burnie SJ, Endicott AR, Busse JW. What is your research question? An introduction to the PICOT format for clinicians. J. Can. Chiropr. Assoc. 2012;56(3):167–171. [PMC free article] [PubMed]
5. Gliklich RE, Leavy MB, Velentgas P, et al. Incorporating stakeholder perspectives in developing a translation table framework for comparative effectiveness research. J. Comp. Eff. Res. 2012;1(3):281–292. [PubMed]
6. Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 2012;27(8):985–991. [PMC free article] [PubMed]• Provides thorough recommendations for stakeholder engagement that can be adapted by those seeking to implement similar processes.
7. Greenberg CC, Wind JK, Chang GJ, Chen RC, Schrag D. Stakeholder engagement for comparative effectiveness research in cancer care: experience of the DEcIDE Cancer Consortium. J. Comp. Eff. Res. 2013;2(2):117–125. [PubMed]• Methods for stakeholder engagement in research are still in development, so this thoughtful and careful use of stakeholder input to develop a list of cancer research priorities is a useful guide.

Articles from Journal of Comparative Effectiveness Research are provided here courtesy of Future Science Group