Search tips
Search criteria 


Logo of jcerLink to Publisher's site
J Comp Eff Res. 2015 August; 4(4): 351–357.
PMCID: PMC4538706

IMPACCT Kids’ Care: a real-world example of stakeholder involvement in comparative effectiveness research


The Patient-Centered Outcomes Research Institute has accelerated conversations about the importance of actively engaging stakeholders in all aspects of comparative effectiveness research (CER). Other scientific disciplines have a history of stakeholder engagement, yet few empirical examples exist of how these stakeholders can inform and enrich CER. Here we present a case study which includes the methods used to engage stakeholders, what we learned from them, and how we incorporated their ideas in a CER project. We selected stakeholders from key groups, built relationships with them and collected their feedback through interviews, observation and ongoing meetings during the four research process phases: proposal development, adapting study methods, understanding the context and information technology tool design and refinement.

Keywords: : community networks, comparative effectiveness research, consumer participation, patient-centered outcomes research, patient engagement


Researchers and funding agencies (e.g., the Patient-Centered Outcomes Research Institute [PCORI]) prioritize stakeholder engagement in all aspects of CER. Deverka and colleagues define stakeholders as ‘individuals, organizations or communities that have a direct interest in the process and outcomes of a project, research or policy endeavor’ [1,2]. Community-based participatory research [3–5], action research [6–8], practice-based research [9–11] and user-centered design [12–14] have rich histories of using a variety of engagement methods and techniques. Building on experiences from these disciplines, CER experts are developing and testing methods for engaging stakeholders to enrich the research process, improve the relevance of research projects and accelerate the implementation of findings [15–19].

In recent years, shared terminology and common definitions have been developed to identify categories of stakeholders and delineate roles [2,20–21]. For example, the ‘7Ps’ framework was developed to identify key groups of people who might inform research: patients and the public, providers, purchasers, payers, policy-makers, product makers and principal investigators [20]. Other researchers have developed similar stakeholder categories [1,22]. Different levels of stakeholder engagement have been defined, ranging from one-time feedback to long-term engagement as study co-investigators [19], and a variety of methods have been used to elicit different perspectives [23]. Empirical examples from CER studies intentional about stakeholder engagement are needed to complement philosophical concepts and models.

Here, we describe stakeholder engagement in the PCORI-funded Innovative Methods for Parents And Clinics to Create Tools for Kids’ Care (IMPACCT Kids’ Care) project, which uses stakeholder-centered design processes to build, implement and test the comparative effectiveness of health information technology (IT) tools to track and document patients’ insurance status in community health center (CHC) clinics. The study is described in detail elsewhere [24], but this case study provides a ‘real-world’ example of the methods used to identify, engage and utilize feedback from diverse stakeholders for this CER project.

Study setting

OCHIN was founded in 2001 by Oregon Community Health Centers (CHCs), originally as the Oregon Community Health Information Network but renamed when other states joined. OCHIN is a nonprofit 501(c)(3) collaborative now serving CHCs in 20 states to develop and implement health IT. It also supports a practice-based research network (PBRN) and is an emerging community laboratory for CER [25–28]. OCHIN hosts a linked instance of the Epic© electronic health record (EHR) providing each CHC patient in the network with a single medical record number. The IMPACCT Kids’ Care project includes eight OCHIN member CHCs from three clinic systems.

Identifying stakeholders

We identified a diverse group of relevant stakeholders based on the 7Ps model described above: patients and the public, providers, payers, policy-makers and principal investigators. Patients and the public included patient partners, families, caregivers and community organizations. Providers included CHC leaders, managers and clinic staff (i.e., ‘end-users’ of the IT tools). Oregon Medicaid payers and policy-makers were also key stakeholders for this study. Since we conducted the CER study, our team of researchers served as the principal investigators.

Existing stakeholder groups

Two stakeholder engagement groups for research exist at OCHIN, the PBRN and the Patient Engagement Panel (PEP). The PBRN, comprised of CHC clinicians and primary care researchers, met regularly to inform research proposals, develop new practice-based research methods and help interpret results [27,28]. The PEP is comprised of patients who inform all OCHIN research projects and advise on organizational priorities [29]. We leveraged these groups, leaders of community organizations and state policy-makers who have contributed to previous OCHIN studies or served on the PBRN [16,30–32] to provide feedback on the research phases listed below.

New stakeholder groups

In addition to existing relationships, we identified several new groups needing representation in this project: CHC staff, patients’ family members and caregivers. Working with managers at the CHCs we identified appropriate representatives from key staff involved in clinic health insurance processes. These ‘end-users’ were recruited to participate in user-centered design sessions and to serve on the project's Clinic Advisory Group to advise the project team on all aspects of study methods and provide interpretation and dissemination of research results. During study interviews with patients’ family members and caregivers, we invited interviewees to participate in the project's Family Advisory Panel; three (of 20) interviewees agreed to join. The Family Advisory Panel joined with the PEP to provide ongoing advice and guidance including monitoring the study's progress and disseminating results. The challenges and lessons learned in developing the PEP (recruiting and retaining participants from underserved populations) are described elsewhere [29]. Figure 1 shows the existing and new stakeholder groups for this project.

Figure 1.
IMPACCT Kids’ Care project: existing and new stakeholder groups.

Collecting & using stakeholder feedback

We engaged stakeholders during four early phases of the research process: proposal development; adapting study methods; understanding the context; and IT tool design and refinement. We collected feedback from stakeholders using a variety of qualitative methods including semistructured interviews, advisory group meetings, observation, ‘Think Aloud’ usability testing and group meetings. We used a qualitative data analysis strategy to analyze feedback received from stakeholders so we could incorporate it into our research project [33]. Stakeholders were modestly compensated for initial interviews and for their roles on advisory groups. However, they were not compensated for participation in IT tool testing activities. Our Institutional Review Board reviewed and approved this study. The methods we used to engage stakeholders, what we learned, and how we incorporated their ideas into each of the research process phases is described in detail below.

Proposal development

The idea for developing IT tools within the EHR, to help clinics identify patients whose insurance was expired or nearing expiration, originated with a CHC staff member during an OCHIN PBRN meeting. This important idea was turned into a CER study and grant funding was sought. As the research team prepared the grant application, we consulted with the PEP, the PBRN, policy-makers and leaders of community organizations who are collaborators on other OCHIN initiatives. Consultation occurred through committee meetings, one-on-one meetings with individual CHC patients and PBRN members, and by circulating key pieces of the proposal to these stakeholders for comment. We gained valuable insights from a diverse group which allowed us to shape the scope of the proposal. This process led to the in-depth involvement of two co-investigators: a patient (KD) and a policy-maker (CG).

In addition to contributing ideas, helping with the proposal writing process, and serving as co-investigators, study advisors and consultants, stakeholders guided development of compensation policies for patients. Although we assumed patients would like cash or gift cards as compensation, we learned that some CHC patients preferred receiving travel support (to attend conferences) or computers (to aid participation in teleconferences) that would help them become more actively engaged with the research team. We also learned that even modest financial payment might adversely impact annual tax returns and/or jeopardize eligibility for public programs (e.g., Medicaid, Social Security) for some patients.

Adapting study methods

After the IMPACCT Kids’ Care project was funded, we engaged CHC patients, clinicians, and staff in preliminary meetings to launch the study, review the timeline and finalize study methods. Stakeholders provided invaluable feedback on the study methods. For example, when reviewing qualitative data collection methods, stakeholders advised our team to conduct interviews rather than focus groups with families, since they thought one-on-one interviews would yield more information and be less intimidating for participants. CHC leaders were also concerned about the staff burden of coordinating and scheduling focus groups and about reports of past experiences with low turn-out for these sessions. After consultation with qualitative research co-investigators, in lieu of the originally planned focus group, we modified our data collection design and conducted individual, semistructured interviews to make the study more ‘stakeholder-friendly’ and relevant. This modification enabled us to collect rich data using rigorous methods while remaining flexible to meet the needs of research participants.

The original proposal called for comparing IMPACCT Kids’ Care IT tools in two intervention and two control clinics. However, we learned from discussions with CHC leaders that all clinics must utilize standardized, consistent workflows throughout the entire clinic system to manage insurance enrollment. Thus, instead of two clinics, we modified our implementation plan to accommodate all four clinics in the intervention clinic system. We also added two matched control sites for a total of four. This change strengthened our study by preventing contamination bias in the intervention sites [34].

Understanding the context

To understand the context of obtaining and maintaining public health insurance for children, we interviewed 20 family members of pediatric patients (11 interviews were conducted in Spanish; nine interviews were conducted in English) and 31 clinic staff members (management, front desk and scheduling, insurance specialists, medical assistants, social workers, community health workers and providers). We also held a focused advisory group meeting with policy-makers from the State of Oregon, patient representatives, community leaders, and members of the OCHIN research team to exchange ideas and information about the project, state health insurance exchange, and the Affordable Care Act [35]. Based on what we learned from these interviews and meetings, we developed the IT tools and accompanying guides with a focus on the end-users (i.e., clinic staff).

IT tool design

Informed and guided by various stakeholder groups, OCHIN developed a set of IT tool prototypes. These prototypes were then shared with key stakeholders from the three participating clinic systems at a half-day retreat. During this retreat, CHC staff and families participated alongside members from OCHIN's research team, including four community-based co-investigators, one of whom (TB) was OCHIN's Chief Medical Informatics Officer at the time of the research and directed the development of the IT tools. The goal of the retreat was for the research team to summarize findings from family and clinic staff interviews and initial stakeholder engagement activities, present possible tool options, and obtain critical feedback about how to improve the prototypes.

We held dedicated IT tool review meetings with the Family Advisory Panel and Clinic Advisory Group to get their feedback on the tool prototypes. The research team also engaged stakeholders in Think Aloud exercises to further refine the tools. This user-centered design process involved key users ‘thinking aloud’ as they performed a set of specified tasks or reviewed a tool prototype [36]. Research team members visited two clinic sites and employed the Think Aloud process during interactions with six end-users of the IT tools [24].

Once prototypes were finalized, IT tools were built in the OCHIN Epic© system and activated for additional testing. In April 2014, to elicit feedback from end-users to fine-tune IT tools prior to full implementation, beta-testing was conducted across four sites with 16 stakeholders who represented a variety of roles in the clinic: insurance eligibility specialists, schedulers and front desk staff. After final modifications, the tools were implemented and available for all staff to use at the four intervention sites in early June 2014. In mid-July 2014, about six weeks after the tools were activated and guided by the Morville Framework [37], we conducted usability trials with 24 end-users (clinic staff) at the four implementation sites. Based on these trials, we redesigned some of the IT tools for ease of use and modified language on several functions to enhance clarity (e.g., we expanded drop-down list options and added data fields).

We discovered during iterative testing cycles that the accuracy of some of the insurance coverage data provided by the State of Oregon was unclear. We met with our policy-maker co-investigator and others from the State Medicaid office to resolve this and, as a result, the quality of the data have been improved and the IT tools were updated to be more reliable.

Key engagement methods, themes arising from our engagement activities and descriptions of stakeholder contributions are summarized in Table 1.

Table 1.
Descriptions of stakeholders, stakeholder engagement methods used, what we learned, and how we used stakeholder ideas throughout the project.


Stakeholders bring different experiences, interests and expertise to research studies which shape both the roles they play and the contributions they make to the process [1]. Engaging stakeholders in CER requires skills in aligning these interests and areas of expertise to the research project. Our study demonstrates effective engagement of different stakeholder groups ensuring that study design and interventions were truly clinic-, patient- and family-centered. Beyond the clinical team, this case study also highlights the importance of engaging healthcare staff.

The method of inviting interviewees to participate in our Family Advisory Panel was particularly effective in engaging stakeholders from ‘hard-to-reach’ groups. Hard-to-reach, as defined by PCORI, is used as a general term for individuals or communities who are historically underrepresented in health care research and/or less likely to be involved in research because of differences or barriers that impede communication or collaboration with researchers [38]. These barriers include language, education, social class, ethnicity, race, culture, geography, physical or cognitive impairments and other differences.

We leveraged previously cultivated relationships (e.g., PBRN clinicians, patient co-investigators and policy-makers) and forged new relationships (e.g., clinic end-users) in the IMPACCT Kids’ Care project. Therefore, the IT tools we developed are highly relevant to the stakeholder groups involved but they may need to be modified for other clinics. Future efforts will use dissemination and implementation science methodology to study the best ways to utilize these tools in additional OCHIN sites and, through additional collaborations, possibly beyond OCHIN. We will also continue to involve stakeholders in the analysis, interpretation and dissemination of project results.

The IMPACCT Kids’ Care project provides an example of how we involved stakeholders in CER proposal development, study method adaptation, context understanding, and IT tool design and refinement. Many of the relationships developed in this study are sustainable and have already influenced future studies.

Future perspective

As the field of stakeholder-engaged CER continues to mature, more real-world examples are needed to discuss methods, challenges and opportunities for meaningfully involving stakeholders in all aspects of research. CER will also continue to learn from other disciplines to sustainably and meaningfully engage stakeholders.

Executive summary


  • Previous work by community-based participatory research, action research, practice-based research, and user-centered design has shown that stakeholder involvement can significantly shape the research process.
  • The Patient-Centered Outcomes Research Institute-funded IMPACCT Kids’ Care project uses stakeholder-centered design processes to build, implement, and test the comparative effectiveness of health information technology (IT) tools to track and document patients’ insurance status in community health center clinics.


  • Stakeholders were selected from key groups (community clinics, patients’ families, and policy-makers). We leveraged previous relationships and built new relationships to collect ongoing feedback through interviews, observation, and meetings during four phases of the research process: proposal development, study design, understanding the context, and tool design and refinement.
  • Feedback and insights from stakeholders have bridged the academic and clinical worlds of our research by helping us understand clinic, family, and state policy workflows around insurance. Stakeholders have contributed to our knowledge about how IT tools can be used by clinics and patients’ families.
  • The stakeholders for IMPACCT Kids Care continue to be engaged in the analysis, interpretation, and dissemination of the results.


  • The IMPACCT Kids’ Care project provides an example of how we involved stakeholders in comparative effectiveness research proposal development, study method adaptation, context understanding and IT tool design and refinement.


The authors gratefully acknowledge the community clinicians, patients, policy-makers and other stakeholders that contributed to this project. The authors are grateful for editing assistance from Ms LeNeva Spires, Publications Manager, Department of Family Medicine, Oregon Health & Science University, Portland, Oregon.


Financial & competing interests disclosure

Funding for this study from the Patient Centered Outcomes Research Institute, Improving Health Systems, Cycle I (2012): “Innovative Methods for Patients and Clinics to Create Tools for Kids’ Care: IMPACCT Kids’ Care” (JE DeVoe, PI); and National Cancer Institute R01CA181452 (JE DeVoe, PI). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.


Papers of special note have been highlighted as: • of interest; •• of considerable interest

1. Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 2012;1(2):181–194. [PMC free article] [PubMed]•• Presents a framework, definitions and examples of concepts in stakeholder participation in comparative effectiveness research (CER), as well as definitions of stakeholder categories.
2. Lavallee DC, Williams CJ, Tambor ES, Deverka PA. Stakeholder engagement in comparative effectiveness research: how will we measure success? J. Comp. Eff. Res. 2012;1(5):397–407. [PubMed]•• Presents six criteria for evaluating the effectiveness of stakeholder engagement in CER.
3. Mikesell L, Bromley E, Khodyakov D. Ethical community-engaged research: a literature review. Am. J. Public Health. 2013;103(12):e7–e14. [PMC free article] [PubMed]
4. Tapp H, White L, Steuerwald M, Dulin M. Use of community-based participatory research in primary care to improve healthcare outcomes and disparities in care. J. Comp. Eff. Res. 2013;2(4):405–419. [PMC free article] [PubMed]
5. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu. Rev. Public Health. 1998;19:173–202. [PubMed]
6. Greenwood J. Action research: a few details, a caution and something new. J. Adv. Nurs. 1994;20(1):13–18. [PubMed]
7. Malterud K. Action research – a strategy for evaluation of medical interventions. Fam. Pract. 1995;12(4):476–481. [PubMed]
8. Hart E, Bond M. Making sense of action research through the use of a typology. J. Adv. Nurs. 1996;23(1):152–159. [PubMed]
9. Westfall JM, Mold J, Fagnan L. Practice-based research – “Blue Highways” on the NIH roadmap. JAMA. 2007;297(4):403–406. [PubMed]
10. Hartung DM, Guise JM, Fagnan LJ, Davis MM, Stange KC. Role of practice-based research networks in comparative effectiveness research. J. Comp. Eff. Res. 2012;1(1):45–55. [PMC free article] [PubMed]• Presents examples of current and potential roles of practice-based research networks in conducting CER, including pragmatic clinical trials, patient-centered decision-making, setting research priorities and interfacing with CTSA Centers.
11. Mold JW, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann. Fam. Med. 2005;3(Suppl. 1):S12–S20. [PubMed]
12. Shah SG, Robinson I. User involvement in healthcare technology development and assessment: structured literature review. Int. J. Health Care Qual. Assur. Inc. Leadersh. Health Serv. 2006;19(6–7):500–515. [PubMed]
13. Coulter A. Evidence based patient information is important, so there needs to be a national strategy to ensure it. BMJ. 1998;317(7153):225–226. [PMC free article] [PubMed]
14. Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J. Am. Med. Inform. Assoc. 2002;9(4):320–330. [PMC free article] [PubMed]
15. Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Aff. (Millwood) 2010;29(10):1834–1841. [PubMed]• Presents five key principles for the effective engagement of a broad coalition of participants in CER intended to improve healthcare and control costs.
16. Angier H, Wiggins N, Gregg J, Gold R, Devoe J. Increasing the relevance of research to underserved communities: lessons learned from a retreat to engage community health workers with researchers. J. Health Care Poor Underserved. 2013;24(2):840–849. [PMC free article] [PubMed]
17. Ahmad R, Kyratsis Y, Holmes A. When the user is not the chooser: learning from stakeholder involvement in technology adoption decisions in infection control. J. Hosp. Infect. 2012;81:163–168. [PubMed]
18. Cottrell E, Whitlock E, Kato E, et al. Defining the Benefits of Stakeholder Engagement in Systematic Reviews. Agency for Healthcare Research and Quality (US); Rockville, MD, USA: 2014. [PubMed]
19. Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematicreview. BMC Health Serv. Res. 2014;14:89. [PMC free article] [PubMed]
20. Concannon TW, Meissner P, Grunbaum JA, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 2012;27:985–991. [PMC free article] [PubMed]•• Presents the 7Ps framework for identifying key groups of stakeholders for CER.
21. Frank L, Basch E, Selby JV. The PCORI perspective on patient-centered outcomes research. JAMA. 2014;312(15):1513–1514. [PubMed]
22. (Ahrq) AHRQ. The Effective Health Care Program Stakeholder Guide.
23. Guise J-M, O'haire C, Mcpheeters M, et al. A practice-based tool for engaging stakeholders in future research: a synthesis of current practices. J. Clin. Epidemiol. 2013;66:666–674. [PubMed]
24. Devoe J, Angier H, Likumahuwa S, et al. Use of qualitative methods and user-centered design to develop customized health information technology tools within federally qualified health centers to keep children insured. J. Ambul. Care Manage. 2014;37(2):148–154. [PubMed]
25. Devoe JE, Gold R, Cottrell E, et al. The ADVANCE network: accelerating data value across a national community health center network. J. Am. Med. Inform. Assoc. 2014;21(4):591–595. [PMC free article] [PubMed]
26. Devoe JE, Sears A. The OCHIN community information network: bringing together community health centers, information technology, and data to support a patient-centered medical village. J. Am. Board Fam. Med. 2013;26(3):271–278. [PMC free article] [PubMed]
27. Devoe JE, Gold R, Spofford M, et al. Developing a network of community health centers with a common electronic health record: description of the Safety Net West Practice-Based Research Network (SNW-PBRN) J. Am. Board Fam. Med. 2011;24(5):597–604. [PMC free article] [PubMed]
28. Heintzman J, Likumahuwa S, Nelson C, et al. “Not a kidney or a lung”: challenges to conducting practice-based research in community health. Centers Fam. Med. 2014;46(2):105–111. [PMC free article] [PubMed]
29. Arkind J, Likumahuwa-Ackman S, Warren N, et al. Early lessons learned from developing a patient engagement panel: challenges and opportunities. J. Am. Board Fam. Med. In Press.
30. Devoe JE, Krois L, Edlund T, Smith J, Carlson NE. Uninsurance among children whose parents are losing medicaid coverage: results from a statewide survey of Oregon families. Health Serv. Res. 2008;43(1 Pt 2):401–418. [PMC free article] [PubMed]
31. Devoe JE, Baez A, Angier H, Krois L, Edlund C, Carney PA. Insurance + access ≠ health care: typology of barriers to health care access for low-income families. Ann. Fam. Med. 2007;5(6):511–518. [PubMed]
32. Devoe JE, Marino M, Angier H, et al. Effect of expanding medicaid for parents on children's health insurance coverage: lessons from the Oregon experiment. JAMA Pediatr. 2015;169(1):e143145. [PMC free article] [PubMed]
33. Miller WL, Crabtree BF. Qualitative analysis: how to begin making sense. Fam. Pract. Res. J. 1994;14(3):289–297. [PubMed]
34. Torgerson DJ. Contamination in trials: is cluster randomisation the answer? BMJ. 2001;322(7282):355–357. [PMC free article] [PubMed]
35. The Henry J. Kaiser Foundation: Summary of the Affordable Care Act.
36. Jaspers MW, Steen T, Van Den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int. J. Med. Inform. 2004;73(11–12):781–795. [PubMed]
37. Morville P. User Experience Design. Semantic Studios.
38. Patient-Centered Outcomes Research Institute: cycle I funding announcement: improving healthcare systems.

Articles from Journal of Comparative Effectiveness Research are provided here courtesy of Future Science Group