We undertook a systematic evaluation of the stakeholder engagement process supporting CANCERGEN, using a published evaluation framework that specifies four key components of any evaluation of a deliberative process (representation, process, information and outcomes) [9
]. Our principal findings were that stakeholders generally felt that the 2-year process was fair and competent and that the goal of establishing a stakeholder-driven priority-setting process for cancer genomics CER in a clinical trials cooperative group was achieved. Nevertheless, based primarily on information obtained through semistructured interviews, stakeholders identified the following challenges:
- Lack of transparency in recruitment processes
- Unclear role expectations for stakeholders
- Limited opportunities to communicate with clinical investigators
- Lack of clarity on how stakeholder input was integrated into study design decisions
- The challenge of balancing timely input from stakeholders with the pace of research needs
They also provided specific recommendations for improvements in each of the four domains for future stakeholder engagement projects in cancer (); however, major lessons learned from the evaluation include the need to develop standard stakeholder recruitment practices, role descriptions and procedures for improved communication with clinical and CER investigators. While we intend to apply these insights in future projects, given that stakeholder engagement is relatively new in its application to CER, we conclude that evaluation should be routinely conducted, particularly for multiyear projects that are likely to require significant resources.
This study used both questionnaire and interview methods to explore known areas of concern in engaging stakeholders in CER, while also encouraging frank discussion with investigators to obtain valuable feedback in order to improve on future stakeholder engagement practices in oncology. For example, stakeholders overwhelmingly supported the small group size chosen for the CANCERGEN ESAG, yet small group size has been a concern expressed by CER researchers worried that this constraint could limit the perspectives represented, thus influencing the decisions or recommendations made by the group [8
]. While preferring the perceived advantages of a small group size, the CANCERGEN ESAG noted this trade-off. For any future projects, investigators will need to weigh the pros and cons of altering the size of their stakeholder group. Most importantly, all parties need to recognize that stakeholder groups will never be ‘representative’ in the statistical sense. They are chosen to give a rich range of perspectives and there are qualitative methods involving purposive sampling for ensuring this goal is achieved [19
]. Another insight gained during the interviews was the importance of formalizing the recruitment process, including developing a project role description and ensuring that conflicts of interest are explicitly disclosed. Stakeholders were selected based on their knowledge, experience and direct interest in a given topic, increasing the likelihood for conflicts of interest. Managing these potential conflicts, while ensuring a fair and transparent process, is an area of intense interest in CER [21
]. At present, while obtaining disclosure of conflicts of interest from all stakeholders is recommended, more guidance is warranted on this topic and should be a continued focus for future research.
Results from the questionnaire and interviews also revealed complexities in the collaboration between CER and clinical investigators that directly affected stakeholder perceptions of engagement processes. The only single rating of disagreement was provided in response to the online questionnaire item regarding the need for greater clarity regarding how ESAG input was incorporated into decision-making processes. While this represents a very small percentage of the questionnaire responses, it warrants examination as the timeliness and completeness of communications were extensively discussed during the follow-up interviews. The practical realities of coordinating CANCERGEN objectives within the existing SWOG decision-making infrastructure sometimes resulted in communication gaps between the ESAG and clinical investigators. This was most notable when the ERCC1 expression test (to predict response to platinum-based adjuvant therapy in patients with non-small-cell lung cancer) was chosen as the top priority test by the ESAG, and then was not selected by clinical investigators for further advancement within SWOG. The reason this study could not be initiated in the short-term was strong competition for non-small-cell lung cancer patients from industry-sponsored trials, although it is likely to be pursued at a later time. In addition, resource constraints were another reality adversely affecting the short-term likelihood of pursuing a prospective evaluation of the second-ranked test, breast cancer tumor markers, given the significant level of investment required to support the proposed definitive clinical trial. A claims-based analysis of current practice patterns and cost implications of monitoring breast cancer patients with these tumor markers is currently underway as an intermediate step to inform future studies.
These factors were outside the control of CANCERGEN but also were not always clearly communicated to the ESAG. This experience directly affected some ESAG members’ perceptions of their ability to meaningfully prioritize research topics. In the future, greater interaction and communication with clinical investigators would help the ESAG understand how their input was being received within the SWOG disease committees and the competing issues influencing research decisions. Stakeholders openly expressed during their individual interviews that they were interested in knowing that their contributions had an impact on project objectives and that different (and hopefully better) decisions resulted from their participation in the project. This view is entirely consistent with the conclusions of UK experts on public participation who hoped to avoid feelings of cynicism and tokenism when they wrote, “It is a waste of everyone’s time unless the decision-maker is willing to listen to others’ views and then do something which it would not have done otherwise” [10
Ensuring that all stakeholders felt they had sufficient information to meaningfully participate in project-related discussions while avoiding ‘stakeholder fatigue’ during the 2 years of the project was another difficult line to navigate to everyone’s satisfaction. Some variable feedback was received regarding the amount and content of technical information provided to the ESAG prior to each engagement activity. A few stakeholders felt more information would be better while others felt that more information would be overwhelming and difficult to process. Similarly, there were mixed views with respect to how technically sophisticated the background information should be to support informed decision-making. This is an important consideration for stakeholder engagement in CER, as the benefit of bringing diverse stakeholders to the table lies in being able to view research topics through the lens of decision-makers such as patients, clinicians and policymakers. Stakeholders will have different information needs and while ideally, tailoring information to the respective stakeholders is recommended, this may not always be feasible owing to time and resource constraints. The goal is always to make information as accessible as possible and minimize the unavoidable power imbalances that occur when there are ‘experts’ and lay persons participating as members of a stakeholder group. The issue of managing information across a diverse group of stakeholders is not a unique challenge to CER. For example, one stakeholder recommended that for future projects researchers look to other healthcare organizations who routinely communicate information to diverse stakeholders with varied information needs such as the American Heart Association and the National Quality Forum.
The time required to actively involve stakeholders throughout research activities has been a noted concern of researchers [8
]. In particular, the additional activities supporting stakeholder involvement can add to timelines for completing research projects, especially those occurring as part of in-person meetings [8
]. Interestingly, in the CANCERGEN project, stakeholders highlighted this issue as a concern and noted the importance of establishing processes to obtain stakeholder input more efficiently and ensure that communication feedback loops with investigators and SWOG were closed in a timelier manner so that stakeholder input could match the pace of scientific advancements in genomics.
This evaluation of stakeholder engagement in a multiyear CER initiative demonstrates an important, but often missed, aspect of CER activities. Evaluation of stakeholder engagement is essential for establishing legitimacy, measuring impact and informing future improvements [22
]. However, there are few formal evaluations of the effectiveness of stakeholder engagement in CER and no established best practices [101
]. We based our evaluation on four domains for evaluating deliberative processes: representation, the structure of the process or procedures, the information used in the process and the outcomes and decisions arising from the process [9
]. The roots of this approach rely on the meta-principles of fairness and competence for judging the effectiveness of deliberative processes, but operationalize these normative criteria into process and outcome measures [22
]. Defining stakeholder engagement in CER as an iterative process of actively soliciting stakeholder perspectives for the dual purposes of decision-making and achieving a shared understanding with investigators allows comparisons to the evaluation literature on deliberative processes [9
]. In particular, a central component of deliberative processes involves informed participants engaging in discussion to understand others’ views on the topic to arrive at a final decision or recommendation on a topic [9
]. Given that stakeholder engagement in CER has exactly these goals, we adapted the evaluation framework from the public engagement literature, recognizing that target stakeholder groups for CER projects routinely go beyond the public (patients) to include clinicians, payers, policymakers and industry.
While the four-component evaluation framework was developed to guide evaluation processes of public involvement in the healthcare sector, it has not been specifically used in the CER setting [9
]. Moreover, the specific adaptation of each domain in the form of questionnaire and interview questions was not pilot-tested or validated prior to administration. We chose the multimodal approach of sequentially obtaining online ratings followed by telephone-based interviews in order to encourage candid feedback from each stakeholder. However, we found that the self-administered questionnaire was not a particularly sensitive measure of stakeholder perceptions of the effectiveness of the engagement process and relied extensively on the semistructured interview responses for insights about lessons learned and recommendations for improvements. In future evaluations, we would expand our questionnaire to also include questions based on the conceptual framework of stakeholder engagement in CER and validate them prior to administration [27
]. The goal would be to have a more comprehensive questionnaire that could be used to evaluate future CER projects [27
In addition, the evaluation only occurred at the completion of a 2-year process. Thus, the responses reflect the stakeholders’ evaluation of the process as a whole and at a snapshot in time, and do not provide specific detail about different engagement activities over the project lifespan. Further, responses at the end of the project were subject to recall bias, potentially limiting the level of feedback stakeholders could provide on early experiences in the project. A particular concern would be that the passage of time obscured notable frustrations or enthusiasms regarding certain aspects of the engagement. Evaluations conducted at regular intervals throughout the engagement process would provide an opportunity to obtain timely feedback specific to different types of engagement and would address any issues or concerns in a timely manner. Finally, we would also include CER and clinical investigators as participants in the evaluation, as they are also stakeholders in the process and their involvement in the development and implementation of best practices is critical.