The regional stakeholder-based prioritization process developed for the ADVICE project addresses several potential obstacles to effective stakeholder engagement in CER. Early involvement of stakeholders enabled strong alignment of the research objectives of greatest priority for the region. Logistically, relying on a group of regional, rather than national, stakeholders and clinical experts made scheduling of multiple face-to-face meetings feasible, and was time and cost efficient. As other researchers have found, face-to-face stakeholder meetings were crucial in establishing common goals, mutual understanding amongst all parties and overall buy-in of the project [24
]. Similarly, stakeholder engagement was maintained by involving the group in data collection, study design and implementation phases. Finally, the same stakeholders will also be involved in disseminating the results to impact community clinical practice as the research is completed.
The stakeholder group and prioritization process developed as a part of this project were unique in many aspects. Stakeholders were involved not only in initial selection of research topics, but also in the research design and implementation. Many of the stakeholders directly contributed data to our studies, which made the research results readily applicable to their organizations, and also streamlined the satisfaction of legal hurdles required at each organization prior to data collection and analysis. Stakeholders also included representatives from organizations with competing financial interests and represented a diverse set of public, managed care and fee-for-service healthcare systems, which allowed us to conduct community-based studies of diagnostic technologies. Our two-step analysis approach involving a rapid-turnaround ‘internal analysis’ of utilization trends before the main analysis allowed stakeholders to consider the feasibility and scope of the CER question as a major part of the prioritization process. An early look at imaging utilization patterns and variability of use in community practice was very helpful, if not essential, to assess study feasibility and inform estimates of economic impact – two important stakeholder criteria in assessing the overall importance of research questions. These data are seldom available from published literature or public data sets and the time and effort required to obtain institutional review board approval and data use agreements to collect this information is significant. Despite the limitations of the analysis, the data were extremely useful for steering study design and addressing feasibility issues.
The sequence of and preparation for stakeholder meetings allowed stakeholders to substantively engage in a wide range of technical questions. By first identifying a fairly broad list of possible topics, allowing adequate time for topic evaluation via a formalized framework with agreed-upon standardized criteria, and then meeting again for a final prioritization session, we found the stakeholders to be well informed and not adversely affected by gaps in knowledge or biased by their particular areas of expertise. As an example, study feasibility was found to be a very influential criterion for the stakeholder decision process. However, it would have been difficult to evaluate feasibility without the information that was collected from literature reviews and clinical experts about regional disease incidence, recommended imaging practices and reimbursement policies between stakeholder meetings.
A problem common to CER prioritization efforts has been the unclear path between completion of the prioritization process and the actual implementation of the stakeholder recommendations in conducting CER studies. An important issue in this is the lack of a well-defined mechanism linking prioritization results to CER funding. Past efforts to prioritize research using a stakeholder approach have usually been distinct from implementation efforts. Prioritization has generally been carried out by groups different from CER researchers who would implement the work, and the mechanism for funding the research is usually unspecified. For ADVICE, linking the results of the prioritization process to study implementation was immediate and implicit within the structure of the funded grant, which was somewhat unique to the ARRA applications. The project was funded from the outset as a unified process including both the stakeholder-guided prioritization and implementation stages. Such a design was possible via funding from the 2009 ARRA Stimulus Bill, but whether this model is acceptable to funding agencies in the future is unknown. Funding open-ended, stakeholder-guided research has not been the norm for agencies such as the AHRQ and the NIH [25
]. With the recent release of requests for proposals from the new Patient Centered Outcomes Research Institute (PCORI) [107
], this trend may change in the future.
The ADVICE methodology for stakeholder engagement was not without challenges. Logistically, focus on a regional network of stakeholders made initial contact via telephone or email feasible and effective, yet scheduling compatible times for even regional stakeholders was not trivial. The limited time available for executive-level stakeholder members to participate in the project required the utmost attention to making each meeting, phone call and email communication as efficient as possible. Identifying appropriate stakeholders could also be an issue. ADVICE stakeholders were identified primarily through outreach to key opinion leaders in the community using the network of contacts known to the study team. This is a commonly used method for past research prioritization efforts [8
], and may be particularly suited to identification of regional stakeholders, but this methodology may not be appropriate or effective for other research teams.
Two areas that were of high importance during the landscape analysis phase that were particularly difficult to address were the expected imaging utilization rates and direct costs of diagnostic tests. Few published or publicly available data sets were available, with the possible exception of Medicare data. In the future, establishment of data registries or distributed data networks, such as the Health Maintenance Organization Research Network Virtual Data Warehouse [26
] and US FDA Sentinel project [27
], may allow for rapid and practical identification of high imaging variability and areas of under- or over-utilization that warrant high-priority studies. Likewise, access to such databases for full CER studies could greatly reduce the time and effort required to obtain institutional review board approval and establish data use agreements and protocols, which otherwise are likely to be of a ‘one time use’ form.
Another challenge was how to adequately represent the interests of patient stakeholder groups in CER, particularly during the early phases of project prioritization and selection. Initially, we included a patient perspective by including a representative from the Puget Sound Health Alliance at the kick-off meeting, but this representative was the extent of patient involvement in our process to date. For the ADVICE project, it was not feasible to include patient stakeholders due to the inherent uncertainty around topic selection, and time and budgetary constraints. The range of potential CER topics at the outset of the project was very large, spanning all cancers and phases of diagnosis and treatment. Identification and selection of a representative patient group or groups that would have a broad and unbiased knowledge over the various conditions, imaging tests, outcomes and claims data may have been possible, but the investigators felt that it would be difficult to do this within the framework of the grant timeframe and budget. This is largely uncharted territory in CER research, and one of the priority areas of PCORI [28
]. It would have been more straightforward to involve patient groups from the start if the initial scope of potential research projects were somewhat narrower. For example, if the scope of research was limited to a topic within breast cancer imaging, it would have been natural to consider local breast cancer advocacy groups in the prioritization process. We intend to incorporate patient stakeholders in the dissemination phase of ADVICE, and in fact, this is the topic of an upcoming stakeholder meeting, but ideally, patients would be involved through all phases of the CER project.