Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Comp Eff Res. Author manuscript; available in PMC 2013 March 1.
Published in final edited form as:
PMCID: PMC3480224

Prioritizing comparative effectiveness research for cancer diagnostics using a regional stakeholder approach



This paper describes our process to engage regional stakeholders for prioritizing comparative effectiveness research (CER) in cancer diagnostics. We also describe a novel methodology for incorporating stakeholder data and input to inform the objectives of selected CER studies.

Materials & methods

As an integrated component to establishing the infrastructure for community-based CER on diagnostic technologies, we have assembled a regional stakeholder group composed of local payers, clinicians and state healthcare representatives to not only identify and prioritize CER topics most important to the western Washington State region, but also to inform the study design of selected research areas. A landscape analysis process combining literature searches, expert consultations and stakeholder discussions was used to identify possible CER topics in cancer diagnostics. Stakeholders prioritized the top topics using a modified Delphi/group-nominal method and a standardized evaluation criteria framework to determine a final selected CER study area. Implementation of the selected study was immediate due to a unique American Recovery and Reinvestment Act funding structure involving the same researchers and stakeholders in both the prioritization and execution phases of the project. Stakeholder engagement was enhanced after study selection via a rapid analysis of a subset of payers’ internal claims, coordinated by the research team, to obtain summary data of imaging patterns of use. Results of this preliminary analysis, which we termed an ‘internal analysis,’ were used to determine with the stakeholders the most important and feasible study objectives.


Stakeholders identified PET and MRI in cancers including breast, lung, lymphoma and colorectal as top priorities. In an internal analysis of breast cancer imaging, summary data from three payers demonstrated utilization rates of advanced imaging increased between 2002 and 2009 in the study population, with a great deal of variability in use between different health plans. Assessing whether breast MRI affects treatment decisions was the top breast cancer study objective selected by the stakeholders. There were other high-priority research areas including whether MRI use improved survival that were not deemed feasible with the length of follow-up time following MRI adoption.


Continuous stakeholder engagement greatly enhanced their enthusiasm for the project. We believe CER implementation will be more successful when undertaken by regional stakeholders.

Keywords: breast cancer, cancer imaging, comparative effectiveness research, research prioritization, stakeholder involvement

The Institute of Medicine (IOM) identified stakeholder involvement as a key element of the comparative effectiveness research (CER) effort stimulated by the American Recovery and Reinvestment Act (ARRA) of 2009 [101]. While stakeholder involvement in community-based participatory research has been advocated for many years to improve the relevance of clinical research [1], this is not the norm for clinical research. In fact, academic research has traditionally been initiated by investigators who select the research topic and then define and implement specific aims [2]. This approach is successful in generating research matched to the interests and expertise of the investigators; however, the results do not always correspond to the evidence gaps most important to clinical decision-makers or patients [3,4].

To address the disconnect between research designed by individual investigators and the evidence needs of decision-makers, there has been increased efforts to engage stakeholders in all aspects of CER, particularly during the initial phases of prioritization and selection of specific research topics [5]. In addition to improving research relevance, stakeholder buy-in can aid researchers with access to data, maximizing the utility of the limited resources that are devoted to research.

This article describes a process for engaging stakeholders in a regional panel to assist researchers in defining and prioritizing topics in cancer diagnostics in western Washington State. Many aspects of the methodologies for stakeholder engagement and prioritization described in this paper are readily generalizable to CER studies conducted elsewhere. Convening the stakeholder group was part of an ARRA-funded CER effort called Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics (ADVICE), the goal of which was to develop the infrastructure to perform rapid-response community-based CER and to conduct two proof-of-principle stakeholder-informed CER studies on cancer diagnostics.

Materials & methods

ADVICE project overview

In western Washington State, leaders in medicine, research and health insurance are embracing the idea that healthcare is regional. Referral patterns, the supply of healthcare professionals, employers and even healthcare payers are largely tied to specific localities [6,7]. Solutions to problems in the healthcare system should, therefore, reflect the interests, needs and priorities of regional stakeholders. ADVICE is a CER effort that was designed to combine a multi-institutional and multidisciplinary research team from the University of Washington (WA, USA), the Fred Hutchinson Cancer Research Center (WA, USA) and the Group Health Research Institute with community stakeholders to conduct stakeholder-informed research. The research team consists of experts in epidemiology, oncology, medical imaging, health economics and health policy. Stakeholders include representatives from the major fee-for-service health plans (Premera Blue Cross and Regence Blue Shield), publicly funded systems (Medicare, Medicaid, the Washington State Health Care Authority and the Puget Sound Veterans Affairs Health Care System), and an integrated healthcare provider (Group Health Cooperative). Together, the ADVICE network represents approximately 80% of insured patients in the western Washington region, allowing us to conduct generalizable population-based research.

We hypothesized that stakeholder-informed research would lead to more engagement from the payers in the research and lead to research that has real-life clinical relevance to payers because the regional stakeholders who are empowered to make changes within their systems are participating. To help curb the use of inappropriate imaging that may be contributing to this trend, most payers have begun using third party systems that require physicians to obtain prior authorization before an advanced imaging procedure is reimbursed. While these programs have done well in general, unfortunately the underlying algorithms do not do a good job of guidance in the setting of most cancers, as there is insufficient evidence or even consensus expert opinion on the right guidelines.

In 2010, with active stakeholder involvement, ADVICE began two observational studies in cancer imaging linking Surveillance, Epidemiology and End Results (SEER) cancer registry data to administrative claims data from the aforementioned health plans. The first study was defined broadly in the original grant proposal to be in the area of advanced imaging use in breast cancer; whereas the second research topic and cancer site were exclusively determined by stakeholders. In addition to defining and prioritizing topics, refining objectives and providing data, stakeholders will disseminate results and consider whether reimbursement and payment policies should change in response to our findings.

Stakeholder group recruitment & development

Three principles guided our stakeholder recruitment process:

  • [filled square] Broad definition: while there are differing interpretations of the term ‘stakeholder,’ our definition was relatively broad and based on a the definition suggested by O’Haire in a recent Agency for Healthcare Research and Quality (AHRQ) report on stakeholder engagement: “Stakeholders are individuals or organizations who have a personal or professional interest in the topic” [8];
  • [filled square] Regional participation: in light of the significant regional variation that exists in healthcare access, utilization, costs and outcomes [911], we felt that identification of priorities and implementation of CER findings to improve system efficiency and clinical care would be best addressed by regional stakeholders;
  • [filled square] High level of involvement: while stakeholder involvement in CER can vary considerably (from ad hoc methods of convening stakeholders with limited participation [2,12,13] to formally defined groups with significant time commitment, including multiple meetings, one-on-one interviews, and contributions to manuscripts and reports [14,15]), we felt significant stakeholder involvement, which we defined as frequent face-to-face meetings and involvement in periodic webinars and email exchanges would increase research relevance, establish lasting relationships and ensure active dissemination and implementation.

As payer claims data were to play a major role in the methods used to evaluate imaging effectiveness, it was crucial to include appropriate representation in the stakeholder group. To supplement this core group, leadership of regional healthcare delivery systems and administrators and policy makers representing Washington State healthcare agencies were also contacted. For the initial round of topic prioritization, no patients or patient advocacy groups were included as voting members. The wide range of potential CER topics, imaging techniques and therapeutic areas made it extremely difficult to identify a manageable number of patient groups that would have broad enough knowledge to contribute impartially to stakeholder group discussions. We therefore decided to first focus on a stakeholder group without direct patient representation as a foundational step in demonstrating the feasibility of stakeholder-influenced research. However, a representative from a nonprofit organization, the Puget Sound Health Alliance [102], which represents patient, provider and payer interests, was included during the initial kick-off meeting to provide some consumer perspective.

An initial kick-off meeting was convened with the research team and approximately 20 potential stakeholders representing regional payers, administrators and policy makers discussed the project goals, assessed interest and obtained a list of potential research projects that were of high importance. In order to stimulate discussion, a pre-meeting packet was emailed describing the general goals and scope of the grant, along with an initial list of possible research topics to focus on initially for two proof-of-principle demonstration projects. Subsequent to the meeting, each organization was asked to identify a representative who was invited to attend a second meeting as a voting member of the ADVICE stakeholder group. All organizations agreed to designate a voting representative and as a result of this process, a group of eight voting stakeholder members were defined, including three collaborating payers, two clinicians with specialties in cancer and three state healthcare representatives. Stakeholder meetings were held approximately every 6 months, with regular email communications and webinars between meetings. Meeting attendance was limited to just the voting stakeholder group and research team to result in a discussion group of manageable size, large enough to benefit from group dynamics, yet small enough to allow natural discourse.

The regional stakeholder group was augmented by a nonvoting National Advisory Board with members representing the areas of research, regulatory issues, medical diagnostics industry and ethics. Whereas the charge of the regional stakeholder group was primarily to influence specific research direction and implementation for the project, the National Advisory Board provided higher level assistance, providing input on national imaging guidelines, device regulatory reform, advice on methodologies for diagnostic CER, and longer term project guidance regarding the science of current and future technologies in diagnostics. Figure 1 provides an overview of the overall structure of the ADVICE project.

Figure 1
Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics overview

Topic selection

Topic selection was a two-phase process, summarized in Figure 2. In the first phase, a ‘landscape analysis’ process [16,17], wherein a broad range of potential topics are identified through literature review and expert consultation, was carried out iteratively between the stakeholders and the project team. This process has also been referred to as ‘horizon-scanning’ in the literature [1820]. During the initial kick-off meeting, stakeholders were asked to brainstorm on topics of importance at their institutions related to medical imaging or lab-based tests for cancer diagnosis, follow-up and surveillance. The ADVICE project team then spent several months gathering background information on these and related topics via a broad scan of peer-reviewed and gray literature, and consultations with local clinical experts in medical imaging and lab-based diagnostics. Grey literature searches targeted abstracts reporting recent developments in the field and included meetings relevant to cancer imaging such as the Radiological Society of North America (RSNA), Society for Nuclear Medicine (SNM) and the American Society of Clinical Oncology (ASCO). The goal of the landscape analysis was not to perform a complete systematic review of the potential topics, but to gather information necessary to support an informed roundtable discussion during the next meeting of the voting members of the stakeholder group. Based on the high initial interest in advanced imaging, especially PET, local clinical experts in PET and MRI oncologic imaging were invited to the stakeholder roundtable. A main objective of the stakeholder roundtable was to identify five to ten well-defined topics that would be the subject of a subsequent prioritization process.

Figure 2
Comparative effectiveness research topic selection

Another important objective of the stakeholder roundtable was developing agreement on a set of criteria to frame formal topic evaluation. The criteria were derived from recommendations in the Federal Coordinating Council for Comparative Effectiveness Research Report to the President and the Congress [103] and modeled on criteria used in the prioritization process of health technology assessment groups [21,104]. As seen in Appendix 1, the criteria were divided into three levels: Minimum Criteria that must be satisfied for the topic to be considered; Primary Criteria that should weigh heavily in the decision process; and Secondary Criteria that should also be considered during the evaluation. These criteria could be used for prioritization of almost any topics of interest. Once an initial list of high-priority topics had been identified, we began the prioritization phase, modeled on the mixed methods, modified Delphi/group-nominal method proposed by the Center for Medical Technology Policy (CMTP) group [105].

We developed concise, one-page topic summaries for each candidate research topic that included descriptions of how well the topic fit the evaluation criteria, as well as an importance rating for each criterion to guide the stakeholder’s overall interpretation. The rating used a three-point ‘low, medium, high’ scale, and was agreed upon by the ADVICE project team based on the reviewed literature and summary of expert clinical opinion. An example of a criteria summary for the topic of ‘PET and lung cancer staging’ is included in Appendix 2. To supplement these summaries, the ADVICE project team also created a set of more extensive technical briefs with sections suggested by the CMTP group [105] including:

  • [filled square] Description of the technology, approved indications, known information about diffusion;
  • [filled square] Description of the patient group (with estimated patient numbers), expected utilization rates;
  • [filled square] Current diagnostic or treatment alternatives;
  • [filled square] Known over-, under- and misuse of technology, variations in use or outcomes;
  • [filled square] High level summary of the current research evidence of clinical effectiveness, benefits, harms, and any known evidence gaps, patient safety issues;
  • [filled square] Details of any ongoing or related research activities;
  • [filled square] Direct and indirect healthcare costs (annual and/or lifetime);
  • [filled square] Potential cost–effectiveness as compared with alternative treatments;
  • [filled square] Known information about reimbursement (public and private payers);
  • [filled square] Political climate surrounding the technology, particular reasons for urgency;
  • [filled square] Social or ethical concerns related to the technology;
  • [filled square] Methodological implications (requirements to complete research).

Through the development of these in-depth technical briefs we identified critical evidence gaps that could potentially be addressed with data from our stakeholder group.

Stakeholders were sent the criteria summaries and technical briefs approximately 1 month before the scheduled face-to-face meeting and were asked to rate each topic via a web survey as low, medium or high’ priority based upon their review of the technical briefs. Results of the survey, compiled before the face-to-face meeting, allowed the project team to identify areas of controversy or disinterest, aiding in the organization of a 2-h meeting. During the meeting, a brief overview of each topic was given by the project team, then, after a short discussion, an open voting procedure was used to obtain an overall rating for each topic. In a manner akin to the Medicare Evidence Development and Coverage Advisory Committee (MEDCAC) panel voting procedure [22,106], each stakeholder was asked to give each topic a strength of preference rating (five-point Likert scale, where one represented the lowest and five represented the highest priority). Stakeholders were allowed to give the same rating to different topics, and the voting process was transparent. Stakeholders, selected in random order, held up a card representing their vote as well as an additional card that identified which of the criterion were most influential for that vote. Votes were tabulated and sent to the stakeholders via email after the meeting.

Stakeholder-influenced study objectives

A novel aspect of our stakeholder engagement process was the two-step analysis approach. After stakeholders identified topics of interest, the research team worked with a subset of health plans to conduct a rapid-turnaround ‘internal analysis’ of utilization trends. The purpose of this initial analysis was to establish study feasibility and refine the scope and direction of the CER questions. Second, we conducted a full analysis of data from all payers.

Internal analysis

From the initial discussions with stakeholders, it became clear that the regional patterns of use for breast cancer imaging modalities was unknown to payers, clinicians and policymakers alike. While stakeholders believed that utilization of advanced imaging was increasing with high variability between clinicians, they did not have data reflecting which types of imaging tests were increasing or the time points at which the imaging occurred (e.g., diagnosis, staging, treatment monitoring, or surveillance).

To address this, the ADVICE research team and stakeholder group designed and implemented a rapid-response internal analysis. This analysis used fairly simple inclusion/exclusion criteria and Current Procedural Terminology (CPT) and International Classification of Diseases, Ninth Revision (ICD-9) codes, which were applied by the three participating payers to their own claims records and then reported as summary data in identical shell tables. ADVICE team members collected the de-identified data and presented these data to the voting members of the stakeholder group. This greatly simplified or obviated the requirement for data use agreements between the research team and payer groups, which typically can take months.

After presentation and discussion of the breast cancer internal analysis results during an in-person stakeholder meeting, the stakeholders were asked to consider a number of possible study objectives and to add or modify them as appropriate. After a list of potential study objectives was generated, the stakeholders gave a ‘strength of preference’ rating using the same five-point scale and procedures used for study topic prioritization. Voting results were compiled, sent to the stakeholders via email, and the project research team immediately proceeded to create an analysis plan for the top study objectives.


Study topic prioritization

At the first stakeholder meeting, use of PET/computed tomography (CT), MRI and in vitro biomarkers were discussed as diagnostic areas with evidence gaps. Stakeholders expressed concern over potentially rising costs due to anecdotal evidence of greatly increasing use. Cancers with the greatest incidence were of highest concern, since these were presumed to have the greatest burden on the healthcare system. The stakeholders were also interested in finding the amount of variability that existed between clinicians, healthcare providers and insurance plan types. They felt that variability in imaging could be used to identify over- or under-utilized diagnostic techniques. The following four general topics were identified: determine the variation in advanced imaging practice among the four most common cancers (breast, lung, colorectal and prostate); evaluate PET or PET/CT effectiveness for measuring treatment response and for post-treatment surveillance for survivors of non-small-cell lung cancer (NSCLC); compare the effectiveness of PET/CT versus CT for staging, restaging and surveillance of lymphoma; and identify lab-based surveillance techniques that are most effective for monitoring tumor recurrence.

After the second in-person stakeholder meeting, six candidate studies reached the prioritization phase, excluding breast cancer which had already been chosen as a topic for the first ADVICE study. The six candidate studies were:

  • [filled square] PET/CT and lung cancer staging: for patients with advanced NSCLC, does PET/CT-guided staging influence diagnostic evaluation and procedures, choice of primary therapy, cancer-specific and overall survival and short- and long-term medical care costs when compared with conventional staging?
  • [filled square] PET/CT and lung cancer surveillance: among patients with regional and advanced stage NSCLC who have completed primary therapy, does the use of PET/CT influence post-treatment surveillance, subsequent treatment choices, patient survival and cost of care compared with CT alone?
  • [filled square] PET/CT and head and neck cancer staging: does the use of PET/CT for staging of head and neck cancers influence diagnostic evaluation and procedures, improve cancer-specific and overall survival, and reduce short- and long-term cost of care?
  • [filled square] PET/CT and lymphoma surveillance: does the use of PET/CT for post-treatment surveillance of lymphoma compared with CT alone influence subsequent treatment choices, improve cancer-specific and overall survival, and reduce short- and long-term cost of care?
  • [filled square] PET/CT and thyroid cancer surveillance: does the use of PET/CT for post-treatment surveillance of thyroid cancer of follicular cell origin compared with no imaging influence treatment choices, and reduce short- and long-term cost of care?
  • [filled square] Carcinoembryonic antigen and colorectal cancer surveillance: does the use of carcinoembryonic antigen post-treatment surveillance of colon cancer influence subsequent treatment choices, improve cancer-specific and overall survival, and reduce short- and long-term costs of care?

Final prioritization votes for the candidate topics gathered at the conclusion of the third stakeholder meeting are displayed in Table 1, and a summary of the most important criteria used by the stakeholders is seen in Table 2. PET use during lung cancer staging was the highest ranked study, primarily due to the high incidence of lung cancer, potential high costs of PET imaging for lung cancer patients and the unknown effectiveness of PET in community (as opposed to randomized control trial) settings. In general, economic impact of diagnostic testing was a main concern to the stakeholder group, and the stakeholders cited this criterion as a major factor affecting their votes for both high- and low-priority topics (Table 2). Study feasibility was also an important criterion, particularly for studies that received low-priority scores. Similarly, PET use during lung cancer surveillance was seen as an important evidence gap to fill, but since surveillance PET is not reimbursed by most payers, it was uncertain whether sufficient claims data would be available to make this study viable.

Table 1
Results of candidate study prioritization.
Table 2
Most important criteria affecting prioritization vote.

Breast cancer internal analysis results & study objective prioritization

Three payers contributed to the breast cancer internal analysis to achieve a study population summarized in Table 3. The 7213 women included in the initial enrollment period represent approximately 30% of all women expected to be included in the full ADVICE breast cancer population. Since these data were gathered to establish study feasibility and guide the study design, these results should only be interpreted as general trends of imaging practices in our community. The inherent limitations of the rapidly collected data were fully disclosed and discussed, including the fact that data were limited to the claims that were easily accessible at each site, data may have been analyzed differently at each of the three sites, cancer cases were unconfirmed, results were not linked with outcomes, and there was no way to determine if the imaging was because of breast cancer or because of an unrelated medical condition.

Table 3
Breast cancer internal analysis population.

Figure 3 displays the utilization rates by year for PET/CT, MRI and CT for the three plans during the study period and shows the marked increase of MRI and PET/CT between 2002 and 2009. Use of mammography and ultrasound remained relatively stable during the same period [Unpublished Data]. Figure 4 displays utilization rates by imaging interval with respect to date of diagnosis. Mammography and ultrasound use were both high in the period before diagnosis, as this period also includes imaging done for conventional screening. Figure 4 also gives an indication of the variability for advanced imaging between health plans. Most advanced imaging was used in the 6-month period after diagnosis, and we observed a great deal of variation in usage between plans. Two plans showed markedly higher usage of both MRI and PET compared with the third plan in the study. The ratio of each modality’s use in the pretreatment period compared with the post-treatment period differed by plan (Figure 5). While mammography and ultrasound were used fairly consistently by all three plans, we observed more plan variability in the use of advanced imaging. For example, site B clinicians used MRI and PET/CT proportionally much more in the post-treatment setting than clinicians who provided care under site A and C insurance plans.

Figure 3
Breast cancer imaging internal analysis results
Figure 4
Breast cancer imaging internal analysis – variability by payer plan
Figure 5
Breast cancer pretreatment/post-treatment imaging variability

Results of the ratings of study objectives are summarized in Table 4. The first four objectives focus on advanced imaging used with early stage cancer; because PET/CT is not generally reimbursed for early stage breast cancer, these objectives reflect MRI imaging only. A fifth potential objective considered PET/CT usage in regionally advanced breast cancer. MRI utilization rates were much higher than PET/CT rates based on the internal analysis, so the stakeholders felt the MRI objectives were more important. Assessing whether MRI affects treatment decisions – potentially increasing the mastectomy rate with no survival benefit [23] – was judged as the top priority study objective, followed by costs of care.

Table 4
Breast cancer study candidate objectives.


The regional stakeholder-based prioritization process developed for the ADVICE project addresses several potential obstacles to effective stakeholder engagement in CER. Early involvement of stakeholders enabled strong alignment of the research objectives of greatest priority for the region. Logistically, relying on a group of regional, rather than national, stakeholders and clinical experts made scheduling of multiple face-to-face meetings feasible, and was time and cost efficient. As other researchers have found, face-to-face stakeholder meetings were crucial in establishing common goals, mutual understanding amongst all parties and overall buy-in of the project [24]. Similarly, stakeholder engagement was maintained by involving the group in data collection, study design and implementation phases. Finally, the same stakeholders will also be involved in disseminating the results to impact community clinical practice as the research is completed.

The stakeholder group and prioritization process developed as a part of this project were unique in many aspects. Stakeholders were involved not only in initial selection of research topics, but also in the research design and implementation. Many of the stakeholders directly contributed data to our studies, which made the research results readily applicable to their organizations, and also streamlined the satisfaction of legal hurdles required at each organization prior to data collection and analysis. Stakeholders also included representatives from organizations with competing financial interests and represented a diverse set of public, managed care and fee-for-service healthcare systems, which allowed us to conduct community-based studies of diagnostic technologies. Our two-step analysis approach involving a rapid-turnaround ‘internal analysis’ of utilization trends before the main analysis allowed stakeholders to consider the feasibility and scope of the CER question as a major part of the prioritization process. An early look at imaging utilization patterns and variability of use in community practice was very helpful, if not essential, to assess study feasibility and inform estimates of economic impact – two important stakeholder criteria in assessing the overall importance of research questions. These data are seldom available from published literature or public data sets and the time and effort required to obtain institutional review board approval and data use agreements to collect this information is significant. Despite the limitations of the analysis, the data were extremely useful for steering study design and addressing feasibility issues.

The sequence of and preparation for stakeholder meetings allowed stakeholders to substantively engage in a wide range of technical questions. By first identifying a fairly broad list of possible topics, allowing adequate time for topic evaluation via a formalized framework with agreed-upon standardized criteria, and then meeting again for a final prioritization session, we found the stakeholders to be well informed and not adversely affected by gaps in knowledge or biased by their particular areas of expertise. As an example, study feasibility was found to be a very influential criterion for the stakeholder decision process. However, it would have been difficult to evaluate feasibility without the information that was collected from literature reviews and clinical experts about regional disease incidence, recommended imaging practices and reimbursement policies between stakeholder meetings.

A problem common to CER prioritization efforts has been the unclear path between completion of the prioritization process and the actual implementation of the stakeholder recommendations in conducting CER studies. An important issue in this is the lack of a well-defined mechanism linking prioritization results to CER funding. Past efforts to prioritize research using a stakeholder approach have usually been distinct from implementation efforts. Prioritization has generally been carried out by groups different from CER researchers who would implement the work, and the mechanism for funding the research is usually unspecified. For ADVICE, linking the results of the prioritization process to study implementation was immediate and implicit within the structure of the funded grant, which was somewhat unique to the ARRA applications. The project was funded from the outset as a unified process including both the stakeholder-guided prioritization and implementation stages. Such a design was possible via funding from the 2009 ARRA Stimulus Bill, but whether this model is acceptable to funding agencies in the future is unknown. Funding open-ended, stakeholder-guided research has not been the norm for agencies such as the AHRQ and the NIH [25]. With the recent release of requests for proposals from the new Patient Centered Outcomes Research Institute (PCORI) [107], this trend may change in the future.

The ADVICE methodology for stakeholder engagement was not without challenges. Logistically, focus on a regional network of stakeholders made initial contact via telephone or email feasible and effective, yet scheduling compatible times for even regional stakeholders was not trivial. The limited time available for executive-level stakeholder members to participate in the project required the utmost attention to making each meeting, phone call and email communication as efficient as possible. Identifying appropriate stakeholders could also be an issue. ADVICE stakeholders were identified primarily through outreach to key opinion leaders in the community using the network of contacts known to the study team. This is a commonly used method for past research prioritization efforts [8], and may be particularly suited to identification of regional stakeholders, but this methodology may not be appropriate or effective for other research teams.

Two areas that were of high importance during the landscape analysis phase that were particularly difficult to address were the expected imaging utilization rates and direct costs of diagnostic tests. Few published or publicly available data sets were available, with the possible exception of Medicare data. In the future, establishment of data registries or distributed data networks, such as the Health Maintenance Organization Research Network Virtual Data Warehouse [26] and US FDA Sentinel project [27], may allow for rapid and practical identification of high imaging variability and areas of under- or over-utilization that warrant high-priority studies. Likewise, access to such databases for full CER studies could greatly reduce the time and effort required to obtain institutional review board approval and establish data use agreements and protocols, which otherwise are likely to be of a ‘one time use’ form.

Another challenge was how to adequately represent the interests of patient stakeholder groups in CER, particularly during the early phases of project prioritization and selection. Initially, we included a patient perspective by including a representative from the Puget Sound Health Alliance at the kick-off meeting, but this representative was the extent of patient involvement in our process to date. For the ADVICE project, it was not feasible to include patient stakeholders due to the inherent uncertainty around topic selection, and time and budgetary constraints. The range of potential CER topics at the outset of the project was very large, spanning all cancers and phases of diagnosis and treatment. Identification and selection of a representative patient group or groups that would have a broad and unbiased knowledge over the various conditions, imaging tests, outcomes and claims data may have been possible, but the investigators felt that it would be difficult to do this within the framework of the grant timeframe and budget. This is largely uncharted territory in CER research, and one of the priority areas of PCORI [28]. It would have been more straightforward to involve patient groups from the start if the initial scope of potential research projects were somewhat narrower. For example, if the scope of research was limited to a topic within breast cancer imaging, it would have been natural to consider local breast cancer advocacy groups in the prioritization process. We intend to incorporate patient stakeholders in the dissemination phase of ADVICE, and in fact, this is the topic of an upcoming stakeholder meeting, but ideally, patients would be involved through all phases of the CER project.


We have described a regional stakeholder engagement process that has proven to be effective for topic prioritization and study implementation for cancer diagnostics CER. Continuous involvement of payer, clinical and health policy stakeholders from all phases of project development, including project prioritization, study design and contribution of patient-level data greatly enhanced stakeholder engagement and enthusiasm for the project. A focus on regional interests and use of regional data will likely lead to improved dissemination of research results to the clinical community and ultimately a change in clinical practice to make more effective use of diagnostic technology applied to cancer therapies.

Executive summary

  • [filled square] Regional participation of stakeholders from the beginning of the project ensured that research objectives were appropriate to the needs of the local community.
  • [filled square] Proximity of stakeholders made multiple in-person meetings logistically efficient.
  • [filled square] Stakeholders included representatives from organizations with competing financial interests from a diverse set of publicly managed and fee-for-service healthcare systems.
  • [filled square] A high level of involvement with data collection and study design helped stakeholders to establish common goals, develop a mutual understanding, and ensure long-term buy-in to the project.
  • [filled square] Use of an internal analysis improved study efficiency. Although the analysis had limitations, early looks at pooled de-identified data across multiple plans was critical for assessing study feasibility and cross-plan variability as well as informing estimates of economic impact.
  • [filled square] Stakeholders participated in selection and modification of study objectives. Researchers incorporated stakeholder preferences into the internal analyses.
  • [filled square] A focus on regional interests and use of regional data will lead to improved dissemination of results to the clinical community to make more effective use of cancer diagnostic technologies.
  • [filled square] Adequately representing the interests of patient stakeholder groups was challenging, particularly during the early phases of project prioritization and selection.


The authors would like to thank the ADVancing Innovative Comparative Effectiveness Research in Cancer Diagnostics (ADVICE) stakeholder group (J Dominitz, Assistant Chief for Academic Af fairs, Gastroenterolog y, Veterans Administration of Puget Sound Health Care System; J Gifford, Senior Medical Officer, Regence BlueShield; R Herr, Medical Director, Regence BlueShield; L Hole-Curry, [former] Program Director, Health Technology Assessment, Washington State Health Care Authority; M Lonergan, Oncologist, Evergreen Hospital Medical Center; DK McCulloch, Medical Director, Clinical Improvement, Group Health Cooperative; M Stewart, Medical Director, Seattle Cancer Care Alliance ; J Thompson, Chief Medical Officer, Health and Recover Services Administration, DSHS; and JB Watkins, Pharmacy Manager, Premera Blue Cross) for their enthusiasm and extensive contributions to the ADVICE project. The authors would also like to thank their clinical imaging experts, C Lehman and D Mankoff for their extremely helpful contributions. Finally, they would like to thank A Drum for her assistance in document preparation and especially in her persistence in coordinating stakeholder meetings.

This work was supported by the Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics grant (1RC2CA148433-01) from the National Cancer Institute.

Appendix 1. ADVICE study criteria selection matrix.

Criteria for selection of studyDescription of criteria
Minimum criteria
Included within statutory limits of Federal Coordinating Council’s definition of CERIs the topic included within the definition of CER detailed in the mandate of the Recovery Act?
Stakeholder/community interestDoes the topic answer to expressed needs and preferences of patients, clinicians, and other stakeholders?
FeasibilityAre there available databases, registries, collaborations and resources to support study design?
Are data from sufficient numbers of patients available for analysis?
Do the data span sufficient time to evaluate desired outcomes?
Evidence neededIs the modality currently being used without adequate evidence?
Primary criteria
Standards for modality and implementation of test resultsAre guidelines in place for the various uses of this modality? Are existing guidelines evidence-based and unbiased? Are they consistently implemented by providers?
Potential patient harm/safety concernsWhat is the potential degree of harm (e.g., anxiety, radiation exposure, unnecessary treatment) an individual may experience from use of the modality? What are potential downstream harms from incorrect diagnosis using this modality?
Diagnostic accuracy and efficacy potentialHow accurate is the modality for disease diagnosis? Does the modality offer potential for changing patient treatment path or outcome?
Economic impactWhat is the estimated total direct cost per year (increase/decrease) of the modality and downstream treatments?
Secondary criteria
Severity of condition treated/diagnosed by modalityAre significant morbidity, mortality, and/or disability associated with disease?
Pressure on payersIs there a high degree of pressure on payers to cover this modality, even though little evidence supports its use?
Potential or observed variationDoes payer/provider use of modality vary widely?
Diverse populationsIs there potential to evaluate CER in diverse populations and patient sub-groups affected by disease and/or modality?
Overlap/redundancyIs/has similar research being/been conducted elsewhere?
Disease burdenHow many persons are affected by disease per year? What is the disease cost to society?

CER: Comparative effectiveness research.

Appendix 2. Example of criteria summary table: PET and lung cancer staging.

CriteriaEvidence for criteriaCriteria
Minimum criteria
Included within statutory limits of Federal Coordinating Council’s definition of CERYesHigh
Stakeholder/community interestYesHigh
FeasibilityHigh incidence and mortality rate result in relatively large data sets available for analysis through insurance claims data linked to SEER. Initially, would be informative to look at claims data to get idea of variations in use of PET/CT for staging, restaging, and post-treatment surveillance of NSCLC patients in western Washington StateHigh
Evidence neededTwo RCTs performed outside the USA found fewer futile surgeries in the pre-operative PET/CT group, but no difference in survival. Studies looking at use of PET/CT in community (vs academic) setting neededMedium
Primary criteria
Standards for modality and implementation of test resultsNCCN guidelines recommend that PET/CT be used for pretreatment evaluation of NSCLC, but PET/CT is still not regarded as a standard of care for NSCLC staging. Controversy exists regarding PET/CT as an add-on or replacement testHigh
Potential patient harm/safety concernsUnnecessary radiation exposure in patients who receive PET/CT scans; potential unnecessary surgeries due to PET/CT false positives, or delayed treatment for false negatives; also potential for unnecessary surgeries in patients who do not receive PET/CT scansHigh
Diagnostic accuracy and efficacy potentialConsiderable evidence of PET/CT offering superior accuracy in NSCLC staging. RCTs and retrospective studies generally agree that PET/CT offers potential to change patient management, but there is little evidence for survival improvementsHigh
Economic impactAccording to SEER–Medicare data, approximately 12% of lung cancer patients filed claims for PET/CT between 2005 and 2007. Cost of PET/CT is approximately US$2500–4000, but might reduce costs by eliminating unnecessary surgeries and by allowing surgeons to better target areas of resectionHigh
Secondary criteria
Severity of condition treated/diagnosed by modalityHigh incidence and low survival rates; accounts for the most cancer-related deaths. US incidence rate: 69.0/100,000; 5-year survival: 16%; US death rate: 53.4/100,000High
Pressure on payersAnecdotal evidence suggests that use of PET/CT scans is increasing, with no evidence that the scans provide information that decreases morbidity, mortality, or recurrence ratesMedium
Potential or observed variationNo literature on thisNo data
Diverse populationsAccess to PET/CT machines is variable depending on patient locationMedium
Overlap/redundancyTwo RCTs looking at PET/CT for detecting recurrence in several cancers, including lung; none in western Washington StateMedium
Disease burdenApproximately 4000 incident cases in Washington State state per year; US prevalence: 370,617; aggregate 5-year Medicare cost (in 2004 dollars) of care for treatment of lung cancer was estimated to be US$4238 millionHigh

CER: Comparative effectiveness research; CT: Computed tomography; NSCLC: Non-small-cell lung cancer; RCT: Randomized controlled trial; SEER: Surveillance, Epidemiology and End Results.


Financial & competing interests disclosure

The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

Ethical conduct of research

The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.


Papers of special note have been highlighted as:

[filled square]of interest

[filled square][filled square] of considerable interest

1. Minkler M, Wallerstein N. Community-based participatory research for health: from process to outcomes (2nd) Jossey-Bass, CA, USA: 2008.
2. Gold R, Whitlock EP, Patnode CD, McGinnis PS, Buckley DI, Morris C. Prioritizing research needs based on a systematic evidence review: a pilot process for engaging stakeholders. Health Expect. 2011 (Epub ahead of print). [PMC free article] [PubMed]
3. Tong A, Sainsbury P, Carter SM, et al. Patients’ priorities for health research: focus group study of patients with chronic kidney disease. Nephrol. Dial. Transplant. 2008;23(10):3206–3214. [PubMed]
4. Tunis SR, Benner J, McClellan M. Comparative effectiveness research: policy context, methods development and research infrastructure. Stat. Med. 2010;29(19):1963–1976. [PubMed] Excellent overview of policy related to comparative effectiveness research (CER).
5. Whitlock EP, Lopez SA, Chang S, Helfand M, Eder M, Floyd N. AHRQ Series Paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care program. J. Clin. Epidemiol. 2010;63(5):491–501. [PubMed]
6. Fisher ES, Bynum JP, Skinner JS. Slowing the growth of health care costs – lessons from regional variation. N. Engl. J. Med. 2009;360(9):849–852. [PMC free article] [PubMed]
7. Philipson T, Seabury S, Lockwood L, Goldman D, Lakdawalla D. Geographic Variation in Health Care: The Role of Private Markets. Brookings Papers on Economic Activity. 2010;1:325–355.
8. O’Haire C, McPheeters M, Nakamoto E, et al. Methods for Engaging Stakeholders To Identify and Prioritize Future Research Needs. Methods Future Research Needs Report No. 4. (Prepared by the Oregon Evidence-based Practice Center and the Vanderbilt Evidence-based Practice Center under Contract No. 290-2007-10057-I.) 2011 AHRQ Publication No. 11-EHC044-EF. Recent overview of stakeholder engagement methods for comparative effectiveness reviews, applicable to CER in general.
9. Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. The implications of regional variations in Medicare spending. Part 1: the content, quality, and accessibility of care. Ann. Intern. Med. 2003;138(4):273–287. [PubMed]
10. Song Y, Skinner J, Bynum J, Sutherland J, Wennberg JE, Fisher ES. Regional variations in diagnostic practices. N. Engl. J. Med. 2010;363(1):45–53. [PMC free article] [PubMed]
11. Welch HG, Sharp SM, Gottlieb DJ, Skinner JS, Wennberg JE. Geographic variation in diagnosis frequency and risk of death among Medicare beneficiaries. JAMA. 2011;305(11):1113–1118. [PMC free article] [PubMed]
12. Douglas PS, Taylor A, Bild D, et al. Outcomes research in cardiovascular imaging: report of a workshop sponsored by the National Heart, Lung, and Blood Institute. JACC Cardiovasc. Imaging. 2009;2(7):897–907. [PMC free article] [PubMed]
13. Pickard AS, Lee TA, Solem CT, Joo MJ, Schumock GT, Krishnan JA. Prioritizing comparative-effectiveness research topics via stakeholder involvement: an application in COPD. Clin. Pharmacol. Ther. 2011;90(6):888–892. [PubMed]
14. Balshem H, Curtis P, Joplin L, Justmann R, Rosenberg A. Stakeholder Involvement in Improving Comparative Effectiveness Reviews: AHRQ and the Effective Health Care Program (Prepared by the AHRQ Effective Health Care Program Product Development Work Group under Contract No. HHSA 290-2007-10057-I) 2011 AHRQ Publication No. 11-EHC079-EF. Discusses how stakeholder input can make CER more useful for healthcare decision-makers.
15. Chalkidou K, Whicher D, Kary W, Tunis S. Comparative effectiveness research priorities: identifying critical gaps in evidence for clinical and health policy decision making. Int. J. Technol. Assess. Health Care. 2009;25(3):241–248. [PubMed] An insightful discussion of stakeholders and the process of prioritizing CER.
16. Savy M, Edmond K, Fine PE, et al. Landscape analysis of interactions between nutrition and vaccine responses in children. J. Nutr. 2009;139(11):2154S–2218S. [PubMed]
17. Thariani R, Wong W, Carlson JJ, et al. Prioritization in comparative effectiveness research: the CANCERGEN Experience. Med. Care. 2012 (Epub ahead of print). [PMC free article] [PubMed]
18. Cornel MC, Van El CG, Dondorp WJ. The promises of genomic screening: building a governance infrastructure. Special issue: genetics and democracy. J. Community Genet. 2012;3(2):73–77. [PMC free article] [PubMed]
19. Nachtnebel A, Geiger-Gritsch S, Hintringer K, Wild C. Scanning the horizon: development and implementation of an early awareness system for anticancer drugs in Austria. Health Policy. 2012;104(1):1–11. [PubMed]
20. Sutherland WJ, Aveling R, Bennun L, et al. A horizon scan of global conservation issues for 2012. Trends Ecol. Evol. 2012;27(1):12–18. [PubMed]
21. Drummond M, Sorenson C. Nasty or nice? A perspective on the use of health technology assessment in the United Kingdom. Value Health. 2009;12(Suppl. 2):S8–S13. [PubMed]
22. Lavertu S, Walters DE, Weimer DL. Scientific Expertise and the Balance of Political Interests: MEDCAC and Medicare Coverage Decisions. J. Public Adm. Res. Theory. 2012;22(1):55–81.
23. Houssami N, Hayes DF. Review of preoperative magnetic resonance imaging (MRI) in breast cancer: should MRI be performed on all women with newly diagnosed, early stage breast cancer? CA Cancer J. Clin. 2009;59(5):290–302. [PubMed]
24. Gliklich Re LM, Velentgas P, Campion DM, et al. Identification of future research needs in the comparative management of uterine fibroid disease. A Report on the Priority-Setting Process, Preliminary Data Analysis, and Research Plan. Effective Healthcare Research Report No. 31. 2011
25. Myers ESG, Ravi D, Matchar D, et al. Evaluating the potential use of modeling and value-of-information analysis for future research prioritization within the evidence-based practice cnter program. Duke Evidence-based Practice Center. 2011 AHRQ Publication No. 11-EHC030-EF. [PubMed] Review of new quantitative methods for prioritizing research.
26. Wallace PJ. Reshaping cancer learning through the use of health information technology. Health Aff. (Millwood) 2007;26(2):w169–w177. [PubMed]
27. Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R. Developing the Sentinel System – a national resource for evidence development. N. Engl. J. Med. 2011;364(6):498–499. [PubMed]
28. Zeliadt SB, Etzioni R, Ramsey SD, Penson DF, Potosky AL. Trends in treatment costs for localized prostate cancer: the healthy screenee effect. Med. Care. 2007;45(2):154–159. [PubMed]


101. Initial National Priorities for Comparative Effectiveness Research. [Accessed 14 November 2011];
102. Puget Sound Health Alliance. [Accessed 3 January 2012];
103. Federal Coordinating Council for Comparative Effectiveness Research Report to the President and the Congress. [Accessed 14 November 2011];
104. Washington State Health Care Authority Prioritization Criteria. [Accessed 14 November 2011];
105. Overview of CMTP’s Priority-Setting Process. [Accessed 3 November 2011];
106. Centers for Medicare and Medicaid Services (CMS): Medicare Evidence Development & Coverage Advisory Committee (MEDCAC) Overview. [Accessed 11 April 2010];
107. Patient-Centered Outcomes Research Institute (PCORI) [Accessed 4 November 2011];