|Home | About | Journals | Submit | Contact Us | Français|
This paper describes our process to engage regional stakeholders for prioritizing comparative effectiveness research (CER) in cancer diagnostics. We also describe a novel methodology for incorporating stakeholder data and input to inform the objectives of selected CER studies.
As an integrated component to establishing the infrastructure for community-based CER on diagnostic technologies, we have assembled a regional stakeholder group composed of local payers, clinicians and state healthcare representatives to not only identify and prioritize CER topics most important to the western Washington State region, but also to inform the study design of selected research areas. A landscape analysis process combining literature searches, expert consultations and stakeholder discussions was used to identify possible CER topics in cancer diagnostics. Stakeholders prioritized the top topics using a modified Delphi/group-nominal method and a standardized evaluation criteria framework to determine a final selected CER study area. Implementation of the selected study was immediate due to a unique American Recovery and Reinvestment Act funding structure involving the same researchers and stakeholders in both the prioritization and execution phases of the project. Stakeholder engagement was enhanced after study selection via a rapid analysis of a subset of payers’ internal claims, coordinated by the research team, to obtain summary data of imaging patterns of use. Results of this preliminary analysis, which we termed an ‘internal analysis,’ were used to determine with the stakeholders the most important and feasible study objectives.
Stakeholders identified PET and MRI in cancers including breast, lung, lymphoma and colorectal as top priorities. In an internal analysis of breast cancer imaging, summary data from three payers demonstrated utilization rates of advanced imaging increased between 2002 and 2009 in the study population, with a great deal of variability in use between different health plans. Assessing whether breast MRI affects treatment decisions was the top breast cancer study objective selected by the stakeholders. There were other high-priority research areas including whether MRI use improved survival that were not deemed feasible with the length of follow-up time following MRI adoption.
Continuous stakeholder engagement greatly enhanced their enthusiasm for the project. We believe CER implementation will be more successful when undertaken by regional stakeholders.
The Institute of Medicine (IOM) identified stakeholder involvement as a key element of the comparative effectiveness research (CER) effort stimulated by the American Recovery and Reinvestment Act (ARRA) of 2009 . While stakeholder involvement in community-based participatory research has been advocated for many years to improve the relevance of clinical research , this is not the norm for clinical research. In fact, academic research has traditionally been initiated by investigators who select the research topic and then define and implement specific aims . This approach is successful in generating research matched to the interests and expertise of the investigators; however, the results do not always correspond to the evidence gaps most important to clinical decision-makers or patients [3,4].
To address the disconnect between research designed by individual investigators and the evidence needs of decision-makers, there has been increased efforts to engage stakeholders in all aspects of CER, particularly during the initial phases of prioritization and selection of specific research topics . In addition to improving research relevance, stakeholder buy-in can aid researchers with access to data, maximizing the utility of the limited resources that are devoted to research.
This article describes a process for engaging stakeholders in a regional panel to assist researchers in defining and prioritizing topics in cancer diagnostics in western Washington State. Many aspects of the methodologies for stakeholder engagement and prioritization described in this paper are readily generalizable to CER studies conducted elsewhere. Convening the stakeholder group was part of an ARRA-funded CER effort called Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics (ADVICE), the goal of which was to develop the infrastructure to perform rapid-response community-based CER and to conduct two proof-of-principle stakeholder-informed CER studies on cancer diagnostics.
In western Washington State, leaders in medicine, research and health insurance are embracing the idea that healthcare is regional. Referral patterns, the supply of healthcare professionals, employers and even healthcare payers are largely tied to specific localities [6,7]. Solutions to problems in the healthcare system should, therefore, reflect the interests, needs and priorities of regional stakeholders. ADVICE is a CER effort that was designed to combine a multi-institutional and multidisciplinary research team from the University of Washington (WA, USA), the Fred Hutchinson Cancer Research Center (WA, USA) and the Group Health Research Institute with community stakeholders to conduct stakeholder-informed research. The research team consists of experts in epidemiology, oncology, medical imaging, health economics and health policy. Stakeholders include representatives from the major fee-for-service health plans (Premera Blue Cross and Regence Blue Shield), publicly funded systems (Medicare, Medicaid, the Washington State Health Care Authority and the Puget Sound Veterans Affairs Health Care System), and an integrated healthcare provider (Group Health Cooperative). Together, the ADVICE network represents approximately 80% of insured patients in the western Washington region, allowing us to conduct generalizable population-based research.
We hypothesized that stakeholder-informed research would lead to more engagement from the payers in the research and lead to research that has real-life clinical relevance to payers because the regional stakeholders who are empowered to make changes within their systems are participating. To help curb the use of inappropriate imaging that may be contributing to this trend, most payers have begun using third party systems that require physicians to obtain prior authorization before an advanced imaging procedure is reimbursed. While these programs have done well in general, unfortunately the underlying algorithms do not do a good job of guidance in the setting of most cancers, as there is insufficient evidence or even consensus expert opinion on the right guidelines.
In 2010, with active stakeholder involvement, ADVICE began two observational studies in cancer imaging linking Surveillance, Epidemiology and End Results (SEER) cancer registry data to administrative claims data from the aforementioned health plans. The first study was defined broadly in the original grant proposal to be in the area of advanced imaging use in breast cancer; whereas the second research topic and cancer site were exclusively determined by stakeholders. In addition to defining and prioritizing topics, refining objectives and providing data, stakeholders will disseminate results and consider whether reimbursement and payment policies should change in response to our findings.
Three principles guided our stakeholder recruitment process:
As payer claims data were to play a major role in the methods used to evaluate imaging effectiveness, it was crucial to include appropriate representation in the stakeholder group. To supplement this core group, leadership of regional healthcare delivery systems and administrators and policy makers representing Washington State healthcare agencies were also contacted. For the initial round of topic prioritization, no patients or patient advocacy groups were included as voting members. The wide range of potential CER topics, imaging techniques and therapeutic areas made it extremely difficult to identify a manageable number of patient groups that would have broad enough knowledge to contribute impartially to stakeholder group discussions. We therefore decided to first focus on a stakeholder group without direct patient representation as a foundational step in demonstrating the feasibility of stakeholder-influenced research. However, a representative from a nonprofit organization, the Puget Sound Health Alliance , which represents patient, provider and payer interests, was included during the initial kick-off meeting to provide some consumer perspective.
An initial kick-off meeting was convened with the research team and approximately 20 potential stakeholders representing regional payers, administrators and policy makers discussed the project goals, assessed interest and obtained a list of potential research projects that were of high importance. In order to stimulate discussion, a pre-meeting packet was emailed describing the general goals and scope of the grant, along with an initial list of possible research topics to focus on initially for two proof-of-principle demonstration projects. Subsequent to the meeting, each organization was asked to identify a representative who was invited to attend a second meeting as a voting member of the ADVICE stakeholder group. All organizations agreed to designate a voting representative and as a result of this process, a group of eight voting stakeholder members were defined, including three collaborating payers, two clinicians with specialties in cancer and three state healthcare representatives. Stakeholder meetings were held approximately every 6 months, with regular email communications and webinars between meetings. Meeting attendance was limited to just the voting stakeholder group and research team to result in a discussion group of manageable size, large enough to benefit from group dynamics, yet small enough to allow natural discourse.
The regional stakeholder group was augmented by a nonvoting National Advisory Board with members representing the areas of research, regulatory issues, medical diagnostics industry and ethics. Whereas the charge of the regional stakeholder group was primarily to influence specific research direction and implementation for the project, the National Advisory Board provided higher level assistance, providing input on national imaging guidelines, device regulatory reform, advice on methodologies for diagnostic CER, and longer term project guidance regarding the science of current and future technologies in diagnostics. Figure 1 provides an overview of the overall structure of the ADVICE project.
Topic selection was a two-phase process, summarized in Figure 2. In the first phase, a ‘landscape analysis’ process [16,17], wherein a broad range of potential topics are identified through literature review and expert consultation, was carried out iteratively between the stakeholders and the project team. This process has also been referred to as ‘horizon-scanning’ in the literature [18–20]. During the initial kick-off meeting, stakeholders were asked to brainstorm on topics of importance at their institutions related to medical imaging or lab-based tests for cancer diagnosis, follow-up and surveillance. The ADVICE project team then spent several months gathering background information on these and related topics via a broad scan of peer-reviewed and gray literature, and consultations with local clinical experts in medical imaging and lab-based diagnostics. Grey literature searches targeted abstracts reporting recent developments in the field and included meetings relevant to cancer imaging such as the Radiological Society of North America (RSNA), Society for Nuclear Medicine (SNM) and the American Society of Clinical Oncology (ASCO). The goal of the landscape analysis was not to perform a complete systematic review of the potential topics, but to gather information necessary to support an informed roundtable discussion during the next meeting of the voting members of the stakeholder group. Based on the high initial interest in advanced imaging, especially PET, local clinical experts in PET and MRI oncologic imaging were invited to the stakeholder roundtable. A main objective of the stakeholder roundtable was to identify five to ten well-defined topics that would be the subject of a subsequent prioritization process.
Another important objective of the stakeholder roundtable was developing agreement on a set of criteria to frame formal topic evaluation. The criteria were derived from recommendations in the Federal Coordinating Council for Comparative Effectiveness Research Report to the President and the Congress  and modeled on criteria used in the prioritization process of health technology assessment groups [21,104]. As seen in Appendix 1, the criteria were divided into three levels: Minimum Criteria that must be satisfied for the topic to be considered; Primary Criteria that should weigh heavily in the decision process; and Secondary Criteria that should also be considered during the evaluation. These criteria could be used for prioritization of almost any topics of interest. Once an initial list of high-priority topics had been identified, we began the prioritization phase, modeled on the mixed methods, modified Delphi/group-nominal method proposed by the Center for Medical Technology Policy (CMTP) group .
We developed concise, one-page topic summaries for each candidate research topic that included descriptions of how well the topic fit the evaluation criteria, as well as an importance rating for each criterion to guide the stakeholder’s overall interpretation. The rating used a three-point ‘low, medium, high’ scale, and was agreed upon by the ADVICE project team based on the reviewed literature and summary of expert clinical opinion. An example of a criteria summary for the topic of ‘PET and lung cancer staging’ is included in Appendix 2. To supplement these summaries, the ADVICE project team also created a set of more extensive technical briefs with sections suggested by the CMTP group  including:
Through the development of these in-depth technical briefs we identified critical evidence gaps that could potentially be addressed with data from our stakeholder group.
Stakeholders were sent the criteria summaries and technical briefs approximately 1 month before the scheduled face-to-face meeting and were asked to rate each topic via a web survey as low, medium or high’ priority based upon their review of the technical briefs. Results of the survey, compiled before the face-to-face meeting, allowed the project team to identify areas of controversy or disinterest, aiding in the organization of a 2-h meeting. During the meeting, a brief overview of each topic was given by the project team, then, after a short discussion, an open voting procedure was used to obtain an overall rating for each topic. In a manner akin to the Medicare Evidence Development and Coverage Advisory Committee (MEDCAC) panel voting procedure [22,106], each stakeholder was asked to give each topic a strength of preference rating (five-point Likert scale, where one represented the lowest and five represented the highest priority). Stakeholders were allowed to give the same rating to different topics, and the voting process was transparent. Stakeholders, selected in random order, held up a card representing their vote as well as an additional card that identified which of the criterion were most influential for that vote. Votes were tabulated and sent to the stakeholders via email after the meeting.
A novel aspect of our stakeholder engagement process was the two-step analysis approach. After stakeholders identified topics of interest, the research team worked with a subset of health plans to conduct a rapid-turnaround ‘internal analysis’ of utilization trends. The purpose of this initial analysis was to establish study feasibility and refine the scope and direction of the CER questions. Second, we conducted a full analysis of data from all payers.
From the initial discussions with stakeholders, it became clear that the regional patterns of use for breast cancer imaging modalities was unknown to payers, clinicians and policymakers alike. While stakeholders believed that utilization of advanced imaging was increasing with high variability between clinicians, they did not have data reflecting which types of imaging tests were increasing or the time points at which the imaging occurred (e.g., diagnosis, staging, treatment monitoring, or surveillance).
To address this, the ADVICE research team and stakeholder group designed and implemented a rapid-response internal analysis. This analysis used fairly simple inclusion/exclusion criteria and Current Procedural Terminology (CPT) and International Classification of Diseases, Ninth Revision (ICD-9) codes, which were applied by the three participating payers to their own claims records and then reported as summary data in identical shell tables. ADVICE team members collected the de-identified data and presented these data to the voting members of the stakeholder group. This greatly simplified or obviated the requirement for data use agreements between the research team and payer groups, which typically can take months.
After presentation and discussion of the breast cancer internal analysis results during an in-person stakeholder meeting, the stakeholders were asked to consider a number of possible study objectives and to add or modify them as appropriate. After a list of potential study objectives was generated, the stakeholders gave a ‘strength of preference’ rating using the same five-point scale and procedures used for study topic prioritization. Voting results were compiled, sent to the stakeholders via email, and the project research team immediately proceeded to create an analysis plan for the top study objectives.
At the first stakeholder meeting, use of PET/computed tomography (CT), MRI and in vitro biomarkers were discussed as diagnostic areas with evidence gaps. Stakeholders expressed concern over potentially rising costs due to anecdotal evidence of greatly increasing use. Cancers with the greatest incidence were of highest concern, since these were presumed to have the greatest burden on the healthcare system. The stakeholders were also interested in finding the amount of variability that existed between clinicians, healthcare providers and insurance plan types. They felt that variability in imaging could be used to identify over- or under-utilized diagnostic techniques. The following four general topics were identified: determine the variation in advanced imaging practice among the four most common cancers (breast, lung, colorectal and prostate); evaluate PET or PET/CT effectiveness for measuring treatment response and for post-treatment surveillance for survivors of non-small-cell lung cancer (NSCLC); compare the effectiveness of PET/CT versus CT for staging, restaging and surveillance of lymphoma; and identify lab-based surveillance techniques that are most effective for monitoring tumor recurrence.
After the second in-person stakeholder meeting, six candidate studies reached the prioritization phase, excluding breast cancer which had already been chosen as a topic for the first ADVICE study. The six candidate studies were:
Final prioritization votes for the candidate topics gathered at the conclusion of the third stakeholder meeting are displayed in Table 1, and a summary of the most important criteria used by the stakeholders is seen in Table 2. PET use during lung cancer staging was the highest ranked study, primarily due to the high incidence of lung cancer, potential high costs of PET imaging for lung cancer patients and the unknown effectiveness of PET in community (as opposed to randomized control trial) settings. In general, economic impact of diagnostic testing was a main concern to the stakeholder group, and the stakeholders cited this criterion as a major factor affecting their votes for both high- and low-priority topics (Table 2). Study feasibility was also an important criterion, particularly for studies that received low-priority scores. Similarly, PET use during lung cancer surveillance was seen as an important evidence gap to fill, but since surveillance PET is not reimbursed by most payers, it was uncertain whether sufficient claims data would be available to make this study viable.
Three payers contributed to the breast cancer internal analysis to achieve a study population summarized in Table 3. The 7213 women included in the initial enrollment period represent approximately 30% of all women expected to be included in the full ADVICE breast cancer population. Since these data were gathered to establish study feasibility and guide the study design, these results should only be interpreted as general trends of imaging practices in our community. The inherent limitations of the rapidly collected data were fully disclosed and discussed, including the fact that data were limited to the claims that were easily accessible at each site, data may have been analyzed differently at each of the three sites, cancer cases were unconfirmed, results were not linked with outcomes, and there was no way to determine if the imaging was because of breast cancer or because of an unrelated medical condition.
Figure 3 displays the utilization rates by year for PET/CT, MRI and CT for the three plans during the study period and shows the marked increase of MRI and PET/CT between 2002 and 2009. Use of mammography and ultrasound remained relatively stable during the same period [Unpublished Data]. Figure 4 displays utilization rates by imaging interval with respect to date of diagnosis. Mammography and ultrasound use were both high in the period before diagnosis, as this period also includes imaging done for conventional screening. Figure 4 also gives an indication of the variability for advanced imaging between health plans. Most advanced imaging was used in the 6-month period after diagnosis, and we observed a great deal of variation in usage between plans. Two plans showed markedly higher usage of both MRI and PET compared with the third plan in the study. The ratio of each modality’s use in the pretreatment period compared with the post-treatment period differed by plan (Figure 5). While mammography and ultrasound were used fairly consistently by all three plans, we observed more plan variability in the use of advanced imaging. For example, site B clinicians used MRI and PET/CT proportionally much more in the post-treatment setting than clinicians who provided care under site A and C insurance plans.
Results of the ratings of study objectives are summarized in Table 4. The first four objectives focus on advanced imaging used with early stage cancer; because PET/CT is not generally reimbursed for early stage breast cancer, these objectives reflect MRI imaging only. A fifth potential objective considered PET/CT usage in regionally advanced breast cancer. MRI utilization rates were much higher than PET/CT rates based on the internal analysis, so the stakeholders felt the MRI objectives were more important. Assessing whether MRI affects treatment decisions – potentially increasing the mastectomy rate with no survival benefit  – was judged as the top priority study objective, followed by costs of care.
The regional stakeholder-based prioritization process developed for the ADVICE project addresses several potential obstacles to effective stakeholder engagement in CER. Early involvement of stakeholders enabled strong alignment of the research objectives of greatest priority for the region. Logistically, relying on a group of regional, rather than national, stakeholders and clinical experts made scheduling of multiple face-to-face meetings feasible, and was time and cost efficient. As other researchers have found, face-to-face stakeholder meetings were crucial in establishing common goals, mutual understanding amongst all parties and overall buy-in of the project . Similarly, stakeholder engagement was maintained by involving the group in data collection, study design and implementation phases. Finally, the same stakeholders will also be involved in disseminating the results to impact community clinical practice as the research is completed.
The stakeholder group and prioritization process developed as a part of this project were unique in many aspects. Stakeholders were involved not only in initial selection of research topics, but also in the research design and implementation. Many of the stakeholders directly contributed data to our studies, which made the research results readily applicable to their organizations, and also streamlined the satisfaction of legal hurdles required at each organization prior to data collection and analysis. Stakeholders also included representatives from organizations with competing financial interests and represented a diverse set of public, managed care and fee-for-service healthcare systems, which allowed us to conduct community-based studies of diagnostic technologies. Our two-step analysis approach involving a rapid-turnaround ‘internal analysis’ of utilization trends before the main analysis allowed stakeholders to consider the feasibility and scope of the CER question as a major part of the prioritization process. An early look at imaging utilization patterns and variability of use in community practice was very helpful, if not essential, to assess study feasibility and inform estimates of economic impact – two important stakeholder criteria in assessing the overall importance of research questions. These data are seldom available from published literature or public data sets and the time and effort required to obtain institutional review board approval and data use agreements to collect this information is significant. Despite the limitations of the analysis, the data were extremely useful for steering study design and addressing feasibility issues.
The sequence of and preparation for stakeholder meetings allowed stakeholders to substantively engage in a wide range of technical questions. By first identifying a fairly broad list of possible topics, allowing adequate time for topic evaluation via a formalized framework with agreed-upon standardized criteria, and then meeting again for a final prioritization session, we found the stakeholders to be well informed and not adversely affected by gaps in knowledge or biased by their particular areas of expertise. As an example, study feasibility was found to be a very influential criterion for the stakeholder decision process. However, it would have been difficult to evaluate feasibility without the information that was collected from literature reviews and clinical experts about regional disease incidence, recommended imaging practices and reimbursement policies between stakeholder meetings.
A problem common to CER prioritization efforts has been the unclear path between completion of the prioritization process and the actual implementation of the stakeholder recommendations in conducting CER studies. An important issue in this is the lack of a well-defined mechanism linking prioritization results to CER funding. Past efforts to prioritize research using a stakeholder approach have usually been distinct from implementation efforts. Prioritization has generally been carried out by groups different from CER researchers who would implement the work, and the mechanism for funding the research is usually unspecified. For ADVICE, linking the results of the prioritization process to study implementation was immediate and implicit within the structure of the funded grant, which was somewhat unique to the ARRA applications. The project was funded from the outset as a unified process including both the stakeholder-guided prioritization and implementation stages. Such a design was possible via funding from the 2009 ARRA Stimulus Bill, but whether this model is acceptable to funding agencies in the future is unknown. Funding open-ended, stakeholder-guided research has not been the norm for agencies such as the AHRQ and the NIH . With the recent release of requests for proposals from the new Patient Centered Outcomes Research Institute (PCORI) , this trend may change in the future.
The ADVICE methodology for stakeholder engagement was not without challenges. Logistically, focus on a regional network of stakeholders made initial contact via telephone or email feasible and effective, yet scheduling compatible times for even regional stakeholders was not trivial. The limited time available for executive-level stakeholder members to participate in the project required the utmost attention to making each meeting, phone call and email communication as efficient as possible. Identifying appropriate stakeholders could also be an issue. ADVICE stakeholders were identified primarily through outreach to key opinion leaders in the community using the network of contacts known to the study team. This is a commonly used method for past research prioritization efforts , and may be particularly suited to identification of regional stakeholders, but this methodology may not be appropriate or effective for other research teams.
Two areas that were of high importance during the landscape analysis phase that were particularly difficult to address were the expected imaging utilization rates and direct costs of diagnostic tests. Few published or publicly available data sets were available, with the possible exception of Medicare data. In the future, establishment of data registries or distributed data networks, such as the Health Maintenance Organization Research Network Virtual Data Warehouse  and US FDA Sentinel project , may allow for rapid and practical identification of high imaging variability and areas of under- or over-utilization that warrant high-priority studies. Likewise, access to such databases for full CER studies could greatly reduce the time and effort required to obtain institutional review board approval and establish data use agreements and protocols, which otherwise are likely to be of a ‘one time use’ form.
Another challenge was how to adequately represent the interests of patient stakeholder groups in CER, particularly during the early phases of project prioritization and selection. Initially, we included a patient perspective by including a representative from the Puget Sound Health Alliance at the kick-off meeting, but this representative was the extent of patient involvement in our process to date. For the ADVICE project, it was not feasible to include patient stakeholders due to the inherent uncertainty around topic selection, and time and budgetary constraints. The range of potential CER topics at the outset of the project was very large, spanning all cancers and phases of diagnosis and treatment. Identification and selection of a representative patient group or groups that would have a broad and unbiased knowledge over the various conditions, imaging tests, outcomes and claims data may have been possible, but the investigators felt that it would be difficult to do this within the framework of the grant timeframe and budget. This is largely uncharted territory in CER research, and one of the priority areas of PCORI . It would have been more straightforward to involve patient groups from the start if the initial scope of potential research projects were somewhat narrower. For example, if the scope of research was limited to a topic within breast cancer imaging, it would have been natural to consider local breast cancer advocacy groups in the prioritization process. We intend to incorporate patient stakeholders in the dissemination phase of ADVICE, and in fact, this is the topic of an upcoming stakeholder meeting, but ideally, patients would be involved through all phases of the CER project.
We have described a regional stakeholder engagement process that has proven to be effective for topic prioritization and study implementation for cancer diagnostics CER. Continuous involvement of payer, clinical and health policy stakeholders from all phases of project development, including project prioritization, study design and contribution of patient-level data greatly enhanced stakeholder engagement and enthusiasm for the project. A focus on regional interests and use of regional data will likely lead to improved dissemination of research results to the clinical community and ultimately a change in clinical practice to make more effective use of diagnostic technology applied to cancer therapies.
The authors would like to thank the ADVancing Innovative Comparative Effectiveness Research in Cancer Diagnostics (ADVICE) stakeholder group (J Dominitz, Assistant Chief for Academic Af fairs, Gastroenterolog y, Veterans Administration of Puget Sound Health Care System; J Gifford, Senior Medical Officer, Regence BlueShield; R Herr, Medical Director, Regence BlueShield; L Hole-Curry, [former] Program Director, Health Technology Assessment, Washington State Health Care Authority; M Lonergan, Oncologist, Evergreen Hospital Medical Center; DK McCulloch, Medical Director, Clinical Improvement, Group Health Cooperative; M Stewart, Medical Director, Seattle Cancer Care Alliance ; J Thompson, Chief Medical Officer, Health and Recover Services Administration, DSHS; and JB Watkins, Pharmacy Manager, Premera Blue Cross) for their enthusiasm and extensive contributions to the ADVICE project. The authors would also like to thank their clinical imaging experts, C Lehman and D Mankoff for their extremely helpful contributions. Finally, they would like to thank A Drum for her assistance in document preparation and especially in her persistence in coordinating stakeholder meetings.
This work was supported by the Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics grant (1RC2CA148433-01) from the National Cancer Institute.
|Criteria for selection of study||Description of criteria|
|Included within statutory limits of Federal Coordinating Council’s definition of CER||Is the topic included within the definition of CER detailed in the mandate of the Recovery Act?|
|Stakeholder/community interest||Does the topic answer to expressed needs and preferences of patients, clinicians, and other stakeholders?|
|Feasibility||Are there available databases, registries, collaborations and resources to support study design?|
Are data from sufficient numbers of patients available for analysis?
Do the data span sufficient time to evaluate desired outcomes?
|Evidence needed||Is the modality currently being used without adequate evidence?|
|Standards for modality and implementation of test results||Are guidelines in place for the various uses of this modality? Are existing guidelines evidence-based and unbiased? Are they consistently implemented by providers?|
|Potential patient harm/safety concerns||What is the potential degree of harm (e.g., anxiety, radiation exposure, unnecessary treatment) an individual may experience from use of the modality? What are potential downstream harms from incorrect diagnosis using this modality?|
|Diagnostic accuracy and efficacy potential||How accurate is the modality for disease diagnosis? Does the modality offer potential for changing patient treatment path or outcome?|
|Economic impact||What is the estimated total direct cost per year (increase/decrease) of the modality and downstream treatments?|
|Severity of condition treated/diagnosed by modality||Are significant morbidity, mortality, and/or disability associated with disease?|
|Pressure on payers||Is there a high degree of pressure on payers to cover this modality, even though little evidence supports its use?|
|Potential or observed variation||Does payer/provider use of modality vary widely?|
|Diverse populations||Is there potential to evaluate CER in diverse populations and patient sub-groups affected by disease and/or modality?|
|Overlap/redundancy||Is/has similar research being/been conducted elsewhere?|
|Disease burden||How many persons are affected by disease per year? What is the disease cost to society?|
CER: Comparative effectiveness research.
|Criteria||Evidence for criteria||Criteria|
|Included within statutory limits of Federal Coordinating Council’s definition of CER||Yes||High|
|Feasibility||High incidence and mortality rate result in relatively large data sets available for analysis through insurance claims data linked to SEER. Initially, would be informative to look at claims data to get idea of variations in use of PET/CT for staging, restaging, and post-treatment surveillance of NSCLC patients in western Washington State||High|
|Evidence needed||Two RCTs performed outside the USA found fewer futile surgeries in the pre-operative PET/CT group, but no difference in survival. Studies looking at use of PET/CT in community (vs academic) setting needed||Medium|
|Standards for modality and implementation of test results||NCCN guidelines recommend that PET/CT be used for pretreatment evaluation of NSCLC, but PET/CT is still not regarded as a standard of care for NSCLC staging. Controversy exists regarding PET/CT as an add-on or replacement test||High|
|Potential patient harm/safety concerns||Unnecessary radiation exposure in patients who receive PET/CT scans; potential unnecessary surgeries due to PET/CT false positives, or delayed treatment for false negatives; also potential for unnecessary surgeries in patients who do not receive PET/CT scans||High|
|Diagnostic accuracy and efficacy potential||Considerable evidence of PET/CT offering superior accuracy in NSCLC staging. RCTs and retrospective studies generally agree that PET/CT offers potential to change patient management, but there is little evidence for survival improvements||High|
|Economic impact||According to SEER–Medicare data, approximately 12% of lung cancer patients filed claims for PET/CT between 2005 and 2007. Cost of PET/CT is approximately US$2500–4000, but might reduce costs by eliminating unnecessary surgeries and by allowing surgeons to better target areas of resection||High|
|Severity of condition treated/diagnosed by modality||High incidence and low survival rates; accounts for the most cancer-related deaths. US incidence rate: 69.0/100,000; 5-year survival: 16%; US death rate: 53.4/100,000||High|
|Pressure on payers||Anecdotal evidence suggests that use of PET/CT scans is increasing, with no evidence that the scans provide information that decreases morbidity, mortality, or recurrence rates||Medium|
|Potential or observed variation||No literature on this||No data|
|Diverse populations||Access to PET/CT machines is variable depending on patient location||Medium|
|Overlap/redundancy||Two RCTs looking at PET/CT for detecting recurrence in several cancers, including lung; none in western Washington State||Medium|
|Disease burden||Approximately 4000 incident cases in Washington State state per year; US prevalence: 370,617; aggregate 5-year Medicare cost (in 2004 dollars) of care for treatment of lung cancer was estimated to be US$4238 million||High|
CER: Comparative effectiveness research; CT: Computed tomography; NSCLC: Non-small-cell lung cancer; RCT: Randomized controlled trial; SEER: Surveillance, Epidemiology and End Results.
Financial & competing interests disclosure
The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.
Papers of special note have been highlighted as:
of considerable interest