Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
JAMA. Author manuscript; available in PMC 2010 July 27.
Published in final edited form as:
PMCID: PMC2910511

Closing the Evidence Gap in the Use of Emerging Testing Technologies in Clinical Practice

New testing technologies – increasingly based on genomic information – are essential in the shift towards personalized medicine and molecular targeted therapies. Considering the rapid proliferation of new tests, healthcare insurers and policymakers are interested in assessing evidence about their use and value.

It is critical to build an evidence base to support effective decision-making related to testing technologies as they are used in clinical practice. The example of human epidermal growth factor receptor 2 (HER2) testing for breast cancer illustrates the challenges and opportunities. Many groups, including the Institute of Medicine; Agency for Healthcare Research and Quality; Secretary’s Advisory Committee on Genetics, Health, and Society; President’s Council of Advisors on Science and Technology; and Evaluation of Genomic Applications in Practice and Prevention Project have cited the need to improve the evidence base for genomic and testing technologies. This Commentary extends previous work, emphasizing the need for evidence to assess how technologies are actually used in clinical practice.

Why HER2 Testing Illustrates the Evidence Gap

HER2 testing to target trastuzumab treatment for patients with breast cancer is perhaps the best known example of testing to target treatment. Gaps in knowledge about clinical practice for even this successful example portend an increasing need for evidence as new testing technologies enter clinical care. In addition, the high cost of many emerging therapies (e.g., trastuzumab therapy costs approximately $100,000/annually) also points to the need for evidence on how to most efficiently target therapies.

HER2 testing determines which patients overexpress the gene HER2; for those 20%–30% of patients, trastuzumab is highly effective. Trastuzumab and an accompanying test were approved in 1998 for metastatic breast cancer patients and its use was expanded to early-stage breast cancer patients after 2005. HER2 testing is now recommended for all patients with invasive breast cancer, and only patients with positive tests are recommended for trastuzumab treatment.

There is no consensus about optimal testing methods. Guidelines recommend using either immunohistochemistry (IHC); (with indeterminate results confirmed by fluorescence in situ hybridization FISH); or FISH to determine HER2 status.1 Although FISH is a better predictor of response to treatment, IHC costs substantially less and is more easily performed in community laboratories.1

Despite the clinical success of trastuzumab, there are concerns about the best methods for selecting patients for treatment. The accuracy and interpretation of HER2 tests have been highlighted in the media, with provocative headlines in some news stories. 2, 3 Even the company marketing trastuzumab publicly acknowledged the serious problem with test accuracy, noting that “about 5,000 patients in the US receive trastuzumab without any clinical benefit, and about 7,000 patients who could derive benefit are not being treated because of a false-negative test result.” (p. 168)4

The Evidence Gap

  1. Little is known about whether all eligible patients are tested for HER2 and, for those tested, which testing methods are used and whether indeterminate results are confirmed. Some eligible patients may not receive HER2 testing because of financial or clinical reasons, or their testing may not be documented in medical records or claims databases. Two studies suggest that some patients either are not tested or testing is not documented, and that there is little evidence to define how many patients receive confirmatory testing with FISH. A study conducted in one health care system shortly after FDA approval of trastuzumab for patients with metastatic breast cancer found that 52% of such patients were tested for HER2.5 In a study of a sample of Medicare enrollees (N=6588) – 32% of patients newly diagnosed with invasive breast cancer were documented in claims data as having undergone an HER2 test.6 Of those documented as receiving trastuzumab, 68% were documented as having had an HER2 test. In this study, 93% of women tested received only IHC, 0.3% received only FISH, and 6% received both tests – although it is not known whether FISH was used for initial or confirmatory testing.6
  2. Several studies show that a substantial percentage of HER2 tests performed by community laboratories are inaccurate. Less is known about the reasons and the implications for patients’ outcomes. One approach to measuring the accuracy of HER2 testing is to examine variability in laboratory procedures and results. The ASCO/CAP guidelines reviewed studies that have compared test results from community-based laboratories and high-volume reference laboratories, and found that approximately 20% of IHC test results from community-based laboratories are inaccurate.1
  3. Little is known about how many patients receive trastuzumab despite negative or indeterminate test results. Patients who are not tested or have negative test results may receive trastuzumab because of doubts about test accuracy, clinical or healthcare system factors, or human error. In one study that examined 2005 United HealthCare data7 involving patients with claims for trastuzumab, 8% had negative test results, and 4% had not been tested. For another 12% of patients, it was questionable whether HER2 testing was conducted because their physicians provided no documentation.
  4. Although studies have shown that trastuzumab treatment is relatively cost-effective, there are no analyses of the most cost-effective testing approaches in actual clinical practice. Cost-effectiveness of trastuzumab in the US – but not in relation to HER2 testing – show that adding trastuzumab to chemotherapy is relatively cost-effective.810 One study examined HER2 testing strategies before expansion of testing to patients with early breast cancer and before information was available on actual testing practices.11 These studies do not address the cost-effectiveness of different strategies for HER2 testing in practice. If patients who would benefit from treatment are not tested, if the most efficient testing algorithm is not used, if tests are inaccurate, or if patients receive treatment that is inappropriate based on their test results, the actual cost-effectiveness of testing and treatment will be reduced.

Improving the Evidence Base

Promising efforts are underway to address those gaps. Importantly, these findings do not challenge the efficacy of HER2 testing and trastuzumab treatment nor are they a reason to slow the pace of innovation or the diffusion of testing technologies into clinical practice. Rather, testing and treatment can be made more effective and efficient by gathering real-world data and incorporating these data into rigorous analyses of best-testing practices. The type and extent of evidence needed will vary by the specific situation as clinical trial data will never be able to address all of the questions. Approaches that facilitate both innovation and evidence must be developed, such as “coverage with evidence development” programs.

The example of HER2 highlights four potential solutions to the evidence gap:

  1. Document Gaps in Knowledge about Actual Clinical Practices. A first step towards solutions is to document what evidence is lacking. Clinicians and insurers were often surprised to hear about the evidence gaps about HER2 testing.
    Documenting evidence gaps will also facilitate a research agenda to examine those gaps. For example, cost-effectiveness analyses can now assess the most effective testing strategies.
    The National Cancer Institute (NCI) and the National Institutes of Health (NIH) have made great strides in supporting research to document and address evidence gaps on testing technologies. The NCI has recently funded grants on personalized medicine for cancer care that will address the evidence gaps for these technologies as well as develop methods for examining other emerging technologies.12 The NIH has issued a request for “transformative” applications to build an evidence base for pharmacogenomics and genetic testing, naming this area as one of six “major contemporary challenges”.
  2. Standardize Testing Procedures and Interpretation. Greater standardization of test procedures and use of decision-support algorithms could improve accuracy and provide more evidence about access and utilization. HER2 testing, like many diagnostic tests, does not provide clear answers about treatment decisions and there is ample room for misinterpretations along the entire testing sequence. Recently developed guidelines recommend that HER2 testing be done in a CAP-accredited laboratory to standardize procedures.1 Health plans are encouraging patients and clinicians to use laboratories that meet these guidelines and will provide coverage for repeat testing if necessary. Efforts are also underway to improve communication between laboratories and clinicians to address situations where indeterminate HER2 scores are treated as positive results. These standardization efforts for HER2 testing may also be important for other complex testing technologies, particularly given the increased scrutiny of in-house laboratory (“home brew”) tests.
  3. Provide Incentives for Closing Gaps. Policies can be implemented that provide incentives to reduce evidence gaps. One example is the policy change implemented by UnitedHealthcare in 2006, after it was found that many patients prescribed trastuzumab had missing or negative test results (personal communication, Lee Newcomer, UHC, 8/14/2008). Their policy requires clinicians to submit documentation of a positive HER2 test with the first trastuzumab claim. The rate of submitted claims for trastuzumab decreased after the policy implementation, suggesting that it may reduce inappropriate use. Because of this policy and other efforts to improve laboratory procedures, the plan has more information about why errors are made in HER2 testing and has been able to implement quality improvement programs. Although there are potential disadvantages to such policies, they may warrant consideration for some technologies.
  4. Develop Creative Approaches to Obtaining Evidence. Currently no databases link testing, test results, treatment, and outcomes and no system in the US monitors tests after their adoption.13 Interviews with several large healthcare insurers revealed that payers generally cannot assess whether eligible patients have received HER2 testing and cannot link test results with treatment.14 It is often impossible to identify the use of testing in administrative databases because of coding issues; for example, without chart review, the use of IHC and FISH for HER2 detection cannot be distinguished reliably from the same types of tests performed for other indications. Although Current Procedural Terminology code modifiers would differentiate specific tests, they are not commonly used in clinical practice. Moreover, test codes may be bundled into a common pathology code that does not permit identification of individual tests.

Creative collaboration between academia, industry, and government is needed to build the evidence base; for example, the Aetna Foundation has funded the University of California at San Francisco and Brigham and Women’s Hospital to determine how physicians use HER2 tests to inform their treatment recommendations. This academia-industry collaboration takes advantage of rich health plan enrollee data and academic research expertise.


Greater use of testing to target treatments is inevitable and has the potential to improve both the quality and efficiency of care but evidence-based information is essential if these new technologies are to be used wisely.15 A comprehensive agenda for translational research is needed to move new discoveries into clinical care. This agenda will require attention to both translation of basic research findings into new therapeutic options and translation of research into practice.


This study was funded by two grants to Dr. Phillips from the National Cancer Institute (R01CA101849 and P01CA130818) and a grant from the Blue Shield Foundation of California (unrestricted). The funding organizations and sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and approval of the manuscript. Dr. Phillips has an unrestricted grant from the Aetna Foundation to examine the utilization of HER2 testing and gene expression profiling for breast cancer.

I am grateful to the following individuals for their invaluable contributions to this manuscript: Su-Ying Liang PhD (UCSF); Deborah A. Marshall, PhD (University of Calgary); Jennifer S. Haas, MD, MSPH (Brigham and Women’s Hospital); Elena B. Elkin, PhD (Memorial-Sloan Kettering Cancer Center); Michael J. Hassett, MD, MPH (Dana-Farber Cancer Institute); Ilia Ferrusi, B.Sc (McMaster University); Jane E. Brock, MBBS, PhD (Brigham and Women’s Hospital); and Stephanie L Van Bebber, M.Sc (UCSF). I also thank i3 innovus and Julia Trosman and the Center for Business Models in Healthcare for their commissioned reports developed for the Department of Clinical Pharmacy at the University of California, San Francisco. Lastly, I thank Mr. Kuo Tong, president and founder of Quorum Consulting in San Francisco, California, for the use of results from his poster presentation given at the 2007 Breast Cancer Symposium.


1. Wolff AC, Hammond ME, Schwartz JN, et al. American Society of Clinical Oncology/College of American Pathologists guideline recommendations for human epidermal growth factor receptor 2 testing in breast cancer. J Clin Oncol. 2007;25(1):118–145. [PubMed]
2. Mathews AW. Bad Cancer Tests Drawing Scrutiny. Wall Street Journal. 2008 Jan 4;:B1.
3. Pollack A. Cancer drug may elude many women who need it. The New York Times. 2007 June 12; [Accessed on June 4, 2008]; Available at
4. Nass SJ, Moses HL, editors. Cancer Biomarkers: The Promises and Challenges of Improving Detection and Treatment. Washington D.C.: The National Academies Press; 2007. Mar 27, Diagnosis, and Treatment Committee on Developing Biomarker-Based Tools for Cancer Screening; p. 252. Contract Nos. HHSH25056133, HHSN261200611002C, 200-2005-13434, HHSM-500-2005-00179P, HHSP23320042509XI, and 223-01-2460. Supported by the National Academy of Sciences and the Department of Health and Human Services; 2007.
5. Stark A, Kucera G, Lu M, Claud S, Griggs J. Influence of health insurance status on inclusion of HER-2/neu testing in the diagnostic workup of breast cancer patients. Int J Qual Health Care. 2004;16(6):517–521. [PubMed]
6. Tong KB, Chen E, Gregory C, Kim D. HER-2 Testing and Trastuzumab Use in the Medicare Population [abstract #141]. Paper presented at 2007 Breast Cancer Symposium; September 7–8, 2007; San Francisco, CA.
7. Culliton BJ. Insurers And 'Targeted Biologics' For Cancer: A Conversation With Lee N. Newcomer. Health Aff. 2008;27(1):w41–w51. [PubMed]
8. Garrison LP, Perez EA, Dueck A, Lalla D, Paton V, Lubeck D. Cost-effectiveness analysis of trastuzumab in the adjuvant setting for treatment of HER2+ breast cancer. Journal of Clinical Oncology. 2006;24(18S)
9. Liberato NL, Marchetti M, Barosi G. Cost effectiveness of adjuvant trastuzumab in human epidermal growth factor receptor 2-positive breast cancer. J Clin Oncol. 2007;25(6):625–633. [PubMed]
10. Kurian AW, Thompson RN, Gaw AF, Arai S, Ortiz R, Garber AM. A cost-effectiveness analysis of adjuvant trastuzumab regimens in early HER2/neu-positive breast cancer. J Clin Oncol. 2007;25(6):634–641. [PubMed]
11. Elkin EB, Weinstein MC, Winer EP, Kuntz KM, Schnitt SJ, Weeks JC. HER-2 testing and trastuzumab therapy for metastatic breast cancer: a cost-effectiveness analysis. J Clin Oncol. 2004;22(5):854–863. [PubMed]
12. [accessed 10/7/08]. 1P01CA130818-01A1
13. DeStefano F, Whitehead N, Lux L, Lohr K. Rockville, MD: Agency for Healthcare Research and Quality; 2008. May, Infrastructure To Monitor Utilization and Outcomes of Gene-Based Applications: An Assessment. (Prepared by RTI International DEcIDE Center under Contract No. HSA290220050036I.) AHRQ Publication No. 08-EHC012.
14. Trosman J. Center for Business Models in Healthcare. Payer HER2 Project: Findings Report. 2008. May, Unpublished report prepared for UCSF.
15. Khoury MJ, Gwinn M, Yoon PW, Dowling N, Moore CA, Bradley L. The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genet Med. 2007;9(10):665–674. [PubMed]