|Home | About | Journals | Submit | Contact Us | Français|
We evaluated clinical practice guideline (cpg) recommendations from Cancer Care Ontario’s Program in Evidence-Based Care (pebc) for molecularly targeted systemic treatments (tts) and subsequent funding decisions from the Ontario Ministry of Health and Long-Term Care.
We identified pebc cpgs on tt published before June 1, 2010, and extracted information regarding the key evidence cited in support of cpg recommendations and the effect size associated with each tt. Those variables were compared with mohltc funding decisions as of June 2011.
From 23 guidelines related to 17 tts, we identified 43 recommendations, among which 38 (88%) endorsed tt use. Among all the recommendations, 38 (88%) were based on published key evidence, with 82% (31 of 38) being supported by meta-analyses or phase iii trials. For the 38 recommendations endorsing tts, funding was approved in 28 (74%; odds ratio related to cpg recommendation: 29.9; p = 0.003). We were unable to demonstrate that recommendations associated with statistically significant improvements in overall survival [os: 14 of 16 (88%) vs. 8 of 14 (57%); p = 0.10] or disease- (dfs) or progression-free survival [pfs: 16 of 21 (76%) vs. 3 of 5 (60%); p = 0.59] were more likely to be funded than those with no significant difference. Moreover, we did not observe significant associations between funding approvals and absolute improvements of 3 months or more in os [6 of 6 (100%) vs. 3 of 6 (50%), p = 0.18] or pfs [6 of 8 (75%) vs. 10 of 12 (83%), p = 1.00].
For use of tts, most recommendations in pebc cpgs are based on meta-analyses or phase iii data, and funding decisions were strongly associated with those recommendations. Our data suggest a trend toward increased rates of funding for therapies with statistically significant improvements in os.
Many new anticancer treatments target molecular aspects of the particular tumour. Because these therapies have increased treatment complexity and cost, there is increasing interest in ensuring that patients receive appropriate, high-quality, evidence-based care. Synthesis of evidence into clinical practice guidelines (cpgs) has been a key tool in developing treatment policies and informing drug-funding decisions in Canada, which has a publicly-funded health care delivery system1–3. When making funding decisions that facilitate access to new treatments, several jurisdictions use processes that include systematic evaluations of clinical evidence4–6. Because molecularly targeted systemic treatments (tts) account for most of the increase in anticancer therapy costs7, we evaluated cpgs and drug-funding decisions for the related therapies in Ontario, Canada. Our objective was to assess the factors in evidence-based cpg recommendations that influenced subsequent funding decisions.
In Canada, most new anticancer drugs are funded by universal insurance provided by provincial governments. Once approved, residents make no direct payment for those agents, or for other institution-based services such as hospitalization, surgery, radiation therapy, or intravenous chemotherapy.
Ontario is Canada’s largest province, with a population of approximately 13.4 million8. In 1995, Cancer Care Ontario (cco), the province’s cancer agency, established a practice guidelines initiative that evolved into the Program in Evidence-Based Care (pebc). The pebc uses a guideline development cycle that includes a systematic process for extracting and analyzing clinical trial data to generate evidence-based recommendations2. The pebc convenes disease site groups for each cancer type, which, with centralized support, are responsible for evaluating evidence and forming cpg recommendations. Those recommendations, and other information such as pharmacoeconomic data, are submitted to drug-approval policy bodies overseen by Ontario’s Ministry of Health and Long-Term Care (mohltc), where final policy decisions are made.
In addition to guideline documents, cco’s pebc also prepares related series of evidence-based reports called “special reports,” based on direct requests from the committee that advises the mohltc on drug-funding decisions. Although the special reports are systematically developed evidence-based statements, they have not completed the full guideline development cycle. For the purposes of the present study, we included both guideline document forms (that is, evidence-based guidelines and special reports), referring to them collectively as cpgs.
A single author (RR) reviewed the pebc Web site at cco (http://www.cancercare.on.ca) and an internal list provided the pebc to identify all pebc guidelines and special reports as of June 1, 2010. All cpgs related to systemic therapy were reviewed in duplicate (by RR and CMB) to identify cpgs for use of tts (defined as nonhormonal agents that interfere with specific molecules involved with tumour growth and progression9) and to extract data, including tumour type, extent of disease, line of therapy, and treatment recommendations. Treatment intent was classified as curative or noncurative based on input from all study authors.
“Key evidence” is a term used by the pebc to refer to the clinical trials data cited in the cpg short report that are most strongly associated with each recommendation. When more than one piece of key evidence was cited, a hierarchy (Figure 1) was used to identify the single data source prioritized by the cpg authors. The present study prioritized published articles over abstracts and meta-analyses or phase iii trials over phase ii trials. If multiple sources of key evidence remained after those criteria had been applied, the study with the largest sample size was used. For each piece of key evidence, we captured publication type, phase of the study, sample size, effect size for overall survival (os) and disease- (dfs) or progression-free survival (pfs), and level of statistical significance associated with the foregoing differences.
Hierarchical identification of key evidence associated with Cancer Care Ontario Program in Evidence-Based Care clinical practice guidelines (cpgs). ma = meta-analysis; Ph3 = phaseiii.
In Ontario, funding for intravenous and oral chemotherapy is provided by the mohltc, which is advised by an independent Committee to Evaluate Drugs (ced). In 2005, an expert subcommittee was formed that included members of the ced and of cco. The subcommittee considers cpgs developed by the pebc, together with pharmacoeconomic and other relevant information, and then provides a recommendation to the ced to fund or not fund an agent for a specific indication. The ced reviews those recommendations and, in turn, provides its recommendation to the mohltc. Final mohltc funding decisions are available on the Web at the ministry site (http://www.health.gov.on.ca/english/providers/program/drugs/ced_rec_table.html) or at the site describing the cco New Drug Funding Program (http://www.cancercare.on.ca/toolbox/drugs/ndfp). We reviewed both Web sites to obtain funding decisions, and we contacted cco and mohltc when additional information was needed. The present study includes all drug-funding decisions reported as of June 1, 2011.
The United Kingdom also has a publicly funded health care system, and its National Institute for Health and Clinical Excellence (nice) conducts appraisals and creates cpgs on selected topics using a mechanism that has some similarities to that used by the pebc. For comparative purposes, we also report the tts that were funded by nice at the same cut-off time (So J, National Health Service Christie Trust. Personal communication).
Descriptive statistics are used to summarize data. Hazard ratios for os, dfs, and pfs (alone or in combination) are reported to describe the effect size for each piece of key evidence. Because not all key evidence reported hazard ratios, we used point estimates of survival distribution or median survival (or both) to derive a hazard ratio by applying a parametric model with assumptions of exponential distribution10. To determine the magnitude of the effect of the experimental therapy compared with the control arm in absolute terms, key evidence trials were classified as showing improvement in os, dfs, or pfs of at least 3 months or less than 3 months. Proportions between groups were compared using the Fisher exact test11; differences were considered statistically significant at p < 0.05 (two-sided).
We identified 221 pebc cpgs (Figure 2), including 115 that evaluated systemic therapies, of which 29 evaluated tts. Of the latter 29 cpgs, 6 were excluded either as duplicates because of updating (n = 2) or because they lacked a tt recommendation (n = 4). The final study sample therefore included 23 tt cpgs.
Identification of targeted therapy (tt) clinical practice guidelines (cpgs) published by Cancer Care Ontario’s (cco’s) Program in Evidence-Based Care (pebc) to June 2011.
The 23 cpgs in the sample covered 10 tumour sites, with hematologic malignancies accounting for 43% (n = 10). Most cpgs (n = 19, 83%) were published during 2006–2009. The tts evaluated included bevacizumab (n = 4), imatinib (n = 3), trastuzumab (n = 3), alemtuzumab (n = 2), cetuximab (n = 2), dasatinib (n = 2), erlotinib (n = 2), gefitinib (n = 2), rituximab (n = 2), sorafenib (n = 2), sunitinib (n = 2), and bortezomib, everolimus, ibritumomab, panitumumab, temsirolimus, and tositumomab (n = 1 each).
All 23 cpgs identified relevant studies by searching medline and the conference proceedings of the American Society of Clinical Oncology. Other disease-specific conference proceedings were searched in 13 cpgs. The guidelines included an average of 14.6 studies (median: 9 studies) related to the specific tt. Evidence informing the recommendations included phase iii randomized trials (n = 23, 100%), published meta-analyses (n = 1, 4%), other practice guidelines (n = 4, 17%), single-arm phase ii trials (n = 10, 43%), and retrospective studies (n = 1, 4%). Data published in abstract form was used in 20 cpgs (87%).
The 23 cpgs led to 43 treatment recommendations (mean: 1.8 recommendations; median: 2 recommendations; range: 1–6 recommendations per cpg). Table i shows the characteristics of the key evidence used to determine the treatment recommendations. In 38 recommendations (88%), use of the tt was supported, and 28 recommendations (65%) cited more than one source of evidence. The mean sample size for key evidence studies was 634 patients (range: 11–5081 patients; median: 462 patients). In 38 cases (88%), the basis for the recommendation was key evidence from published articles, with the articles in 31 of those cases (82%) being reports of phase iii trials or meta-analyses; 7 recommendations (16%) were based on phase ii studies. Among the 38 recommendations endorsing use of a tt, funding was approved for 28 (74%).
Characteristics of clinical practice guideline recommendations and associated key evidence for targeted anticancer therapy from Cancer Care Ontario’s Program in Evidence-Based Care (n = 43)
Table ii shows the characteristics of the key evidence for recommendations that were and were not funded. Comparative measures of os were available in 70% of cases (n = 30) and dfs or pfs in 60% (n = 26), with a statistically significant difference being detected between treatment arms in 53% (n = 16) and 81% (n = 21) of cases respectively. Absolute differences between treatment arms were reported in 12 (40%, os) and 20 (77%, dfs/pfs) cases. An improvement of at least 3 months was observed for 6 (50%) and 8 (40%) of those cases respectively.
Factors associated with drug funding status for 43 recommendations for targeted therapy from Cancer Care Ontario’s Program in Evidence-Based Care
By univariate analysis, a cpg recommendation endorsing treatment was the only variable associated with funding approval. We did not detect differences in funding decisions based on recommendations associated with detection of a statistically significant improvement (compared with a nonsignificant improvement) in os (14 of 16, 88% funded vs. 8 of 14, 57% funded; p = 0.10) or in dfs or pfs (16 of 21, 76% funded vs. 3 of 5, 60% funded; p = 0.59). Likewise, we did not observe an association between funding decisions and reported absolute improvements of at least 3 months (compared with less than 3 months) in median os (6 of 6, 100% funded vs. 3 of 6, 50% funded; p = 0.18) or in dfs or pfs (6 of 8, 75% funded vs. 10 of 12, 83% funded; p = 1.00). Table iii classifies treatment recommendations and corresponding effect sizes by funding status and treatment intent.
Comparison of effect size for targeted drugs identified in 43 recommendations from Cancer Care Ontario’s (cco’s) Program in Evidence-Based Care
Despite access to the same data sources on the part of the pebc and the nice, a comparison of their recommendations showed that funding approval status was discordant between Ontario and the United Kingdom for 14 recommendations (33%, Table iv). Among the 28 therapies funded in Ontario, 15 (54%) were funded in the United Kingdom; conversely, 1 therapy not funded in Ontario was approved in the United Kingdom (gefitinib for the first-line treatment of advanced non-small-cell lung cancer—a treatment that was approved in Ontario after the June 2011 cut-off in the present study). The remaining 13 therapies are funded in Ontario but not in the United Kingdom. The only factor that predicted concordance in the funding status between Ontario and the United Kingdom was a dfs or pfs effect size that exceeded 3 months or that was statistically significant.
Comparison of anticancer drug funding decisions for 43 clinical practice guideline recommendations in Ontario and through the U.K. National Institute for Health and Clinical Excellence (nice)a
We reviewed 43 recommendations associated with 23 cpgs evaluating 17 tts, finding that 38 recommendations (88%) supported use of the tt and that Ontario government funding was approved for 28 of them (74%). Funding approval decisions were strongly associated with cpg recommendations.
Several other important findings emerged. First, most cpg recommendations (79%) are supported by data from phase iii trials. Second, among cpg recommendations with key evidence describing comparative measures of os and dfs or pfs, the survival differences were statistically significant in 53% (16 of 30) and 81% (21 of 26) of cases respectively. Third, for more than 50% of the foregoing cases, the reported absolute difference between treatment arms was less than 3 months. Finally, although absolute effect size was not found to be associated with funding-approval decisions, we did observe a trend in association between drug funding and a statistically significant improvement in os.
In Ontario, 9 cpg treatment recommendations did not receive funding approval. Of those 9 recommendations, only 2 were related to a treatment plan associated with potentially curative intent (alemtuzumab for T-cell leukemia); both recommendations were based on single-arm phase ii studies. Among the 7 recommendations associated with noncurative intent, only 1 was associated with a statistically significant benefit in os (bevacizumab for advanced non-small-cell lung cancer); in 2 other cases, there were statistically significant improvements in pfs or time to progression (second-line trastuzumab for advanced breast cancer and tositumomab for non-Hodgkin lymphoma). The magnitude of benefit associated with the key evidence for each of those indications was modest, with median differences between the treatment arms of less than 3 months.
Two earlier studies have evaluated approval processes for new anticancer therapies in the United Kingdom and Ontario. In their review of decision-making at Christie Hospital NHS Trust (United Kingdom), Foy et al.54 described funding that was based on thresholds related to effectiveness. In a related study, Martin and colleagues55 reported that priority-setting decisions in Ontario were based largely on clinical benefit and that rationales could change with changing costs and budgets. The present comparison of funding approval status for anticancer drugs in Ontario and the United Kingdom shows that tts were more often approved in Ontario. Given that recommendations from nice may place greater weight on formal economic evaluations56 and cost effectiveness, it is possible that differences in funding decisions were related to different thresholds associated with economic evaluations. Among the 13 recommendations funded in Ontario but not approved for funding by nice (Table iv), 2 were associated with curative intent (imatinib for Philadelphia chromosome–positive acute lymphoblastic leukemia), and 1 of those 2 was based on phase iii data demonstrating a significant improvement in os. The other 11 recommendations were associated with noncurative intent. In 3 cases, statistically significant improvements in os had been reported (temsirolimus and sorafenib for renal cell cancer and cetuximab for metastatic headand-neck cancer). Of the remaining 8 recommendations, 6 were associated with statistically significant improvements in pfs or time to progression. Among the 9 therapies associated with statistically significant improvements in os, pfs, or time to progression, the magnitude of the benefit was greater than 3 months in only 2 cases (temsirolimus for renal cell cancer and bortezomib for myeloma).
While not a study specific to oncology, work by Clement et al. recently evaluated relationships between evidence, cpgs, and drug funding decisions in three jurisdictions: the United Kingdom (nice), the Australia (Australian Pharmaceutical Benefit Advisory Committee), and Canada (Canadian Common Drug Review)57. Between 2000 and 2008, nice recommended listing for 87% of submissions compared with 50% for the Common Drug Review and 54% for the Pharmaceutical Benefit Advisory Committee. In a related study of nice, Mason and Drummond58 reported that, among 55 cancer therapies assessed between 2000 and 2008, 53% were ether rejected for routine use in the U.K. National Health Service (5 of 38, 13%) or restricted to a narrow set of indications (15 of 38, 39%). Those authors also observed a trend toward more negative decisions in recent years. In another report written with colleagues59, the same authors compared cancer drug therapy approval decisions in the United States and the United Kingdom and concluded that drug coverage decisions that include processes to consider cost-effectiveness (such as those made by nice) are associated with greater restrictions and slower times to coverage.
The present study is the first comprehensive evaluation of the relationship between evidence, cpgs, and funding approval decisions for anticancer drugs, but the results should be interpreted in the context of study limitations. Drug-funding approval processes in Ontario have recently changed. As of October 2011, a national advisory committee [the pan-Canadian Oncology Drug Review (http://www.pcodr.ca)] has begun issuing funding recommendations to participating provincial ministries of health and the existing Ontario subcommittee of the ced. Accordingly, our conclusions may not be generalizable to future funding approval processes in Ontario or to guideline programs or funding agencies in other jurisdictions. Our study may also include more specific limitations based on our hierarchical framework for identifying key evidence (including prioritizing articles over abstracts) and a relatively small sample size of tt recommendations that likely precluded an ability to identify statistically significant findings. In only a single case (sorafenib for hepatocellular carcinoma) was a published phase ii study prioritized over a conference abstract of a randomized controlled trial. Furthermore, our analyses do not take into account cases in which more than one published phase iii randomized controlled trial might support a tt. To be able to explore how effect size is related to funding decisions, we needed to identify treatment effects. We chose a 3-month threshold for that analysis because we felt that most patients and clinicians would agree that 3 months is clinically significant, and others authors have suggested the same magnitude of effect. We compared drug funding statuses between Ontario and nice in the United Kingdom. However, in the United Kingdom, local Cancer Drug Funds can provide funding for cancer therapies that are not approved for funding by nice. Accordingly, it is possible that, at the local level in the United Kingdom, there is less discordance in funding status for cancer therapies than our results would suggest. Finally, we did not independently evaluate pharmacoeconomic aspects of treatment, and we suspect that, given the substantial costs of many new drugs7, pharmacoeconomic analyses, including those provided in reports by nice and those performed but not widely reported in Ontario evaluations, may account for the differences observed.
We reviewed 23 cpgs produced by cco’s pebc that relate to 17 targeted cancer therapies. Funding decisions were strongly associated with the evidence-based cpg recommendations. Treatments that were endorsed by cpgs, but not approved for funding in Ontario were associated with absolute median differences in time-dependent endpoints of less than 3 months. Further work is required to better understand how effect size and pharmacoeconomic factors relate to cpg recommendations and drug-funding and policy decisions.
CMB is supported as a cco Chair in Health Services Research. The ncic Clinical Trials Group, including the salaries of RMM and BEC, are supported by the Canadian Cancer Society Research Institute with funds provided by the Canadian Cancer Society. The authors gratefully acknowledge assistance provided by Scott Gavura and June So during the course of this study.
CMB is supported as a cco Chair in Health Services Research. MB is director of cco’s pebc. RMM has served as a consultant reviewer for pebc guidelines.