|Home | About | Journals | Submit | Contact Us | Français|
New treatments tested in randomized trials are as likely to be inferior as they are to be superior to standard treatments.1 The ethical implication of this empirical observation is that unevaluated new treatments should be assessed carefully in well-designed research before decisions are taken about whether or not they should become widely available. Indeed, the same principle applies when there are uncertainties about the effects of ‘standard’ treatments. The dangers of acquiescing when it is uncertain whether treatments do more good than harm has been illustrated dramatically by the results of the CRASH trial:2 tens of thousands of iatrogenic deaths have occurred because of decades of failure to address uncertainties about the effects of systemic steroids for patients with acute traumatic brain injury
It is against this background that some clinicians believe that unproven treatments should only be offered within the context of appropriately designed research. In the perinatal field, for example, obstetricians decided that chorionic villus sampling should be used in low-risk pregnancies only within the context of a randomized comparison with amniocentesis.3 Similarly, neonatologists restricted the use of neonatal extracorporeal membrane oxygenation (ECMO) to a randomized comparison with standard respiratory support,4,5 and they are currently restricting the use of prolonged whole-body cooling in term infants with perinatal asphyxial encephalopathy to a randomized comparison with standard treatment (the TOBY trial: Whole Body Hypothermia for the Treatment of Perinatal Asphyxial Encephalopathy)
Ethical considerations have been the main motivation for clinicians who have dealt with uncertainties about treatments in this way. But ethical considerations have also informed similar decisions made by agencies managing health service funds on behalf of defined populations to whom they are answerable for cost-effective use of resources.6 For example, the Centre for Medicare and Medicaid Services (CMS) in the USA decided that, until more was known about its relative merits and demerits, lung reduction surgery should only be supported if given within the context of a randomized comparison with non-surgical treatments.7 The Centre is currently formalizing the criteria for applying this principle,8 which has also been considered9 and implemented on a single occasion so far (personal communication, Professor Wing) by the Pharmaceuticals Benefit Scheme in Australia.10
The principle of recommending that when there are important uncertainties about a treatment's effects, that treatment should only be used in the context of research, was explicitly recognized by the National Institute for Health and Clinical Excellence (NICE) at its inception in 1999. NICE was set up to develop evidence-based guidance for the NHS and the broader public sector with the explicit objectives of promoting cost-effective use of resources while encouraging innovation, improving quality and reducing variation in practice.11 To do so, NICE and its advisory bodies were given three decision options, set out by the Department of Health shortly before NICE's establishment in 1999.12 When appraising health interventions (including health technologies, health promotion and disease prevention programmes, screening and diagnostic tests, surgical procedures, etc.), NICE would be able to recommend:
This paper discusses NICE's third option, its characteristics, use and implications for the evidence base, and its future prospects as a viable policy option for NICE and the NHS. This seems to us a good time to promote discussion on the issue, given NICE's recent launch of the single technology appraisal to assess new technologies closer to licensing, with clear implications on the available evidence base, and the reorganization of the medical research infrastructure in the UK following the Cooksey review. To make the most of the ‘Only In Research’ (OIR) option for patients and the Health Service, this discussion needs to start as early as possible
The evidence base on specific interventions or management strategies is rarely complete, so NICE and its advisory bodies are almost always faced with uncertainty when making their decisions. Despite this, NICE has to develop guidance on the clinical and cost-effectiveness of the interventions referred to it by the Department of Health, and it has to do so within specific timelines. Starting by explicitly acknowledging its existence, NICE has adopted a number of approaches to dealing with uncertainty. Throughout the topic selection process, NICE has always made a case against developing guidance in areas where the evidence is scarce;13 instead, these topics would be forwarded to NHS Research and Development Programmes. Furthermore, in 2003, NICE's Board approved the establishment of an R&D Programme supported by the NICE R&D Advisory Committee, with the explicit objective of improving the quality of the evidence base underpinning its guidance through promoting appropriate research.14 Over the years, NICE has encouraged the development of methodological tools to deal with uncertainty and has since adopted a number of them, including probabilistic sensitivity analysis,15 now explicitly referred to in the NICE Technology Appraisal Methods Manual.16 Consensus methods and elicited expert opinion are often used to supplement systematic reviews when developing clinical guidelines in areas where there is little good quality evidence,17 based on NICE's methods manuals.18 Finally, when important evidence gaps are identified during the guidance development process, NICE issues research recommendations, encouraging further research to help fill those gaps and inform future updates of its guidance. Calling for further research to inform clinical practice is an important NICE function, highlighted in both the NICE Framework document19 and the WHO Review of Technology Appraisals.20
Nevertheless, there will always be cases where the available evidence is insufficient to support a positive—or negative—recommendation. In these cases, NICE advisory bodies can consider issuing a recommendation for the use of the intervention only in the context of research.
Sixteen (approximately 4%) of NICE's technology appraisal recommendations published between 1999 and early 2007 advised the use of a technology only in the context of research (Table 1). For example, in 2000, NICE recommended that laparoscopic resection for colorectal cancer should only be undertaken in the context of a clinical trial.21 This decision encouraged recruitment to the then ongoing UK-based Medical Research Council CLASICC trial, which has since provided the necessary evidence to support a NICE recommendation that this intervention be introduced in routine NHS practice.22 NICE's advice on the use of liquid based cytology is another situation in which NICE recommended further research before subsequently issuing a positive recommendation for the adoption of this new technology across the NHS.23 Five of the technology appraisal OIR recommendations became supportive recommendations when the original guidance was reviewed in light of additional research evidence (Table 1).
This type of recommendation is now found not only in appraisals of specific technologies but also in clinical guidelines, which deal with broader disease management strategies, and (since April 2005) in public health guidance, which looks at specific interventions or broader programmes of prevention and health promotion. Indeed, in one of the first pieces of public health guidance published by NICE, three of the four interventions considered were recommended only in the context of research studies24 (Table 2). Almost half of NICE's recommendations in its interventional procedures guidance (which advises on the safety and efficacy of surgical procedures) fall in the category of ‘cautionary guidance’ and are recommended only with additional requirements for consent, audit and, occasionally, for more formal research.25
Finally, NICE often recommends that interventions are used only if outcome data are assembled in appropriate registers or ‘databases’ to assess their safety as well as their clinical and cost-effectiveness (Table 3).
Implementing the OIR option in the NICE and broader NHS context has been challenging.
NICE operates within an environment of inevitable uncertainty.26 Over-exercising the OIR option in this setting could result in indecisiveness and inaction, hamper innovation, and negate NICE's very objective: to provide advice based on the best available evidence.
To ensure the OIR option is used in a pragmatic way, NICE is exploring the decision-making trail leading to and the criteria underpinning these decisions. These criteria should ideally reflect how well NICE and the NHS tolerate uncertainty. A number of questions need addressing:
However challenging these questions may be, a methodological framework is already in place to help support the thinking in this area. Value of information principles already referred to in the NICE appraisals manual, supplemented by options theory,27 could provide the methodological underpinnings for tackling some of these questions. At the same time, a qualitative checklist could also be developed to help guide the deliberations of the advisory bodies. For example, when further research is necessary and could help resolve uncertainty, advisory committee members could be asked to consider what the implications of making a positive recommendation would be:
No formal arrangements exist between NICE, industry and the clinical research community, including the NHS and patient organizations, for pursuing the research recommended by NICE in a timely manner, so that uncertainties are reduced to the extent that definitive guidance can be developed. This contrasts with the active arrangements for promoting the uptake of positive NICE recommendations. As a result, OIR recommendations are hard to implement, particularly when there is no ongoing trial addressing the question. Indeed, they are often perceived to be negative recommendations by patients, clinicians and industry alike. In a recent BMJ paper reviewing NICE recommendations between 1999 and 2005, Raftery classifies all recommendations for the use of a technology in the context of research as ‘no’ decisions.28
The development of a co-ordinated approach with other interested parties could encourage NICE advisory bodies to consider OIR recommendations in a more systematic and consistent way. For example, in guidance on the use of beta interferon and glatiramer in the treatment of multiple sclerosis NICE recommended that the Department of Health and the National Assembly for Wales, along with the manufacturers, consider what action could be taken so that the NHS could obtain these drugs in a way that would be cost-effective.29 A national data collecting system was established to ascertain which sub-groups of patients are most likely to benefit. However, the lack of a coordinated approach has made it difficult to ensure that the information collected will ever inform an update of the NICE guidance
Furthermore, as the burden of evaluation lies with NICE, mandating research from the technology sponsors is unrealistic and wishful thinking, particularly in cases where head-to-head trials are needed. The prospect of an appeal, and the pressure generated by lobbying groups keen to ensure access to the new technology, make the OIR option even less attractive to NICE advisory committees.
As a result, the danger is that OIR recommendations turn into a means of saying no politely, instead of generating the additional evidence needed to make more confident decisions.
Indeed, the recently published Cooksey Review30 recommends that special arrangements are set up and funding earmarked to ‘... implement NICE recommendations calling on the NHS to use health interventions in a research context’. This is an exciting development, directly addressing the need for co-ordination between publicly funded research and evidence-based policy making, as reflected in the NICE process of appraising new technologies. Key stakeholders, including NICE, research funders, patient organizations, industry, government and the NHS, will have to work together to implement Cooksey's recommendations. In the meantime, some have hastened to interpret the Cooksey proposals as a call for conditional coverage of technologies, across the NHS, long before phase III trials are completed.31 However important it is to accelerate access to new technologies, without the right processes in place, such moves could compromise patient safety and overall health outcomes in a financially restrained system.
Furthermore, the OIR option is now applied not only to new medical technologies with interested sponsors as originally anticipated,12 but also to established clinical and public health strategies that almost always lack sponsorship. This has reinforced the implementation challenges described above but has also highlighted how important it is for NICE advisory bodies to be able to recommend data collection on existing, often well-established practices or interventions to inform recommendations for increased investment, or for disinvestment. The need to support the NHS by advising which ineffective, wasteful or even harmful practices should be discontinued2 makes this evaluative dimension of the OIR option particularly relevant.
Linking research funding to NICE decision making is a pressing need; the recent single technology appraisal of Herceptin for early breast cancer is a case in point: in August 2006, only three months after the drug gained its license, NICE recommended the use of the drug for eligible women. The decision was supported by the available evidence at the time32 and recently published studies further reinforce this decision.33 However, a positive decision based on the best available evidence may not be enough. NICE-mandated, publicly-funded research into alternative therapeutic regimes beyond the single licensed indication determined by the manufacturer could inform future updates of the guidance and ensure the Health Service is obtaining value.34 Carrying out health technology assessment earlier on in the development of a technology in the context of the single technology appraisal poses a significant challenge if the evidential standards are to be maintained. Alternative approaches to the yes/no dichotomy, which encourage the development of the evidence base, will need to be applied to avoid widespread use of unproven technologies or denying access to promising treatments (Table 4).
NICE set up the Citizens' Council in 2002 (http://www.nice.org.uk/citizenscouncil) to advise NICE and its advisory bodies on the social value judgments that should underpin its guidance. Its 30 members reflect the demography of England and Wales and meet twice a year to debate issues put to them by NICE. Their reports inform the Social Value Judgments principles which constitute official guidance to NICE's advisory bodies.
In January 2007 the Council met to discuss NICE's approach to uncertainty. In their draft report (currently under public consultation) they unanimously endorse OIR as a valuable decision option for NICE and its advisory bodies, and go on to discuss the circumstances under which NICE should consider issuing such a recommendation. A number of key considerations are identified:
Having discussed the need for explicit criteria for OIR decisions and the difficulties of implementing the necessary research, the Council report concludes ‘... patients would be reassured to know that clinicians and the healthcare system in general could face up to uncertainty, and were confident enough to deal with it in a mature, scientific way, and avoid wasting money on unproven technologies.’35
OIR recommendations have been statutory valid alternatives to yes and no decisions since NICE was set up by government in 1999. They represent the only rational way for addressing uncertainty, consistent with NICE's principles of transparency and methodological robustness. As demonstrated in a number of cases so far, OIR recommendations can be viable and workable decision options as long as there is some co-ordination between the research and decision-making communities.
Our study brings together all occurrences of such NICE decisions and describes how they have evolved over the years in response to the generation or absence of new evidence. We have not discussed here those cases where an OIR decision should have been made by NICE but was not, neither have we sought to identify any OIR that were made inappropriately given the circumstances. To do either of the above we would need to establish clear decision criteria as to when OIR recommendations are justified. To achieve this, additional work is underway to:
NICE guidance is based on the best available evidence; however, there are several situations in which there may be significant uncertainties about the intervention under consideration. In these cases, recommending the use of the intervention in the context of research can help fill the evidence gaps, while ensuring the efficient use of resources and encouraging innovation that adds value. Making OIR recommendations has been a policy option for NICE since it was established in 1999. However, setting out clear and consistent criteria leading to such decisions and ensuring that the recommended research is undertaken in a timely and responsive manner are challenging tasks. Through reviewing the cases where NICE has issued OIR recommendations, this paper reinforces the importance of acknowledging uncertainty and highlights the difficulties in implementing this policy option within the NHS. With the recent publication of the Cooksey review of UK medical research and NICE introducing new streams of work, including faster technology appraisals closer to licensing and reviews of established treatments to reduce wasteful activities, the OIR option needs to be revisited.
In light of the recently published Citizens' Council report on OIR, we intent actively to engage with the broader public in this exercise so that any policy initiatives are not only consented upon but also driven by those of us who use the NHS.
Competing interests None declared.
Contributorship All authors have participated in data collection and analysis and the drafting of this article. All authors have seen and approved the final version.
Acknowledgments We thank the NICE Research and Development Advisory Committee for encouraging NICE actively to consider the way it addresses uncertainty and the NICE Technical Forum for debating the role of OIR as a NICE decision option. We are grateful to Steven Pearson and Sean Tunis for valuable discussions and Kelly Stevens for her help with identifying ‘OIR recommendations’ from NICE guidance. Finally, we would like to acknowledge the contribution of Jeremy Wyatt and the input of the NICE Research and Development Advisory Committee.