|Home | About | Journals | Submit | Contact Us | Français|
Randomized clinical trials, which aim to determine the efficacy and safety of drugs and medical devices, are a complex enterprise with myriad challenges, stakeholders, and traditions. While the primary goal is scientific discovery, clinical trials must also fulfill regulatory, clinical, and ethical requirements. Innovations in clinical trials methodology have the potential to improve the quality of knowledge gained from trials, the protection of human subjects, and the efficiency of clinical research. Adaptive clinical trial (ACT) methods represent a broad category of innovations intended to address a variety of long-standing challenges faced by investigators, such as sensitivity to prior assumptions and delayed identification of ineffective treatments. The implementation of ACT methods, however, requires greater planning and simulation compared to a more traditional design, along with more advanced administrative infrastructure for trial execution. The value of ACT methods in exploratory phase (phase II) clinical research is generally well accepted, but the potential value and challenges of applying ACT methods in large confirmatory phase clinical trials is relatively unexplored, particularly in the academic setting. In the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) project, a multidisciplinary team is studying how ACT methods could be implemented in planning actual confirmatory phase trials in an established, NIH funded clinical trials network. The overarching objectives of ADAPT-IT are to identify and quantitatively characterize the ACT methods of greatest potential value in confirmatory phase clinical trials, and to elicit and understand the enthusiasms and concerns of key stakeholders that influence their willingness to try these innovative strategies.
The use of randomized clinical trials to determine the efficacy and safety of drugs and medical devices is a complex and often cumbersome enterprise with myriad challenges, stakeholders, and traditions. While the primary goal is scientific discovery, clinical trials must also fulfill multiple regulatory, clinical, and ethical requirements. Innovations in clinical trials methodology have the potential to improve the quality of knowledge gained from confirmatory phase trials, the protection of human subjects, and the efficiency of clinical research. Whether traditional or innovative, poorly designed clinical trials can increase the risk to subjects and the likelihood of error. In traditional designs these risks are familiar – although sometimes poorly understood, but innovative methods may introduce risks in ways that are initially unexpected or difficult to interpret.
Adaptive clinical trial (ACT) methods represent a broad category of innovations intended to address a variety of challenges that investigators have faced for a long time, including sensitivity to prior assumptions and delayed identification of ineffective treatments. ACTs have the potential to accelerate both the evaluation of effective treatments and the identification of ineffective therapies, and are now generally well accepted in exploratory phase trials but not in pivotal, confirmatory trials. The implementation of ACT methods requires greater preparatory statistical analysis and simulation, along with more advanced administrative infrastructure. Since traditional designs typically have a closed form solution to the statistical test used for the primary hypothesis, simulation is not required for the estimation of sample size – although in many cases it would be advisable as it would allow the trial planners to more comprehensively evaluate the impact of varying assumptions used in the sample size calculations. In addition, many statisticians in current practice do not have experience or are uncomfortable with ACT methods.1 These considerations may be contributing to slower adoption of adaptive methods in larger, multi-center, confirmatory phase trials. Assessing the potential benefits of ACT methods in confirmatory phase trials, ensuring their validity and safety, and promoting careful acceptance of innovation when warranted has the potential to accelerate the development and evaluation of new therapeutic drugs and devices.
The NIH and FDA jointly funded research proposals aiming to evaluate the applicability of novel approaches to the development of and testing of medical products and fostering the development and evaluation of new tools to accelerate scientific discovery which improves the health of the population.2 In the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) project, a multidisciplinary group of researchers are applying ACT principles and studying the design process within an existing clinical trials cooperative group, the Neurological Emergencies Treatment Trials network (NETT). NETT’s purpose is to perform confirmatory clinical trials of promising drugs and devices for the treatment of patients with acute neurological illness or injury. NETT therefore provides a “real world” laboratory in which to study the process of collaboration necessary to design and implement ACTs.3 This paper describes some of the current limitations in the contemporary clinical trials and the opportunities adaptive designs may present to improve the efficiency of research. The approach used by the ADAPT-IT project will be briefly summarized to illustrate key, potentially high yield adaptive design concepts.
Drug and medical device development in the United States is cumbersome, expensive, and inefficient.4–6 The reasons for this are complex and to some extent, differ by indication and drug/device class. The extraordinary costs of development often discourage sponsors and investigators from pursuing new drugs or exploring additional indications, especially for time limited acute illnesses. In 2004, the Food and Drug Administration (FDA) issued a Critical Path report “to address the growing crisis in moving basic discoveries to the market where they can be made available to patients” and noted a “stagnation” in drug development, manifested by a drop in new drug and biologic entities submitted to regulatory agencies for approval worldwide.4 The FDA report also noted the stifling effect of development costs on new therapies. The 2006 FDA Critical Path Opportunities Report highlighted streamlining clinical trials as an important step towards accelerating and reinvigorating drug and device evaluation.7 The recent Institute of Medicine (IOM) Report “A National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program” highlighted immense obstacles to be surmounted in moving ideas to successful clinical trials.8
One potential contributor to the current state of affairs with respect to the pace and efficiency of discovery is the fundamental structure of most confirmatory phase, contemporary clinical trials. In such trials, most if not all key design characteristics (e.g., the randomization probabilities used to allocate patients to each of the treatment arms, the number of treatment arms, the maximum sample size, eligibility criteria) are held constant during the conduct of the trial. While this approach makes it straightforward to determine the operating characteristics of the trial and provides an intuitive safeguard against bias and error, it may leads to financial inefficiencies.9–12
One of the most concerning and expensive issues in drug development is the increasing rate of failure during phase III confirmatory testing.13 While the root causes of this are multifactorial, one contributor is the limited information available during the design of the confirmatory trial, especially residual uncertainty in the optimal dose, timing of the treatment intervention, rates of side effects, and the likely treatment effect size.14 Importantly, some failed trials may not represent bad treatments, but rather false negative results attributable to design characteristics. In short, traditional confirmatory phase testing requires investigators to commit, prior to the collection of trial data, to a fixed set of study parameters. Historically, these parameters are typically informed by data from much smaller exploratory phase (pilot/phase 2) studies. These pilot studies are often based upon small numbers of subjects which can limit the number of parameters that can be explored and the accuracy and precision of those estimates. It is common for confirmatory phase studies to fail, in part, because of such problems as misestimates of the optimal dose, or overly optimistic or pessimistic estimates in placebo group primary outcome event rate.
During the conduct of a trial that is adaptive by design, key clinical trial characteristics may be altered according to predefined rules, over the course of the trial and in response to information that accumulates within the trial itself (Table 1).9, 10 At least five possible types of adaptations have significant promise for confirmatory trials, but further experience and research is needed to better assess the advantages and disadvantages of these techniques in specific situations (Table 2). These approaches may lead to reduced exposure of trial subjects to ineffective experimental arms, more efficient and accurate dose finding, and more rapid identification of drugs and devices lacking clinically important benefit, so that resources and future subjects may be allocated to other promising interventions.11, 15 The FDA draft guidance on adaptive designs notes that “Compared to non-adaptive studies, adaptive design approaches may lead to a study that (1) more efficiently provides the same information, (2) increases the likelihood of success on the study objective, or (3) yields improved understanding of the treatment’s effect” but warns against “the possible introduction of bias and the increased possibility of an incorrect conclusion.”9 Adaptive designs add complexity, require additional planning, and may be more difficult for unfamiliar clinicians and statisticians to understand.
Despite increasing interest in the potential of adaptive designs to accelerate the evaluation of new therapies, important questions remain regarding optimal approaches to the design and implementation of such trials, the degree with which theoretical improvements in efficiency can be realized in actual practice, and methods to avoid or mitigate threats to validity.9 It is important to emphasize that an ACT, as defined within this overview, includes only adaptations that are predefined, both in terms of the changes that may be made in the trial parameters and in terms of the decision rules dictating when these changes would be made. When both the range of possible adaptations and the rules that dictate these adaptations are defined a priori, the study design is completely defined and its operating characteristics may be rigorously determined by numerical simulation.16–18 In contrast, changes in a trial design that are conceived on an ad hoc basis, such as protocol revisions suggested by a data monitoring committee, lead to difficult-to-define type I error rates that may compromise the validity and integrity of the trial results.9
In the ADAPT-IT project, we will apply and characterize adaptive strategies in planning large scale NIH sponsored confirmatory studies, a setting in which such strategies are infrequently used. Specifically, the ADAPT-IT team will design innovative ACTs to address important questions regarding the efficacy of treatments for patients with acute neurological diseases and injuries and, at the same time, measure attitudes about the innovations and qualitatively study the development process, with the goal of generating new and generalizable information regarding the use of adaptive approaches in confirmatory clinical trials conducted in the academic setting.
The Neurological Emergencies Treatment Trials network (NETT) was funded by NINDS to perform phase III (confirmatory) clinical trials of new treatments for patients with neurological emergencies.1 The purview of NETT covers all neurological conditions in which emergency treatment may improve outcomes. The trial development process within NETT is outlined in Figure 1. At present, NETT is simultaneously conducting three NIH funded confirmatory clinical trials; it recently completed its first trial. To date, all of NETT’s studies have incorporated a group sequential design with early stopping for efficacy and/or futility and have incorporated blinded sample size re-estimation plans. In addition to the conduct of ongoing clinical trials, NETT is actively developing four new clinical trials, involving both drugs and medical devices, at varying stages of development. These trial design activities provide an opportunity to explore the application of ACT designs and assess the barriers to the acceptance and implementation of these designs. Using actual proposed clinical trials enhances this process by incorporating both practical and theoretical considerations. Adaptive trial experts are working closely with the established NETT and clinical teams to plan alternative innovative designs. The design team presents the adaptive design including its advantages, disadvantages, and logistical requirements to each trial’s PI, collaborators and the NETT team of clinicians and biostatisticians who evaluate the design, provide feedback, and potentially incorporate the proposal or its elements into the clinical trial protocol.
The major design goal of ADAPT-IT is to develop, and then test through simulation, adaptive clinical trial features that best address the challenges facing a set of real confirmatory clinical trials. Broad examples of the types of ACTs that may be used to solve these problems are summarized here, as well as the collaborative process that is being used to develop and refine the adaptive elements of these new trial designs.
The adaptive design statisticians on the ADAPT-IT team use Bayesian methods for the majority of the models used in clinical trial design. A full review of Bayesian statistics and the relative advantages and disadvantages of Bayesian versus frequentist designs is beyond the scope of this review, but other works address these questions in some depth.19–22 It is important to note, however, that both frequentist and Bayesian methods are used in both fixed and adaptive designs; not all Bayesian clinical trials are adaptive and not all adaptive trials are Bayesian. One of the advantages of Bayesian approaches is that relatively complex models can be estimated (such as predicting the overall likelihood of trial success when only short term response outcomes are available from some patients) using all of the data accrued from subjects. Advances in computing power have made allowed biostatisticians to feasibly fit these more complicated models and incorporate them into the operation of clinical trials.
Six key design features or elements characterize the approach to ACTs that are used in this project (Table 3). All elements have not necessarily been used in every trial, as a feature of the adaptive approach is tailoring the design to trial-specific research questions. In general, the approach hinges on frequently-updating Bayesian probability distributions for key trial parameters used to predict future events, which in turn support ongoing decision-making according to pre-planned rules. Parameters whose probability distributions are updated during the course of a trial include measures of treatment effects, rates of adverse events, characteristics of dose-response relationships, or the strength of relationships between the primary and intermediate endpoints. Examples of using predictive probabilities for ongoing decision making include predicting whether the current trial will ultimately demonstrate a clinically-important treatment effect, whether the current experimental treatment would be successful in a separate phase III trial, or whether a particular dose meets defined efficacy requirements.14, 23, 24 As a specific example, consider a trial to determine the optimum duration of therapeutic hypothermia after cardiac arrest. Traditionally, a Phase II trial would first examine the safety of several durations (i.e. 12, 24, 36, and 48 hours) and then two would be selected to go forward to a Phase III trial. An adaptive approach allows the investigators to actually estimate the shape of the “dose” (in this case duration) response function and estimate the duration most likely to improve outcomes. Response adaptive randomization could be used to allocate subjects in such a trial to maximize the learning in the part of the duration response curve that appears to have the most efficacy. This would simultaneously improve the precision of the estimate of the treatment effect and minimize the number of patients who are exposed to ineffective durations.
Probability distributions are used to represent the current state of knowledge for each parameter. As the trial progresses, these distributions are updated with accumulating results.25, 26 These methods naturally lead to an estimation of trial parameters and future patient outcomes, in a manner analogous to multiple imputation, incorporating multiple sources of uncertainty and using all available information. A key component of the Bayesian approach is that the ‘act of looking at the data’ or ‘taking apossible design action’ does not alter future probability calculations, making the Bayesian approach ideal for adaptive trials.
Design activities are occurring as follows. First, an initial face-to-face “kick off” meeting is held for each trial where the participants discuss the goals and requirements of the trial, the originally proposed design, potential efficiencies to be realized from an adaptive design, and possible barriers to implementation. Next, an initial design approach is proposed by the adaptive design consultants and presented to the broader group. This design is then subjected to further refinement and simulations, incorporating a wide range of assumptions includingvarying treatment effect, accrual rates, and covariate distributions. Operating characteristics of the proposed ACT designs are estimated using trial simulation. Additional discussions occur as necessary to iterate the design adjustments and re-simulation. In many cases, simulations are programmed in open source statistical packages, including R (cran.r-project.org) and OpenBugs (http://www.openbugs.info/w/) allowing other interested researchers and regulatory bodies to verify the design assumptions that were used in trial planning. The final designs are presented at summative face-to-face meetings.
Innovation is a double-edged sword in scientific research. While innovation is valued and widely recognized as a fundamental driver of scientific progress, initially it can be thought to entail additional risk compared to well-established and familiar methodology. This concern is particularly relevant in confirmatory phase clinical trials because the results are not just intended to inform a subsequent design but instead are meant to influence patient care outside the trial.
ADAPT-IT employs a mixed-methods evaluation of the design and implementation process for confirmatory phase ACTs. Mixed methods research, which uses both quantitative and qualitative methods, has come to be recognized in the social and health sciences as a useful model for conducting research to improve the understanding of observations.27, 28 The evaluation includes mini-focus group evaluations of meeting participants; unstructured observations during meetings; quantitative visual analog scale attitude assessments and qualitative short-answer written assessments about the strengths and limitations of each proposed trial; interviews with study section members and journal peer reviewers; summative evaluations and content analysis of the summary statements returned after study section peer review of the proposed trials.
Through better awareness of what adaptive innovations can and cannot offer in confirmatory clinical trials, and what may or may not be acceptable to stakeholders, the ADAPT-IT project will advance our understanding of how to improve this critical phase of the drug development enterprise. At the same time, the ADAPT-IT project will have a very pragmatic and concrete payoff in the form of detailed and innovative ACT designs for several important neurological emergency conditions, including stroke, status epilepticus, spinal cord injury, and post-cardiac arrest encephalopathy that can be immediately implemented. The designs will either be entirely implemented in the clinical trials, or will help inform important improvements in the final protocols of the trials.
We hope to expand the understanding of circumstances in which adaptive designs for confirmatory trials offer meaningful advantages and in which they do not. In addition, we will gain important insights into the barriers and challenges that may be inhibiting the more rapid adoption of these designs in situations where they could be beneficial. Finally, we hope to be able to offer a model for discussing, considering, and communicating adaptive trial designs for the confirmatory setting across the entire clinical trial process – from the initial idea, to the grant proposal, the trial conduct, the reporting of the trial results, and the interpretation of the results by the broader clinical community.
The ADAPT-IT team will explore important questions regarding the challenges of adaptive designs. Adaptive designs add complexity and cost to designing and implementing a trial and require more resources prior to grant submission – when a project is at high risk of being terminated. Optimal methods will be developed to express the potential efficiencies and costs of different types of clinical trial designs and which situations are suitable to more complicated, adaptive designs. Such designs can behave unpredictably if not carefully simulated, and the designs themselves are more opaque given broad lack of current understanding of statistical simulation in the clinical community. Since it will be crucial for adaptive trials to be credibly interpreted and translated into clinical practice, our process evaluation and data collection process will gain important insights into the pathway of clinical discovery which uses adaptive designs in confirmatory phase trials.
This work was supported jointly by the National Institutes of Health Common Fund and the Food and Drug Administration, with funding administered by the National Institutes of Neurological Disorders and Stroke (NINDS) U01NS073476. In addition, the Neurological Emergencies Treatment Trials (NETT) Network Clinical Coordinating Center (U01NS056975) and Statistical and Data Management Center (U01NS059041) are funded by the NINDS. We specifically recognize the efforts of Dr. Robin Conwit and Dr. Scott Janis, from NINDS for their work with NETT and their assistance in facilitating this project.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
William J. Meurer, Departments of Emergency Medicine and Neurology, University of Michigan, Ann Arbor.
Roger J. Lewis, Department of Emergency Medicine, Harbor-UCLA Medical Center, Los Angeles; Los Angeles Biomedical Research Institute; David Geffen School of Medicine - UCLA.
Danilo Tagle, Extramural Research Program, National Institutes of Neurological Disorders and Stroke.
Michael D Fetters, Department of Family Medicine, University of Michigan, Ann Arbor.
Laurie Legocki, Department of Family Medicine, University of Michigan, Ann Arbor.
Scott Berry, Berry Consultants.
Jason Connor, Berry Consultants.
Valerie Durkalski, Division of Biostatistics and Epidemiology, Medical University of South Carolina.
Jordan Elm, Division of Biostatistics and Epidemiology, Medical University of South Carolina.
Wenle Zhao, Division of Biostatistics and Epidemiology, Medical University of South Carolina.
Shirley Frederiksen, Department of Emergency Medicine, University of Michigan, Ann Arbor.
Robert Silbergleit, Department of Emergency Medicine, University of Michigan, Ann Arbor.
Yuko Palesch, Division of Biostatistics and Epidemiology, Medical University of South Carolina.
Donald A. Berry, Department of Biostatistics, University of Texas M.D. Anderson Cancer Center, Houston, Texas; Berry Consultants.
William G. Barsan, Department of Emergency Medicine, University of Michigan, Ann Arbor.