Search tips
Search criteria 


Logo of jcoHomeThis ArticleSearchSubmitASCO JCO Homepage
J Clin Oncol. 2009 April 10; 27(11): 1761–1766.
Published online 2009 March 2. doi:  10.1200/JCO.2008.19.9133
PMCID: PMC2668703

Steps and Time to Process Clinical Trials at the Cancer Therapy Evaluation Program



To examine the processes and document the calendar time required for the National Cancer Institute's Cancer Therapy Evaluation Program (CTEP) and Central Institutional Review Board (CIRB) to evaluate and approve phase III clinical trials.


Process steps were documented by (1) interviewing CTEP and CIRB staff regarding the steps required to activate a trial from initial concept submission to trial activation by a cooperative group, (2) reviewing standard operating procedures, and (3) inspecting trial records and documents for selected trials to identify any additional steps. Calendar time was collected from initial concept submission to activation using retrospective data from the CTEP Protocol and Information Office.


At least 296 distinct processes are required for phase III trial activation: at least 239 working steps, 52 major decision points, 20 processing loops, and 11 stopping points. Of the 195 trials activated during the January 1, 2000, to December 31, 2007, study period, a sample of 167 (85.6%) was used for gathering timing data. Median calendar days from initial formal concept submission to CTEP to trial activation by a cooperative group was 602 days (interquartile range, 454 to 861 days). This time has not significantly changed over the past 8 years. There is a high variation in the time required to activate a clinical trial.


Because of their complexity, the overall development time for phase III clinical trials is lengthy, process laden, and highly variable. To streamline the process, a solution must be sought that includes all parties involved in developing trials.


Studies that detail various barriers to oncology clinical trials once those trials are open to accrual are plentiful.1,2 Of particular importance are barriers to phase III trials, which require the most patients, time, and resources, serving as gatekeepers for translating promising results obtained in phase II trials to standard of care. There has been little research that investigates preaccrual barriers to clinical trials.35 Before strategies can be developed to overcome these barriers, research must be conducted to identify the process steps required to develop trials for activation.

This process is often a long and laborious one at National Cancer Institute (NCI)–sponsored Comprehensive Cancer Centers (CCCs) and Clinical Trials Cooperative Groups (CTCGs), requiring multiple concept iterations and extensive time commitments.36 Not only do phase III trials have process steps internal to each stakeholder, but they also require steps at Cancer Therapy Evaluation Program (CTEP)7 and Central Institutional Review Board (CIRB)8—steps that may be opaque to other organizations.

CTEP is the largest sponsor of oncology phase III trials in the world. Approximately 130 phase III trials are active annually, and they are primarily completed by CTCGs. Whereas previous analyses have documented the processes unique to trial development at CCCs and CTCGs, this study documents the processes involved in developing and activating CTCG phase III trials. Both CTEP and CIRB review times are included such that the total time involved from initial study conception to activation at CTCGs is captured.35


Study Setting and Timeframe

The first setting for this study is CTEP, a program within the Division of Cancer Treatment and Diagnosis at NCI, located in Rockville, MD. CTEP was established as a program to administer, coordinate, and support clinical trials.9CTEP works closely with CTCGs, CCCs, and other government agencies to accomplish its goal of finding and developing new methods of treating, controlling, and curing cancer.10 CTEP comprises nine offices and branches: Office of the Associate Director, Administrative Office, Clinical Grants and Contracts Branch, Clinical Investigations Branch, Clinical Trials Monitoring Branch, Investigational Drug Branch, Pharmaceutical Management Branch (PMB), Protocol and Information Office (PIO), and Regulatory Affairs Branch (RAB). Each of these fulfills a specific function in the opening of a trial, helping CTEP sponsor roughly 900 of the 1,500 trials of all phases sponsored annually by NCI.11 Representatives from each of these offices and branches were interviewed as part of this study.

The second setting is the CIRB Initiative, which was established in 2001 to help streamline local institutional review board (IRB) review processes.12 Currently, CIRB reviews all phase III trials submitted by CTCGs and approved by CTEP in an effort to reduce the administrative burden on local IRBs while maintaining a high level of protection for human subjects.13

Additional stakeholders involved in the development of phase III trials include (but are not limited to) other components of NCI, such as Biometric Research Branch and the Cancer Trials Support Unit. For certain trials, the US Food and Drug Administration (FDA) and pharmaceutical or device company partners contribute to the review process.

Process Mapping

Using the Dilts and Sandler method,3,4 (1) process steps were identified and mapped, and (2) timing analyses were conducted. The process map demonstrates in graphical fashion the processes required to activate a clinical trial.

A team of experts from the Center for Management Research in Healthcare was engaged to produce the process map, which includes members from the Vanderbilt-Ingram Cancer Center, the Vanderbilt University School of Engineering, and the Vanderbilt University Owen Graduate School of Management (Nashville, TN). The team conducted initial onsite interviews with CTEP and CIRB, with follow-up e-mail correspondence and teleconferences for clarification. A preliminary process map was created representing all major activities involved in activating a phase III trial, which include concept development and review, regulatory affairs/FDA review, protocol development and review, CIRB review, Common Data Element (CDE) review, PMB review, CTEP final review, and CTCG trial activation. It should be noted that concept and protocol development steps are completed through joint efforts between CTEP and CTCGs. To verify the process flows, additional onsite interviews were conducted with relevant participants, and standard operating procedures were consulted to validate the overall process as represented in the map. The final process map was presented to the CTEP team to verify its accuracy.

This study was approved by the Vanderbilt University IRB (IRB No. 060602).

Timing Analysis

Timing analysis was conducted on all CTEP-sponsored CTCG phase III trials activated between January 1, 2000, and December 31, 2007. CIRB was instituted in 2001 and CDE review was initiated in 2002. The timing data are reflective of these initiation dates.

Data were collected from records maintained by PIO, which acts as a central hub between CTEP and CTCGs for all protocol-related submissions. All activities, including submissions of concept and protocol documents for CTEP review, are routed through this office to track and monitor trial progress.

A timing analysis regarding the preparation time required by CTEP, CIRB, and CTCG collaborative efforts was conducted by determining the calendar time from the initial receipt of the concept at CTEP to the date that the trial was activated by CTCGs.

There is additional development time before formal concept submission to CTEP, namely the time required by a study chair to convert the informal concept idea into a formal, CTCG-approved concept for submission to CTEP. Calendar time for these activities was not included in this study. Also omitted are the times required by Cancer Centers or local sites to open the trial to patient accrual.

Trials were partitioned by year activated to identify any trending that may have occurred. Development time was analyzed through descriptive statistics by year and overall.

Timing analysis has a survivorship bias, that is, the sample includes only those trials that were successfully activated and does not include trials withdrawn, disapproved, currently in development, or disapproved but still undergoing development.


Process Mapping

It is impossible to reproduce the complete process flow diagram here because of its size. A high-level summary diagram for CTEP is presented in Figure 1, with the full process map available online.13

Fig 1.
Level 0 process flow map for activating a phase III trial through Cancer Therapy Evaluation Program and Central Institutional Review Board.

There are 11 major aspects to activating phase III trials through CTEP and CIRB: (1) CTCG concept development (with disease-specific CTEP task force assistance, if requested), (2) CTEP Concept Review Meeting or disease-specific Steering Committee Meeting, (3) CTEP RAB coordination of FDA review (if required), (4) drug company support (if required), (5) CTCG protocol development, (6) CTEP Protocol Review Committee (PRC) meeting, (7) CIRB review, (8) CTEP CDE review, (9) CTEP PMB review (if required), (10) CTEP final review, and (11) trial activation.

During concept development, CTCGs, working with potential study chairs (all located external to CTEP), prepare concepts for submission to CTEP. During this stage, organizations may contact a disease-specific CTEP Task Force for assistance in preparing the concept (Fig 2). Task Forces are a new and developing aspect of the CTEP concept submission and review process, intended to aid in clarifying, focusing, and improving concepts before formal CTEP submission. If the Task Force indicates that major revisions are necessary or it does not support the concept, the first of 23 processing loops may occur.

Fig 2.
A detail process flow of initial concept submission illustrating a process loop and a stopping point.

A loop is defined as whenever a concept or protocol is returned to a previous step in the process flow for change. There are many causes of loops in the overall process. Figure 2 illustrates a loop that occurs if the CTCG decides to revise and resubmit a concept for Task Force rereview.

Once the Task Force indicates support for a concept (or if the CTCG decides to submit the concept without Task Force concurrence), the concept is forwarded by the Task Force and/or CTCG to a Steering Committee (if one exists for the disease site) or is submitted to CTEP CRM review for diseases for which a Steering Committee does not yet exist. Although a protocol receives a review from only one of these review bodies, either review may require revision and resubmission, creating additional loops in the process. Such revisions may occur several times before final concept approval. Depending on the extent of reviews required, select members of the committee or the full committee may conduct the rereviews.

After concept approval by CTEP, RAB receives a copy of the concept and communicates with the FDA, whenever a specific trial has licensing potential, to determine whether the FDA has recommendations about the trial design. Organizations progress with trial development while the concept is under FDA evaluation, though all FDA issues must be addressed before CTCG trial activation.

Simultaneously, a letter requesting commitment to the trial is sent, when required, to the pharmaceutical partner. As with FDA evaluation, organizations proceed with trial development before receiving comments on the study design or a commitment response from the pharmaceutical partner.

CTCGs can either begin developing a protocol once they have submitted a concept to CTEP or they can wait for CTEP concept approval. Regardless of which choice is made, CTCG will submit a protocol to CTEP for review after CTEP concept approval.

The CTEP PRC has a process similar to concept review, though it also includes reviewers from CTEP branches involved with regulatory, pharmaceutical, and quality assurance management. Outcomes of PRC review include (1) approved, (2) approved with recommendations (which are suggestions for minor alterations of the protocol), (3) decision pending response (which requires resubmission and rereview), and (4) disapproved (which may result in arbitration if a CTCG wishes to contest the review decision). Protocol development and review consume the largest amount of resources in the process, involving 88 process steps, 17 decision points, and eight possible looping points (Table 1).

Table 1.
Comparison of Works Steps, Decision Points, Loops, and Stopping Points by Major Stage of Development

During the protocol review process, PMB begins the process of negotiating with the pharmaceutical partner for packaging and labeling if the potential trial is the first trial using a particular drug that CTEP will be distributing. PMB also verifies drug supply, making arrangements for additional drug if the supply on hand at CTEP is insufficient. If investigational agents are being used in the trial, then CTEP or the CTCG must also submit the trial as an amendment to the Investigational New Drug application during this time. Though these steps run concurrent to protocol review, they may limit CTEP final approval.

During CTEP review of the protocol, the CTCG may concurrently begin to prepare the CIRB application as well as trial case report forms (CRFs), which are submitted for CDE review. Once the protocol has been CTEP-approved, the CIRB reviews the protocol for human subjects protections by means of a full board review. Review outcomes are approved, approved pending modifications, or disapproved. Trials that are approved pending modifications (“stipulations”) are returned to the CTCG for revision and resubmission. On some occasions, the resubmitted protocol will have changes that extend beyond those suggested by CIRB full board review, which requires that the protocol be returned to CTEP for review of those specific, non-CIRB requested changes. Once those changes are approved by CTEP, the protocol is returned to the CIRB and all changes are reviewed. For the CIRB review outcome of approved pending modifications, the CIRB chair can either approve the changes or recommend that the protocol be rereviewed by the full board. Protocol changes resulting from a review outcome of tabled must be returned to the full board for review. CIRB-approved trials are forwarded to CTEP for review of changes that the CIRB made.

At the same time as CIRB review, the CTCG completes the steps required for development of CRFs and setup of the trial database. Once forms are complete, they are compiled into a CRF packet and sent to CTEP CDE, which reviews them for CDE compliance.

The final major process flow before trial activation is CTEP final protocol approval. This stage is initiated by receipt of notification of CIRB approval and of CDE compliance. Once these two notifications are received, PIO compiles a final review packet, which is sent to PMB and RAB (if applicable). It is at this step that PMB finalizes any issues with drug supply and investigator registration before signing off on final protocol approval. The CTCG is notified of CTEP final approval once PIO receives sign-off from PMB, RAB, and the lead CTEP medical reviewer. CTEP sends a notice to the CTCG that the trial is now CTEP-approved and ready for activation. At this point, the CTCG begins preactivation activities. If there are any preactivation amendments, the protocol must loop back for additional reviews.

A level 0 process map is informative with respect to the overall flow of processes (Fig 1). However, it does not indicate the number of work steps, decision points, processing loops, or stopping points found on a detailed full map. The process for developing a phase III trial through CTEP and CIRB requires at least 296 steps, composed of at least 239 work steps, 52 major decision points, 21 loops, and 11 stopping points (Table 1).

Because a loop may return a concept or protocol for revision and resubmission, the 296 steps identified represent the minimum number of possible steps. More likely, the actual number of steps for a given trial will be considerably greater as a document passes back through previously executed steps. This is particularly the case with CIRB where none of the trials investigated were approved during initial submission.

Timing Analysis

A total of 167 CTEP-sponsored CTCG phase III trials were identified in the system with activation dates between January 1, 2000, and December 31, 2007. Trials in the sample originated across 10 different CTCGs. At the time the sample was collected, 88 trials (52.7%) were open to patient accrual, 75 trials (44.9%) were closed to patient accrual, and four trials (2.4%) were temporarily closed to accrual.

The median clinical trial development time for phase III trials was 602 days (n = 167; interquartile range, 454 to 861 days; Table 2). This time does not include the calendar days required by CTCGs to develop, review, and prepare the concept for CTEP submission. A comparison with a normally distributed curve seems to show that the development time of phase III trials over the 7-year study period is right-skewed with a relatively long tail—with one trial taking 5.2 years (1,908 days; Fig 3).

Table 2.
Phase III CTEP-Sponsored Clinical Trials by Activation Year
Fig 3.
Number of Cancer Therapy Evaluation Program (CTEP) –sponsored phase III therapeutic oncology clinical trials activated from January 2000 to December 2007 organized by development time. Development time is calculated in calendar days from the initial ...

To investigate the question of whether the development time of phase III CTEP-sponsored trials has changed over time, data were segmented into groups based on the year in which the trial was activated (Table 1). Trials that were activated in the year 2007 displayed a significant difference between the individual years 2000, 2002, and 2005 (Kruskal-Wallis; posthoc Bonferroni-corrected α < 0.001). No other comparisons yielded significant statistical differences. No general trend analysis could be extracted because of high annual variances in time, with interquartile ranges varying between 236 days (year 2000) and 482 days (year 2001). There was a consistent presence of trials for which the development time exceeded the 95% CIs of the median (Fig 4).

Fig 4.
Development time (in calendar days) for a phase III trial, Cancer Therapy Evaluation Program concept receipt to activation. Box ranges (H-Spread) indicate lower 25th and upper 75th percentile of the sample. T-bars indicate the 95% CIs by year. Dots indicate ...


Similar to results discovered at CCCs and CTCGs, there are hundreds of steps and dozens of decision points and loops in the activation and opening of a phase III oncology clinical trial. For this research, the focus was on the processes and time required for phase III clinical trials to transit through CTEP and CIRB systems. The calendar time evaluated was measured from initial formal concept submission to CTEP to trial activation at the CTCG. Development stages such as CTEP concept review, RAB coordination of FDA review, CTEP protocol review, CDE review, CIRB review, and CTEP final review were studied. It is important to note that these times are an intermixture of CTEP, CIRB, CTCG (primarily), and study chair times. With rare exception, it is impossible to untangle the threads of time requirements among the various participants in the process.

The results presented here are consistent with and complement research that analyzed the development processes and time at two CTCGs, Cancer and Leukemia Group B and Eastern Cooperative Oncology Group.4,5 As shown in that research, time to trial activation is critical to clinical research. Some research has implicated various individual process stages (eg, contract development) as primary barriers to trial activation.14 However, we have found, to the contrary, that no single development aspect or agency can shoulder the total burden for slow activation time. Rather, timing barriers are created by the numerous loops in the system as a whole and by the need to overcome procedural, structural, infrastructural, and synchronicity barriers pervasive throughout activities conducted by all major participants involved in opening a phase III trial: study chair, CTCG, CTEP, CIRB, pharmaceutical sponsor, and FDA.

As confirmed here and reported elsewhere,35 the steps and time required to develop and activate a phase III clinical trial are extensive. In that previous research, those stages where organizations interface with CTEP and CIRB were a “black box.” Having shed light into that box indicates that it is important that all parties involved in these activities work together to create a more streamlined and effective process. Although there are numerous steps within the CTEP process, it is also important to note that these steps are merely a portion of the total steps required when a potential clinical trial is developed conjointly with CCCs and CTCGs.

Our study has attempted to render CTEP and CIRB review processes more transparent. Steps to parlay some of the obstacles and reduce barriers must be explored if the total development time is to become more efficient without reducing overall clinical trial effectiveness and patient safety. It is our view that efforts to develop a more coordinated, team-based approach to protocol development between CTCGs and CTEP are most likely to be successful. Decreases in the time from concept submission to protocol activation for phase III trials is a worthy goal as clinical trials remain critical in the battle to improve outcomes for patients with cancer.


Supported by Grant No. 3U10 CA 21115-32 from the National Cancer Institute (R.L.) Comis and by subcontract (D.M.D. and A.B.S.).

Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.


The author(s) indicated no potential conflicts of interest.


Conception and design: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Steven Friedman

Financial support: David M. Dilts, Alan B. Sandler

Administrative support: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Steven Friedman, Jeffrey Abrams

Provision of study materials or patients: Shanda Finnigan, Jeffrey Abrams

Collection and assembly of data: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Steven Friedman, Jeffrey Abrams

Data analysis and interpretation: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Margaret Mooney, Jeffrey Abrams

Manuscript writing: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Steven Friedman, Jeffrey Abrams

Final approval of manuscript: David M. Dilts, Alan B. Sandler, Steven K. Cheng, Joshua S. Crites, Lori B. Ferranti, Amy Y. Wu, Shanda Finnigan, Steven Friedman, Margaret Mooney, Jeffrey Abrams


1. Lara PN, Jr, Higdon R, Lim N, et al. Prospective evaluation of cancer clinical trial accrual patterns: Identifying potential barriers to enrollment. J Clin Oncol. 2001;19:1728–1733. [PubMed]
2. Comis RL, Miller JD, Aldige CR, et al. Public attitudes toward participation in cancer clinical trials. J Clin Oncol. 2003;21:830–835. [PubMed]
3. Dilts DM, Sandler AB. Invisible barriers to clinical trials: The impact of structural, infrastructural, and procedural barriers to opening oncology clinical trials. J Clin Oncol. 2006;24:4545–4552. [PubMed]
4. Dilts DM, Sandler AB, Baker M, et al. Processes to activate phase III clinical trials in a Cooperative Oncology Group: The Case of Cancer and Leukemia Group B. J Clin Oncol. 2006;24:4553–4557. [PubMed]
5. Dilts DM, Sandler A, Cheng S, et al. Development of clinical trials in a cooperative group setting: The Eastern Cooperative Oncology Group. Clin Cancer Res. 2008;14:3427–3433. [PMC free article] [PubMed]
6. National Cancer Institute. Report of the Clinical Trials Working Group of the National Cancer Advisory Board: Restructuring the National Cancer Clinical Trials Enterprise.
7. National Cancer Institute. Cancer Therapy Evaluation Program.
8. National Cancer Institute. The Central Institutional Review Board Initiative.
9. National Cancer Institute. Cancer Therapy Evaluation Program: The Mission of the Cancer Therapy Evaluation Program.
10. Ansher SS, Scharf R. The Cancer Therapy Evaluation Program (CTEP) at the National Cancer Institute: Industry Collaborations in New Agent Development. Ann N Y Acad Sci. 2001;949:333–340. [PubMed]
11. National Cancer Institute. Division of Cancer Treatment and Diagnosis.
12. Christian MC, McCabe MS, Korn EL, et al. The National Cancer Institute Audit of the National Surgical Adjuvant Breast and Bowel Project Protocol B-06. N Engl J Med. 1995;333:1469–1474. [PubMed]
13. Center for Management Research in Healthcare. Process Maps.
14. Mello MM, Clarridge BR, Studdert DM. Academic medical centers' standards for clinical-trial agreements with industry. N Engl J Med. 2005;352:2202–2210. [PubMed]

Articles from Journal of Clinical Oncology are provided here courtesy of American Society of Clinical Oncology