|Home | About | Journals | Submit | Contact Us | Français|
Centralized outcome adjudication has been used widely in multi-center clinical trials in order to prevent potential biases and to reduce variations in important safety and efficacy outcome assessments. Adjudication procedures could vary significantly among different studies. In practice, the coordination of outcome adjudication procedures in many multicenter clinical trials remains as a manual process with low efficiency and high risk of delay. Motivated by the demands from two large clinical trial networks, a generic outcome adjudication module has been developed by the network’s data management center within a homegrown clinical trial management system. In this paper, the system design strategy and database structure are presented.
A generic database model was created to transfer different adjudication procedures into a unified set of sequential adjudication steps. Each adjudication step was defined by one activate condition, one lock condition, one to five categorical data items to capture adjudication results, and one free text field for general comments. Based on this model, a generic outcome adjudication user interface and a generic data processing program were developed within a homegrown clinical trial management system to provide automated coordination of outcome adjudication.
By the end of 2014, this generic outcome adjudication module had been implemented in 10 multicenter trials. A total of 29 adjudication procedures were defined with the number of adjudication steps varying from 1 to 7. The implementation of a new adjudication procedure in this generic module took an experienced programmer one or two days. A total of 7,336 outcome events had been adjudicated and 16,235 adjudication step activities had been recorded. In a multicenter trial, 1144 safety outcome event submissions went through a three-step adjudication procedure and reported a median of 3.95 days from safety event case report form submission to adjudication completion. In another trial, 277 clinical outcome events were adjudicated by a six-step procedure and took a median of 23.84 days from outcome event case report form submission to adjudication procedure completion.
A generic outcome adjudication module integrated in the clinical trial management system made the automated coordination of efficacy and safety outcome adjudication a reality.
To standardize outcome assessments and prevent ascertainment bias, centralized adjudications for primary safety and efficacy outcomes were recommended by the Food and Drug Administration and the European Medicines Agency1,2 and have been used widely in multicenter clinical trials.3–11 Independent blinded central adjudication has been considered as an effective measure to prevent potential biases from site investigators when perfect treatment allocation concealment is not available.12 It has also been a practical strategy to reduce the variation in clinical assessments due to uneven expertise among investigators.13 Outcome adjudication coordination encounters four special challenges. First, there is not a fixed model for various adjudication procedures. Some simple procedures involve only one step, and some others involve multiple steps with specific logic interrelationships. For example, a tiebreaker is required only when the previous two adjudicators disagreed with each other. Second, the timing of some outcome events, such as adverse events, is hard to predict. Third, the allowed time frame for completing an adjudication procedure is often narrow. For example, according to the U.S. Code of Federal Regulations (21 CFR 312),14 an expedited adverse event report must be submitted within 24 hours of learning of the event, followed by a complete expedited report within 5 calendar days of the initial 24-hour report. To fulfill the regulatory obligation, adjudication on the seriousness, the relationship to study agent, and the expectedness of the adverse event must be completed within this time frame. Finally, information on the outcome event may change while the event is being adjudicated.
As a result of these challenges, an automated coordination module for outcome adjudication is rarely available. A recent search on www.PubMed.gov and www.google.com with various combinations of key words “web based outcome endpoint adjudication system” returned only two relevant articles.15,16 Nolen et al. reported a web-based endpoint adjudication system for the central adjudication on eligibility criteria and clinical endpoints prior to two interim analyses in the Caspofungin Study for Fungal Infections in Adults in Critical Care Settings (NCT00095316) planned to enroll 1200 subjects.15 Nolen et al. described the background and the process of the central adjudication procedure, and presented the features of the web-based adjudication system. There was no information on the system design and implementation provided. The trial was terminated after 38 subjects enrolled, and therefore no data on the benefit of using the system were available. Kuurstra et al. reported a web-based adjudication system, served as both an image management system and electronic data capture.16 The paper described the adjudication process for fracture healing trials, the functions and advantages of the web-based adjudication system. No information on the development strategies and techniques were covered, and no data on the benefit of using such system were provided. Outcome/endpoint adjudication modules have been included in some commercial clinical trial management system packages.17–19 However, system architecture design strategies and database structures are often considered as the company’s intellectual properties, and therefore are not expected to be disseminated in public accessible research journals.
As the Data Management Center for the Neurological Emergencies Treatment Trials network and the National Institutes of Health (NIH) Stroke Trials network, both funded by the National Institute of Neurological Disorders and Stroke, the Data Coordination Unit at the Medical University of South Carolina was in charge of providing full scope information support for data management and project management activities. Motivated by the demands from trials in these networks, a comprehensive homegrown web-based clinical trial management system has been developed and has served for more than twenty multicenter clinical trials.20–22 The system uses the three-tier architecture23 with the data tier in a Microsoft Structured Query Language (SQL) Server database, the application tier developed with Microsoft Active Server Pages for data processing and user interface generation, and the presentation tier on internet web browsers. The outcome adjudication module is one of the primary components of this home grown clinical trial management system. (Screenshots of the clinical trial management system are included in the online Appendix 1.)
Dynamic data-driven user interfaces were used in the clinical trial management system. Attributes for each data field, data validation rules, and data access permissions on individual data forms were stored in database metadata tables. User interfaces were composed by system functions based on metadata. Data captured through user interfaces were processed by system functions based on the metadata, triggering database actions for inserting, updating or deleting a record. These system functions were developed generically and applied to all data forms, including case report forms (CRFs) and project management forms. This architecture design provided a foundation for the creation of a generic outcome adjudication module, which used the same system functions to process adjudication data for different adjudication procedures across multiple trials.
Outcome adjudication procedures are heterogeneous. To create a database model supporting different adjudication procedures and also to control the system complexity, the following assumptions were made as system specifications for the generic outcome adjudication module:
In practice, not all outcome adjudication procedures satisfy the above six assumptions. Some simplification and reformatting works may be needed when implementing complex adjudication procedures in to this module.
The database structure of the generic outcome adjudication module is shown in Figure 1. The [Form] table was created to maintain definitions of all data forms. One [Form Table] was created to store data on one data form. Users’ data access permissions were managed by the [User Group Permission] and [User Group Member] tables. Definitions of all adjudication procedures were stored in the [Adjudication Procedure] table. The [Adjudication Step] table was created to store attributes of each adjudication step within an adjudication procedure, including the activate and lock conditions, the categorical adjudication questions, the permissible answers to each of these questions defined in the [Code Item] table, and the view and edit user group IDs.
An adjudication procedure with multiple steps may take days, weeks, or even months to complete. During this period, data changes on the CRF could occur. The [Form Freeze Table] was created for each form table with adjudication tasks. When an adjudication step is submitted and the current CRF is newer than the last saved freeze record or there is no freeze record previously saved, a copy of the current CRF record is saved in the corresponding freeze table. Each adjudication result record is linked to one data freeze record via the freeze record ID. Each adjudication step has the activate condition and lock condition defined in the [Adjudication Step] table, using SQL. A dichotomous value, true or false, is returned when a query condition is evaluated. The activate condition determines if the adjudication step should be performed or not under the current condition composed by information collected in the study database. The lock condition determines if the previously performed adjudication step record can be edited or not. An adjudication step record can be edited only if the step is activated but not locked yet. Depending on the study operation requirements, for each adjudication step, the edit user group and the view user group may have one or multiple members.
The primary goal of the web-based outcome adjudication module is to automate the coordination of adjudication activities, by precisely controlling the workflow, in real-time, regarding who and when needs to do what based on data collected in the clinical trial management system. To achieve this goal, the following generic system features were developed in the outcome adjudication module:
The work flow of the automated coordination of outcome adjudication activities is shown in Figure 2. The activate condition and the lock condition are created to prevent conflicts among the actions of multiple adjudicators. The use of the freeze records allows the CRF to be edited while the adjudication procedure is ongoing. This design minimizes the risk of delayed reporting and updating of important safety and efficacy information. The roles of these designed features will be illustrated with examples in the results section.
This generic outcome adjudication module has been implemented in 10 multicenter clinical trials managed by the Data Coordination Unit at the Medical University of South Carolina. From 2009 to 2014, a total of 29 adjudication procedures were defined, with the number of adjudication steps varying from 1 to 7. Adjudication procedures managed by the generic system included primary efficacy outcome adjudication, serious adverse event (SAE) medical safety monitoring, and central image review, etc. The following two examples are presented to illustrate the system design features and the implementation of actual outcome event adjudication procedures in to the generic adjudication module of the clinical trial management system.
Funded by the National Institute of Neurological Disorders and Stroke, the Progesterone for the Treatment of Traumatic Brain Injury Trial (ProTECT III, NCT00822900) was a randomized, placebo controlled, phase III, multicenter clinical trial to determine if progesterone along with standard medical care for brain injury is more effective at limiting the amount of damage caused by a traumatic brain injury than standard medical care alone.24 The ProTECT III study had a three-step SAE adjudication procedure, which included project manager’s review for CRF completeness, a medical safety monitor’s adjudication for seriousness, unexpectedness, and relationship to study treatment, and project manager’s review of the need for FDA Form 3500 (MedWatch) reporting. Figure 3 shows the work flow of this three-step adjudication procedure. To implement this three-step SAE adjudication procedure in the clinical trial management system, the following tasks were completed by a database programmer:
The automated coordination of the adjudication procedure was assisted by the use of event driven emails programmed in the database. As shown in Figure 3, when an SAE CRF was submitted, auto email #1 was sent to the project manager. If the project manager indicated that the CRF had not been properly completed, auto email #2 was sent to the site study coordinator. When the project manager indicated that the SAE was properly completed and required medical safety monitor adjudication, auto email #3 was sent to him/her. If the medical safety monitor voted for [Serious = No] or [Unexpected = No] or [Relationship = Unrelated or Unlikely], auto email #4 was sent to the project manager to close the adjudication procedure. Otherwise, auto email #5 was sent to the project manager to consider the necessity of the FDA MedWatch form. If the project manager indicated that the SAE required a MedWatch form, the computer system composed a draft of the form by auto-populating data fields based on information in the database and then sent auto email #6 to the site, asking the site to edit and complete the MedWatch form. When the site submitted the MedWatch form to the system, auto email #7 was sent to the project manager, who finalized the MedWatch form and submitted it to the FDA. After that, auto email #8 was sent to all clinical sites and other relevant agencies, notifying them of the SAE and the FDA MedWatch form submission.
Funded by the National Institute of Neurological Disorders and Stroke, the Platelet-Oriented Inhibition in New TIA and Minor Ischemic Stroke Trial (POINT, NCT00991029) is an ongoing prospective, randomized, double-blind, multicenter trial to determine whether clopidogrel is effective in preventing major ischemic vascular events at 90 days when initiated within 12 hours of transient ischemic attack or minor ischemic stroke onset in patients receiving aspirin 50–325 mg/day.25 As the primary efficacy outcome of the study, new ischemic vascular events required a central adjudication for the severity of the event, and the type of death if the severity was fatal. The study was planned to enroll 5840 subjects from more than 200 clinical sites in the USA in 8 years. Due to the importance and the complexity of the outcome events, the potential uneven outcome event assessment expertise across the large number of sites and the long study period, a rigorous outcome event adjudication procedure was implemented in order to protect the quality of the study. The workflow of this outcome event adjudication procedure is illustrated in Figure 4. When a clinical outcome event CRF was submitted by the site, the project manager was alerted by auto email #1 to review the event report type (New or Follow-up) and to determine if the clinical event coordinator review was needed. If the project manager indicated that the CRF did not need the clinical event coordinator review, the adjudication procedure was ended. This scenario occurred when site submitted a clinical outcome event CRF with data edits after the event had been previously adjudicated, and the project manager believed that the data edits did not need to be reviewed by the clinical event coordinator and other adjudicators. If the project manager voted “Yes” for requiring clinical event coordinator review, the clinical event coordinator was notified by auto email #2 to check if the uploaded event packet was complete and the event type was neurological, cardiac, or systemic. If the event packet was incomplete, the site was notified by auto email #3 to provide more information. Otherwise, the system started an independent double adjudication for the outcome event category. If the outcome event was fatal, the adjudicators were required to judge if the death type was ischemic, hemorrhagic vascular, or nonvascular. The adjudication team included three neurologists and three cardiologists. When the clinical event coordinator’s submitted review indicated the outcome event packet was complete, two adjudicators were randomly selected by the system based on the outcome event type identified by the clinical event coordinator, and were notified by auto emails #4 and #5 to vote independently on the outcome event category and the death type if applicable. If discrepancies occurred between the results of the two adjudicators, auto email # 6 was triggered, notifying the third adjudicator to independently adjudicate the outcome event. If the third adjudicator disagreed with both previous adjudicators, the adjudication committee chair was notified by auto email #7 to conduct the final adjudication. When the adjudication procedure completed, auto email #8 was sent to the project manager asking for the closure of the adjudication procedure. If the clinical outcome event CRF was edited by the site at any point during the adjudication process, the project manager was required to decide whether or not the entire adjudication procedure needed to be repeated. The attributes of each adjudication step were specified in the [Adjudication Step] table, as shown in Table 1. The workflow of this sixstep clinical outcome event adjudication procedure is shown in Figure 4.
Although this six-step adjudication procedure was more complex than the three-step SAE adjudication procedure discussed in Example 1, its implementation in the generic outcome adjudication module followed exactly the same seven steps outlined in Example 1.
Table 2 shows the time between two consecutive adjudication steps and the overall adjudication duration length for these two adjudication procedures. In the ProTECT III study, a total of 1253 SAE CRFs were adjudicated. After the adjudication procedure started, 407 (32.5%) CRFs had no data edit submitted, and 82 (6.5%) CRFs had 3 or more submissions. A total of 2521 SAE submissions were reviewed by the project manager. Among them, 1218 SAE submissions were adjudicated by the medical safety monitor. After that, 1144 SAE adjudication procedures were closed by the project manager. For the 2521 SAE submissions, the entire adjudication procedure, from CRF submission to adjudication completion, took a median of 2.76 days. About 95% of the procedures completed within 2 weeks. The 1144 procedures went through all three steps took a median of 3.95 days to complete; about 5% of them took longer than 4 weeks.
The six-step clinical outcome event adjudication procedure for the POINT study was more complex than the three-step serious adverse event adjudication procedure used in ProTECT III trial. A total of 673 clinical outcome events had been reported. Among them, 316 (47%) outcome event CRFs had no data edits after adjudication started, and 41 (6%) had 3 or more submissions after adjudication started. A total of 1210 clinical outcome event CRF submissions were reviewed by the project manager. Based on the review decisions made by the project manager and the clinical event coordinator, 277 submissions were independently adjudicated by the two randomly selected adjudicators. The median duration from CRF submission to adjudication completion for these 277 cases was 23.84 days. Sixteen (5.8%) cases took more than 90 days to complete. Among these 277 adjudication procedures, the two adjudicators disagreed with each other on 40 (14.4%) cases, and the adjudication committee chair was called in for 10 cases where the third adjudicator did not agree with either of the previous two adjudicators.
The integrated, generic outcome event adjudication module provided an ideal solution for our home grown clinical trial management system to automate the coordination of different adjudication procedures in different trials. The database architecture design presented in this manuscript demonstrates a practical strategy to achieve this goal. Our experiences shown that the implementation of a new adjudication procedure into this generic adjudication module took one programmer one or two days to set up.
The described adjudication system has some limitations, mainly due to the generic database structure. The current generic design does not cover the logical relationships, such as skip patterns and conditional selections, among the adjudication questions. During the adjudication process, adjudicators may want to communicate with site investigators for additional information or data clarification. This requirement has not been covered by the current generic adjudication system. For complex adjudication procedures, investigator may find that the five categorical adjudication questions are not sufficient. For example, in the Established Status Epilepticus Treatment Trial (NCT01960075) which we are currently working on, the central adjudication procedure requires two adjudicators to independently answer 25 questions, including 17 categorical, 1 numerical, 2 date/time, and 5 text questions and 15 skip pattern logics among. The complexity of the ESETT clinical adjudication procedure exceeds the capacity of the generic adjudication system presented in this paper. Therefore, a customized application is developed for this ESETT trial.
It is also worth indicating that the automated coordination of adjudication activities does not eliminate the need for human intervention in the adjudication procedures. Often, a project manager is included in the adjudication procedure to ensure that the CRFs are properly completed and the information for clinical adjudication is sufficient. In some cases, the project manager will also assist clinical adjudicators in communicating with site investigators, and will ensure that all required adjudication activities are completed within the protocol specified time windows.
The implementation of an adjudication procedure in the generic adjudication module requires professional database programmers. The setup of the activate condition and the lock condition requires knowledge about the database structure and the relationship among different adjudication steps. The architecture design strategy, the unified framework for adjudication procedures, the generic database structure, and the implementation methods described in this manuscript will be of particular interest to other homegrown clinical trial management system developers.
The authors gratefully acknowledge the comments and advisors received from all members of the Data Coordination Unit at the Medical University of South Carolina. This research is partly supported by NIH/NINDS grants U01NS0059041, U01NS062778, U01NS062835, U01NS087748.
Partially supported by:
Clinical trials: NCT00822900: ProTECT III, David Wright, Emory University NCT00991029: POINT, University of California, San Francisco