PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Clin Trials. Author manuscript; available in PMC 2010 December 11.
Published in final edited form as:
PMCID: PMC3000902
NIHMSID: NIHMS254301

A web-based endpoint adjudication system (WebEAS) for interim analyses in clinical trials

BACKGROUND

A data monitoring committee (DMC) is often employed to assess ongoing trial progress and review safety data and efficacy endpoints. As part of this monitoring, formal interim analyses are sometimes required. Complexity is added to interim analyses when study endpoints are subjective or require clinical expertise to assess. Examples of these endpoints include: AIDS-defining conditions; incidence of infection or disease; disease severity and progression; cardiovascular events; and determination of cause of death. In these circumstances, the sponsor will often require a central endpoint adjudication committee (EAC) composed of clinical experts to determine each subject’s status with respect to the study endpoint [1].

Interim analyses generally occur after a specified number of subjects have completed study assessments, a predefined number of events have occurred, or on a predetermined time schedule (independent of assessment completion or events). Ideally, data are verified and endpoints adjudicated prior to inclusion in interim analyses [2]. To ensure interim analyses are completed as scheduled, efficient methods for completing validation and adjudication are required. Electronic data capture (EDC), automated data checks and query generation, close tracking of site enrollment, and open communication between the investigators and the coordinating center are all used to facilitate expedient data collection and data quality prior to interim analyses.

However, strategies for addressing data adjudication prior to interim analyses are not as refined. Existing strategies include using locally reported outcomes as surrogate endpoints, adjusting analysis methods, and simply performing the adjudication as rapidly as possible.

Existing Methods of Endpoint Adjudication

When adjudicated data are not available, the locally reported outcome is often used as a surrogate for a subject’s endpoint status. By using data reported by the site investigator, the interim analyses can be performed promptly on all available data. However, the DMC risks making a decision about the future of the trial on the basis of unadjudicated and possibly incorrect information. Further, if the results of the interim analysis barely meet study stopping criteria, the DMC may be hesitant to recommend halting the study. Disparities between locally reported and centrally adjudicated endpoints have been documented in studies including the TRIM study, the PERT trial of the Women’s Health Initiative, the PURSUIT study, and MODE Selection trial [36].

A second option is to use the available adjudicated data for the interim analyses while adjusting for any unadjudicated data. Statistical methods for adjusting the interim analyses for unadjudicated data that have been proposed and implemented often involve adjusting estimates using predicted values for unadjudicated cases and weighting unadjudicated events according to the probability that the unadjudicated event will be classified as an event. Many of these statistical methods rely on assumptions about future endpoint adjudications that cannot be verified until after all data have been adjudicated [79]. These statistical methods appear to be a promising area of research; however, available methods are focused primarily on time-to-event endpoints.

Ideally, fully adjudicated data should be used for interim analyses. However, endpoint adjudication is, by nature, a labor-intensive and multi-step procedure. The first step of the process is receiving and validating data used for endpoint adjudication. Methods for facilitating this step include use of EDC and automated data checks. Once all necessary data have been received, the coordinating center has 2 primary methods for disseminating reports. The first method involves distributing review packets to the EAC as either paper copies or electronically. EAC members review the packet, complete an assessment form, and then return the form to the coordinating center. When more than 1 committee member reviews each subject’s data, the reviews are compared. If the reviews do not match, an additional reviewer is sent the subject’s packet and the assessment is considered the tiebreaker. Alternatively, the EAC may meet in person or via teleconference to discuss and adjudicate each case together. Once the final determination has been made on a subject’s status, the adjudicated endpoint is incorporated into the database. Due to the time-consuming nature of these methods, interim analyses may be performed when only a portion of the data available has been adjudicated. Therefore, the DMC may be reluctant to make recommendations about a study based on the analysis results. In a multicenter study of hydroxyurea in sickle cell anemia, the DMC would not recommend study termination after the stopping rule was met because only 78% percent of the endpoints were adjudicated [10, 11].

The ideal adjudication process is to have the EAC meet once in person to review all cases since this approach is the least time-consuming and leads to the most consistent adjudication decisions. However, this approach is usually not feasible when adjudicating endpoints prior to a formal interim analysis. Therefore, a more efficient method for data adjudication in this scenario is needed.

METHODS

Electronic Adjudication

For a National Institute of Allergy and Infectious Diseases (NIAID) and Merck & Co. sponsored protocol, BAMSG 2-01, the Bacteriology and Mycology Biostatistical and Operations Unit (BAMBU) was responsible for developing a strategy for EAC review of both study eligibility criteria and clinical endpoints. BAMSG 2-01 was a randomized, double-masked, 1200 subject trial of caspofungin versus placebo as prophylaxis for invasive candidiasis in high-risk adults in intensive care units (ICU). Subjects were to receive either study therapy or placebo for the duration of their ICU stay or until diagnosed with an invasive fungal infection (IFI). Eligibility status items requiring adjudication included confirmation that the subject met high-risk criteria for invasive candidiasis, did not have any major protocol deviations, and did not have an IFI at baseline. The clinical endpoint requiring adjudication was diagnosis of IFI, including identification of specimen type, fungal species, and date of diagnosis.

Formal interim analyses of efficacy were planned for when one-third (400) and two-thirds (800) of the subjects had completed the study. Enrollment for this study was expected to be approximately 60 subjects per month. The protocol development team and the sponsors felt strongly that identification of IFIs should be performed by experts in invasive candidiasis; therefore, the use of unadjudicated endpoints for interim analyses was considered undesirable. Furthermore, given the planned interim efficacy analyses and anticipated rate of enrollment, data adjudication via a paper-based process or via a meeting of the adjudication committee prior to each interim analysis was considered unacceptably time-consuming. A paper-based process would require time for sending and receiving documents related to the adjudication process, data entry of adjudication results, and subsequent comparison of the reviews. A meeting of the adjudication committee prior to the interim analysis would require that data for all completed subjects to be collected prior to starting the adjudication process. For example, in the time between the 400th subject completing the study and the interim analysis, all 400 subjects’ endpoints would need to be adjudicated and the results of this adjudication implemented in the database. This approach could have potentially required that study enrollment be halted to prevent a significant number of subjects enrolling in the study prior to completion of the interim analysis.

To streamline the adjudication process, BAMBU proposed using a web-based endpoint adjudication system (WebEAS). Conceptually, the system would identify when a subject’s data had been validated and was ready for adjudication, create profiles of the data to be reviewed, automatically select EAC reviewers and notify them that a case required review, electronically capture and compare the resulting reviews, and store the adjudicated results for use in interim analyses. Figure 1 presents a flow diagram of the system’s logic. Furthermore, this process was to be adaptable to other studies. The resulting system was developed on Oracle® and SAS® platforms and relies on pulling data from a clinical database that can be read into SAS® datasets.

Figure 1
Flow Diagram of WebEAS Process

A core team comprising the protocol chair, project manager, study statistician, a software developer, sponsor representatives and representatives of the EAC was established to develop the requirements for the WebEAS process. The core team was responsible for identifying the data requiring verification, designing the subject profile and review form, detailing the review process (including the types and timing of notification e-mails to be sent by WebEAS), preparing timelines for review completion and methods for allocating reviews, as well as developing a detailed guidance document on review completion for the EAC members. The study statistician was assigned to be the system administrator responsible for monitoring the activities of the EAC and coordinating communication among EAC members and the WebEAS core team.

WebEAS Process

The first step in implementing the web-based adjudication system was to develop a method to assess the status of data in the clinical database. Data flag variables, which are included in the clinical database to identify the status of each data point, are used by the adjudication system to automatically identify when a subject’s data are complete and verified. The system checks the data via a SAS® program that determines the status of each subject, outputs the status information into a SAS® dataset and creates a subject profile of the information necessary for EAC review. If auxiliary information is needed for the EAC, such as radiology results or autopsy reports, these documents are also made available electronically to the EAC via the WebEAS. For BAMSG 2-01, the subject profile contained details about study eligibility, study therapy, baseline characteristics, and evidence of IFIs (including the investigator narrative, daily fungal infection assessments, clinical assessments and adverse experiences). Critical data values such as insufficient study therapy dosing, major protocol deviations, and site-diagnosed IFIs were displayed in red text in the subject profile.

An Oracle® program then performs all subsequent tasks related to adjudication, including uploading the profiles to the web-based system and notifying the system administrator (who is responsible for approving and activating the case via the web-based interface). Once the case is activated, the system assigns 2 EAC reviewers and notifies the selected reviewers via email that the case is ready for review. The EAC reviewer logs in to the password-protected WebEAS and is directed to a screen containing the review form, the subject profile, and links to any supporting documentation provided. Reviewers complete and submit their review worksheet online. Once both reviews have been submitted for a case, the system automatically compares the review responses.

For BAMSG 2-01, the core team defined which review form responses would be compared between reviewers. For example, IFIs were defined by the incidence and timing of clinical symptoms as well as microbiology, histopathology, cytology, and radiology results. In order to facilitate the EAC members’ review, detailed questions about all of these criteria were included in the review form. However, only responses for critical fields related to the final diagnosis of IFI were compared. Furthermore, some endpoints requiring adjudication were not dichotomous responses. For example, if a subject had an IFI, the DRC members were to specify specimen type(s) and organism(s) identified. Pull down menus were included in the review form to eliminate the possibility of mis-matching due to spelling errors or multiple naming options for the same organism.

If the reviews match for predetermined critical fields, the review for the case is complete and a final determination regarding the endpoints has been obtained. If the reviews do not match, the system administrator is notified. The tiebreaker reviewer can be assigned and notified either automatically or manually by a system administrator. Once the tiebreaker review is completed, the 3 review responses are automatically compared, and the decision of the majority is considered final. For BAMSG 2-01, if a majority decision was still not reached for the non-dichotomous endpoints (i.e. all reviewers provided different responses), the system notified the system administrator who was responsible for setting up a teleconference of all reviewers to discuss the case and reach consensus. The system administrator then entered the final result into the WebEAS.

Data Storage

Each time a reviewer submits or saves the review, all data entered into the WebEAS review form are stored in an Oracle® database. The Oracle® database contains 3 tables of interest. The first table contains data for all active reviews, the second table contains data for all completed and submitted reviews and the third table contains data for the composite final assessment of each case based on the completed reviews. The composite assessment data from the third table are used for analyses and reporting.

WebEAS Features

To maximize the WebEAS capabilities for obtaining reliable reviews in an expedited fashion, several features were included in the system. These features include scheduled, automated email notifications to EAC members with overdue cases for review and scheduled emails to the system administrator listing overdue cases and the names of assigned reviewers.

Other elements have been implemented to ensure ease of use by both reviewers and the administrator and to obtain quality assessments. Such features in the review form include dynamic control logic and saving and printing capabilities. Dynamic control logic disables a question if the response to a previous question makes the subsequent question irrelevant. An example of the dynamic control logic employed for BAMSG 2-01 was the disabling of questions requesting details of an IFI diagnosis if the reviewer responded that the subject did not develop an IFI. Further, the reviewer is able to save a partially completed review as needed and can return to complete the review at any time. The profile and other supportive documentation are formatted to allow for electronic viewing or, if preferred, printing. When reviews are submitted, the system automatically performs a series of edit checks to ensure that there are no inconsistent or illogical responses. The system administrator defines these checks with input from the EAC. Examples of automated checks incorporated into the system for BAMSG 2-01 included checking that all necessary details of any diagnosed IFIs were provided and dates were logical. If there are any unexpected responses, a message box appears detailing the items on the form that need to be revised. These items are also highlighted within the review form itself. Once all revisions have been made, the review can be successfully submitted.

The WebEAS system also uses version control. In the rare event that data used in a subject profile are updated, a new profile is created for the subject and the case undergoes another round of reviews. All versions of the profile reviewed by the EAC are stored as PDF files and are linked to the review data stored in the Oracle® database by a version number.

DISCUSSION

BAMSG 2-01 study was administratively closed after 10 months with only 38 subjects enrolled. The adjudication process was finalized and the WebEAS system activated prior to study closure. The EAC members had participated in training that involved review of a guidance document; a teleconference discussion of a sample case (during the call, the EAC members had access to the system via their own computer); and EAC member review of 5 additional sample cases. All reviews were consistently filled out with respect to eligibility criteria. There was one difference in the determination of whether or not the subject experienced an IFI resulting from different clinical interpretation of the data provided. Initially EAC members experienced some website accessibility issues; however, once resolved, the reviewers found the system user-friendly and easy to navigate.

Advantages

Performing endpoint adjudication via a web-based system has several advantages. Since the system is automated and utilizes electronic data capture, the EAC process can occur throughout the course of the study and reviews are immediately incorporated into the database and thus available for use in interim analyses. The automated nature of this system also reduces errors that can be introduced while organizing EAC materials, assigning cases and tracking reviews. Additionally, similar to EDC systems for capturing clinical data, the system has built-in edit checks and dynamic control logic to reduce errors introduced during the data entry process, all of which lead to higher quality EAC review data [12,13]. These features work together to allow the interim analyses and DMC reviews to use data that are both current and adjudicated.

Additional advantages to the WebEAS include the ease with which the system tracks the adjudication process and allows for monitoring of EAC performance. Since all data related to the reviews are captured in the database in real time, up-to-date reports can be created to manage the endpoint adjudication. These reports can be used to ensure that EAC members complete all assigned reviews promptly and to track trends to determine if any of the EAC members are consistently interpreting cases differently from other members. All of the WebEAS status reports can be made available to the DRC to aid in their evaluation of study progress.

Limitations

While a web-based data adjudication system offers many advantages, it does have limitations. Unless it is acceptable to perform endpoint adjudication using incomplete data, web-based data adjudication still depends on efficient data collection and verification processes. Methods for ensuring these items are performed efficiently include electronic data capture, automated edit checks and frequent data monitoring. In circumstances where additional source documentation is required, obtaining this information in a timely manner is still challenging. For BAMSG 2-01, some cases did require additional source documentation (such as radiology reports or autopsy reports). Whether or not this information was needed for a case was identified using the clinical data (i.e. the site reported whether or not an autopsy was performed and also whether they used radiology reports to diagnose a possible IFI). The system generated reports for the administrator that identified missing supporting information.

To utilize the capabilities of the WebEAS fully, a dedicated and trained EAC is also needed. The system can only capture the endpoint adjudication data expeditiously if EAC members are diligent in completing reviews as they are assigned. Careful selection, training and continual monitoring of the EAC members’ progress throughout the study can enhance the performance of both EAC members and the system. Even with using the WebEAS, complete adjudication may not be possible in the allotted timeframe. In these instances, the use of the WebEAS in combination with one of the other proposed methods above such as excluding unadjudicated data, including locally adjudicated data, or adjusting the analyses may be required.

Conclusion

The automated nature, convenience, and adaptability of WebEAS made it a plausible solution for obtaining adjudicated data for interim analyses in a timely manner for the BAMSG 2-01 study. The authors believe a similar approach could be useful for handling endpoint adjudication for future clinical trials. Because this system was developed using NIH funds, the programs comprising WebEAS are not proprietary; however, the system requires access to both SASR and OracleR platforms and would require adaptations in order to be used for other study environments (for additional information, contact D. Wallace at gro.itr@ecallawd).

Acknowledgments

Financial Support: This project has been funded in part with Federal Funds from the National Institute of Allergy and Infectious Diseases, National Institutes of Health, under contracts N01-AI-15440 and N01-AI-15441. Merck & Co, Inc provided support for the clinical sites and data review committee.

References

1. Isaacsohn JL, Khodadad TA, Soldano-Noble C, Vest JD. The challenges of conducting clinical endpoint studies. Curr Atherosclerosis Reports. 2003;5:11–14. [PubMed]
2. Pocock S, Furberg CD. Procedures of data and safety monitoring committees. Am Heart J. 2001;141:289–94. [PubMed]
3. Näslund U, Grip L, Fischer-Hansen J, et al. The impact of an end-point committee in a large multicentre, randomized, placebo-controlled clinical trial. Eur Heart J. 1999;20:771–777. [PubMed]
4. Curb JD, Mctiernan A, Heckbert SR, et al. Outcomes ascertainment and adjudication methods in the Women’s Health Initiative. Ann Epid. 2003;13(9, Supplement 1):S122–S128. [PubMed]
5. Mahaffey KW, Harrington RA, Akkerhuis M, et al. Disagreements between central clinical events committee and site investigator assessments of myocardial infarction endpoint in an international clinical trial: review of the PURSUIT study. Curr Contr Trials Cardiovasc Med. 2001;2:187–194. [PMC free article] [PubMed]
6. Peterson JL, Haque G, Hellkamp AS, et al. Comparing classifications of death in the Mode Selection Trial: Agreement and disagreement among site investigators and a clinical events committee. Contemp Clin Trials. 2006;27:260–268. [PubMed]
7. Cook TD. Adjusting survival analysis for the presence of unadjudicated study events. Control Clin Trials. 2000;21:208–222. [PubMed]
8. Cook TD, Kosorok MR. Analysis of time-to-event data with incomplete event adjudication. J Am Stat Assoc. 2004;99:1140–1152.
9. Xie J, Liu C. Adjusted Kaplan-Meier estimator and log-rank test with inverse probability of treatment weighting for survival data. Stat Med. 2005;24:3089–110. [PubMed]
10. US Food and Drug Administration Center for Drug Evaluation and Research. Meeting Transcripts for the Oncologic Drugs Advisory Committee; Dec 181997. [Accessed June 23, 3006]. Available at: http://www.fda.gov/ohrms/dockets/ac/97/transcpt/3367t1.pdf.
11. McMahon RP, Waclawiw MA, Geller NL, et al. An extension of stochastic curtailment for incompletely reported and classified recurrent events: the multicenter study of hydroxyurea in sickle cell anemia (MSH) Control Clin Trials. 1997;18:420–430. [PubMed]
12. Litchfield J, Freeman J, Schou H, et al. Is the future for clinical trials internet based? A cluster randomized clinical trial. Clin Trials. 2005;2:72–79. [PubMed]
13. Lopez-Carrero C, Arriaza E, Bolanos E, et al. Internet in clinical research based on pilot experience. Contemp Clin Trials. 2005;26:234–243. [PubMed]