|Home | About | Journals | Submit | Contact Us | Français|
The ClinicalTrials.gov trial registry was expanded in 2008 to include a database for reporting summary results. We summarize the structure and contents of the results database, provide an update of relevant policies, and show how the data can be used to gain insight into the state of clinical research.
We analyzed ClinicalTrials.gov data that were publicly available between September 2009 and September 2010.
As of September 27, 2010, ClinicalTrials.gov received approximately 330 new and 2000 revised registrations each week, along with 30 new and 80 revised results submissions. We characterized the 79,413 registry and 2178 results of trial records available as of September 2010. From a sample cohort of results records, 78 of 150 (52%) had associated publications within 2 years after posting. Of results records available publicly, 20% reported more than two primary outcome measures and 5% reported more than five. Of a sample of 100 registry record outcome measures, 61% lacked specificity in describing the metric used in the planned analysis. In a sample of 700 results records, the mean number of different analysis populations per study group was 2.5 (median, 1; range, 1 to 25). Of these trials, 24% reported results for 90% or less of their participants.
ClinicalTrials.gov provides access to study results not otherwise available to the public. Although the database allows examination of various aspects of ongoing and completed clinical trials, its ultimate usefulness depends on the research community to submit accurate, informative data.
The clinicaltrials.gov trial registry was launched more than a decade ago. Since that time, it has been evolving in response to various policy initiatives. The registry now contains information on more than 100,000 clinical studies and has emerged as a key element of many public health policy initiatives aimed at improving the clinical research enterprise. In 2008, a database for reporting summary results was added to the registry. In this article, we present an update on relevant policies, summarize the structure and contents of the results database, and show how ClinicalTrials.gov data can be used to gain insight into the state of clinical research.
Section 801 of the Food and Drug Administration Amendments Act (FDAAA)1 expanded the legal requirements for trial reporting at ClinicalTrials.gov. It was passed into law amid concerns about ethical and scientific issues affecting the design, conduct, and reporting of clinical trials,2 including the suppression and selective reporting of results based on the interests of sponsors,3 unacknowledged alterations of prespecified outcome measures,4 “offshoring” of human-subjects research,5 and failure to report relevant adverse events.6 Among other things, the FDAAA mandates the submission of summary results data for certain trials of drugs, biologics, and devices to ClinicalTrials.gov, whether the results are published or not,7 and imposes substantial penalties for noncompliance. The law’s scope is not limited to industry-sponsored trials intended to support marketing applications but includes studies not intended to inform FDA action (e.g., comparative-effectiveness trials of approved drugs or devices), regardless of sponsorship. Table 1 summarizes the scope of key reporting requirements of the FDAAA and two other policies: the registration policy of the International Committee of Medical Journal Editors8 and regulations being implemented by the European Medicines Agency for registration and results reporting of clinical drug trials conducted in the European Union.9,10
Data in ClinicalTrials.gov are self-reported by trial sponsors or investigators by means of a Web-based system.7 Registration information is generally reported at trial inception. Each record contains a set of mandatory data elements that describe the study’s purpose, recruitment status, design, eligibility criteria, and locations, as well as other key protocol details.11 Additional information may be provided with the use of optional data elements. Before public posting, ClinicalTrials.gov conducts a quality review of the submitted information. Each trial (regardless of the number of study sites) is represented by a single record, which is assigned a unique identifier (i.e., NCT number). Each record is expected to be corrected or updated throughout the trial’s life cycle, and all changes are tracked on a public archive site that is accessible from each record (through a “History of Changes” link). Summary results data are entered in the results database after a trial is completed or terminated (Table 2). Once posted, results records are displayed with corresponding registry (summary protocol) information for each study. Resources and links to additional information are inserted by the National Library of Medicine to enhance the overall usefulness of the database. ClinicalTrials.gov is designed to benefit the general public by expanding access to trial information, but different parts of the database are likely to be of more or less direct use to different audiences.
ClinicalTrials.gov uses automated business rules to alert data providers when required information is missing or when certain data elements are internally inconsistent. After passing automated validation, all submissions are individually reviewed before public posting to assess whether entries are complete, informative, internally consistent, and not obviously invalid; specific criteria for this assessment are described on the Web site.15 Although the review of summary protocol information is generally straightforward, that of results submissions is more complex. The goal, at a minimum, is to determine whether entries provide an accurate depiction of the study design and whether the results can be understood by an educated reader of the medical literature. Some invalid data can be detected by ClinicalTrials.gov staff; however, other data cannot be verified because ClinicalTrials.gov does not have an independent source of study data (e.g., “624 years” is clearly an invalid results entry for mean age, whereas “62.4 years” may or may not be the true mean age). Submissions are not posted on the public site until quality requirements are met; if any important problems are detected (Table 3), results records are returned to the data providers for revision. However, individual record review has inherent limitations, and posting does not guarantee that the record is fully compliant with either ClinicalTrials.gov or legal requirements.
ClinicalTrials.gov is designed to complement, not replace, journal publication. The results database provides public access to a complete set of summary results in a structured system that supports search and analysis. These data are primarily tabular in format and lack significant narrative portions. The database facilitates identification of acts of omission (e.g., incomplete reporting of outcome measures) and acts of commission (e.g., unacknowledged changes to prespecified outcome measures). Journals select research articles for publication on the basis of their target audiences, and the articles supplement reported data with peer-reviewed discussions of background, rationale, context, and implications of findings. Journal editors who abide by the standards set by the International Committee of Medical Journal Editors recognize these complementary roles and consider manuscripts for publication even when the results of a trial have already been posted on ClinicalTrials.gov.8
Table 4 provides summary data on registry and results records for interventional studies that were publicly available on September 27, 2010. As of this date, approximately 330 new registrations and 2000 revised registrations had been submitted each week.
All studies registered at ClinicalTrials.gov are eligible for results submission; however, submission of results is required only for trials covered by the FDAAA (Table 2). Approximately 30 new and 80 revised records had been received each week; we estimate that full compliance with the FDAAA would lead to results submission for approximately 40% of newly registered studies, or over 100 new records per week.
The results of 3284 registered trials had been submitted by 666 data providers. Of these trials, 2324 had been posted publicly; the remainder either were undergoing quality-assurance review by ClinicalTrials.gov staff or were returned for correction.
Of 2178 clinical trials with posted results records, 20% had more than two reported primary outcome measures and 5% had more than five. For some studies, posted results include more than 100 primary and secondary outcome measures. The FDAAA requires the reporting of all primary and secondary outcome measures, and ClinicalTrials.gov does not limit the number of primary and secondary outcome measures that can be listed. Other prespecified and post hoc outcome measures may also be listed.
Of the 2324 posted results entries, 14% were linked to a PubMed citation through an indexed NCT number16; other publications that may exist could be found only through focused PubMed searches. We randomly selected a sample of 150 posted results records in September 2009 and conducted manual searches in an attempt to identify all associated publications. Using all available data, we found that 38 of these studies (25%) had an associated citation in September 2009, and 78 (52%) by November 2010. Although this percentage may continue to increase, it is unlikely that all outcomes from these studies will be published.17
A growing number of researchers are using ClinicalTrials.gov data to examine various aspects of the clinical research enterprise. For example, recent studies evaluated registration records to analyze trends in the globalization of the clinical research enterprise,5,18 the level of selective publication of study results,19,20 and the degree of correspondence between registered and published outcome measures.19-21 Scoggins and Patrick reviewed registration records to identify the types of trials for which patient-reported outcomes were most likely to be reported and the specific instruments used.22 Some authors of systematic reviews have also integrated ClinicalTrials.gov into their search strategies.23 The integrity of trial reporting is a common theme among these studies, which generally focus on whether prespecified procedures in the study protocol (and any subsequent amendments) are appropriate and were followed. This interest has been fueled by highly publicized cases in which trial protocols were not followed, and the subsequent publication of partial results was considered to be misleading.24 The requirement for registering outcome measures at trial inception is designed to address two problems: publication of only some measures and unacknowledged changes in prespecified measures.17,21,25
We used ClinicalTrials.gov data to examine two data fields that are integral to the interpretation of study results: outcome measure and analysis population. These can be considered the “numerator” (outcome measure) and “denominator” (analysis population), respectively, of a study result (e.g., the number of events per number of participants in each group studied). The accuracy and specificity of the information within these fields partly determine their usefulness to a reader as well as their usefulness for assessing the fidelity of published reports to prespecified protocols. (Summaries of the methods used in these analyses are available in the Supplementary Appendix, available with the full text of this article at NEJM.org.)
ClinicalTrials.gov instructs data providers to report the specific measure and time frame for each primary and secondary outcome measure at registration, reflecting the current international standard for trial reporting.26,27 Experience with reporting outcomes in a tabular format in the results database has emphasized the need for the description of a measure to be specific in order to sufficiently form the rows for the results table (with comparison groups as columns). In addition to time frame, a fully specified outcome measure includes information about the following: domain (e.g., anxiety), specific measurement (e.g., Hamilton Anxiety Rating Scale), specific metric used to characterize each participant’s results (e.g., change from baseline at specified time), and method of aggregating data within each group (e.g., a categorical measure such as proportion of participants with a decrease greater than or equal to 50%) (Fig. 1).
We reviewed the first primary outcome measure, as initially registered, from 100 randomly selected non-phase 1 clinical trials in August 2010. Entries were assessed for whether a specific time frame was provided and were categorized according to level of specification (Fig. 1). We categorized 36% as level 1 (i.e., domain only), 25% as level 2, 26% as level 3, and 13% as level 4; of these, 72% included a specific time frame. When only a specific measurement or domain is registered, as occurred in 61% of the entries in our sample, post hoc choices of the specific metric or method of aggregation could mask the fact that multiple comparisons were conducted, potentially invalidating the reported statistical analyses and allowing for cherry-picking of results. Some argue that the method of aggregation (level 4) is part of the statistical analysis plan and may properly be specified later — after data accrual but before unblinding. The archive feature of ClinicalTrials.gov enables those viewing such records to see the originally registered outcome measure and the full timeline of changes (if relevant).
The analysis population is another source of potential bias in results reporting. Substantial distortion of results can occur if all data are not accounted for or if missing data are not handled appropriately. The use of different analysis populations for different outcomes may not be noticed by many readers, but it can exert a strong effect on reported results. In a sample of 700 records (representing 1749 study groups and 5160 outcome measures), the mean number of different analysis populations per study group with at least one participant was 2.5 (median, 1; range, 1 to 25). The magnitude of the difference across groups and outcomes varied. To further explore the magnitude of these differences, we evaluated the percentage of participants who started the study and were analyzed for the first primary outcome measure in a sample of 684 eligible studies (representing 1706 groups). Approximately 31% of trials included 100% of participants in the analysis, and 24% of trials reported results for 90% or less of their participants (see the Supplementary Appendix). Determination of the appropriateness of the analysis population for any specific outcome analysis would require a detailed methodologic review of each study and would potentially involve subjective judgments.
In the past 5 years, prospective registration of clinical trials has become standard practice. Public reporting of summary results, independent of the interests of the trial sponsor, represents the next step in this international experiment in systematic disclosure of clinical trial information. It has been 2 years since the launch of the ClinicalTrials.gov results database, and people can now access summary trial data that were not previously available publicly. Researchers, policymakers, and others can now examine features and trends of the clinical research enterprise that were previously difficult to study. For example, methodologists may evaluate the appropriateness of designing trials with many primary outcome measures. Policymakers may consider how current patterns in the use of data monitoring committees might affect the quality and safety of the resulting research. Others may use the data to monitor trends in the clinical research enterprise and raise questions about the portfolio of trials relative to public health needs. The ultimate usefulness of the registry and results database will become apparent as more trial information and results are posted and as persons with different interests and needs incorporate these data into their analyses. ClinicalTrials.gov is continually adding features and linkages to facilitate the use of the data by different audiences, and other groups repackage these data for more specific audiences. For example, BreastCancerTrials.org (www.breastcancertrials.org) augments registry data to serve the breast-cancer community. The Clinical Trials Transformation Initiative is leading an effort to develop a publicly accessible research-quality data set in order to facilitate examination of the clinical research enterprise.28
When one is using the data in ClinicalTrials.gov, however, certain limitations should be kept in mind. First, there are undoubtedly trials that are not registered in ClinicalTrials.gov or any other publicly accessible registry. Coverage in ClinicalTrials.gov is likely to be most complete for trials of drugs or devices that are sponsored by U.S.-based or multinational organizations (e.g., major pharmaceutical companies). Second, some records are missing information (e.g., optional data elements) or contain imprecise entries. We are not able to impose requirements beyond those of the prevailing federal law; trials registered since the passage of the FDAAA have to meet more requirements than do older trials, but investigators who use all ClinicalTrials.gov data will encounter many records with missing fields. In addition, given the demands of individual record review, some problematic entries will find their way onto the public site. Third, new registry and registration policies are being implemented in specific regions and countries around the world. The World Health Organization has established a search portal that includes data from ClinicalTrials.gov and 11 other registries, totaling more than 123,500 records as of November 23, 2010. However, overlapping scope and inadequate coordination internationally have contributed to the difficulty in determining the precise number of unique trials being conducted.
Disclosure requirements for clinical trials continue to evolve. In the United States, the FDAAA calls for the expansion of the basic results database through rulemaking “to provide more complete results information” and mandates the consideration of issues such as requiring results reporting for trials of drugs and devices that have not been approved by the FDA, the inclusion of narrative summaries, and the submission of full study protocols. In general, a guiding principle is that expansion of the registry and results database should only improve on, not reduce, the functionality and usefulness of ClinicalTrials.gov. Information about the status of the rulemaking process, including notification of the opportunity to provide comments, can be found at http://prsinfo.clinicaltrials.gov/fdaaa.html. Internationally, the European Medicines Agency is planning to make summary protocol and results information publicly available for clinical trials of approved and unapproved drugs conducted in the European Union. Efforts are under way to ensure the compatibility of the European Medicines Agency database with ClinicalTrials.gov, thus potentially minimizing reporting burdens for those conducting multinational trials and supporting seamless access to results from many parts of the world.29
Despite the change in cultural expectations regarding trial disclosure and the fact that many trial sponsors and investigators are successfully meeting the requirement to submit summary results, our experience to date indicates that others are still struggling. In addition, the poor quality of some submitted entries is troubling. As Beecher observed in 1966, a “truly responsible investigator [emphasis in the original]” is essential if the rules governing clinical research are to have the intended effect.30 Similarly, the usefulness of ClinicalTrials.gov ultimately depends on whether responsible investigators and sponsors make diligent efforts to submit complete, timely, accurate, and informative data about their studies.
We thank Drs. Alastair J.J. Wood, Harlan M. Krumholz, and Joseph S. Ross for comments on earlier versions of this manuscript; Sarah O. Kornmeier and Annice M. Bergeris for help with data analysis; and Jonathan McCall for editorial assistance.
Disclosure forms provided by the authors are available with the full text of this article at NEJM.org.