In a sample of NIH funded clinical trials registered in ClinicalTrials.gov, we found that fewer than half of trials were published in a peer reviewed biomedical journal indexed in Medline within 30 months of trial completion, although there were more recent improvements in timely publication. Furthermore, after a median of 51 months after study completion, we found that about a third of NIH funded trials remained unpublished. Twenty years ago, a single study focused on trials funded by NIH in 1979 found that 93% of completed trials were published within 10 years, using a broader definition of publication that included research abstracts, book chapters, and other non-Medline indexed sources.4
In contrast, a larger body of work focused on industry sponsored studies suggested that between 25% and 50% of clinical trials remain unpublished even several years after completion.1
Our results suggest that patterns of publication are similar for publicly funded and for industry funded clinical trial research and that substantial amounts of publicly funded research data are not published and available to inform future research and practice.
The current culture of research needs to prioritise the timely public dissemination of research findings via peer reviewed journals among research funded by both public and private sources. Sponsors and investigators need to prioritise the publication process, managing it as closely and rigorously as any other part of the scientific process, regardless of the outcome: positive, negative, or inconclusive. In previous years, publishing negative findings from clinical trials was considered far more difficult, and trials with positive findings were more likely to be recommended for publication.26
Two factors, however, facilitate the publication of negative findings. Firstly, the growth in online only journals, such as those offered by the Public Library of Science (PLoS) and BioMed Central (BMC), and the practice of print journals publishing articles “online only” has reduced competition for space among print journals. Secondly, the ICMJE, among its uniform requirements for manuscripts submitted to biomedical journals, has issued a statement compelling the publication of trials with negative results.27
Clearly, many steps occur between the completion of data collection and the publication of the central research findings from a clinical trial, only some of which are under the control of the investigators. Preparations for publication and dissemination, however, can be made in advance of the trial’s completion, including data management and cleaning, analysis planning, and drafting of parts of the manuscript. Among those NIH funded trials that were published in a peer reviewed biomedical journal indexed by Medline, some required as little as a few months for publication, others as long as five years. Trials that required several years to be published could potentially have been moved forward more expeditiously. Clear and reasonable expectations for timely dissemination are needed; perhaps the goal should be to have results from clinical research reported publicly 12-24 months after study completion. Interestingly, in our sample, trials completed in 2007 or 2008 were published more quickly after study completion than trials completed before 2007, suggesting that the timeliness of dissemination could be improving. This issue deserves re-examination in future years.
To this end, US policies have been recently enacted that might further improve public reporting of results. Enacted in 2007, the FDA Amendments Act requires the sponsors or designated principal investigators of nearly all non-phase I trials of FDA regulated drugs, biologicals, and medical devices to register their trials at inception within ClinicalTrials.gov. Critically, this law also requires public reporting of summary results among trials of FDA approved drugs and devices within a legally mandated timeframe, often within 12 months of trial completion, regardless of publication status. Specifically, investigators must report results for all primary and secondary outcomes to the ClinicalTrials.gov results database for public posting.28
Importantly, the ICMJE has clarified that member journals should not consider reported results to be a previous publication if they were posted in ClinicalTrials.gov.29
Although the act should make findings from NIH funded research more available, there are many NIH funded trials that are not covered by it, such as trials of behavioural interventions and surgical procedures. Results from these trials can be voluntarily submitted to ClinicalTrials.gov, but there is no law or policy requiring reporting or publication of results from these studies. Since April 2008, federal law requires that any published article that describes the results of research funded wholly or in part by the NIH is to be made publicly available no more than one year after the date of publication through PubMed Central, as part of the NIH’s public access programme.30
No policies exist, however, to ensure that the public has access to results from NIH funded research that is not published, although there are policies promoting data sharing among NIH projects receiving more than $500
000 in direct costs per year.31
Lack of dissemination of research findings disrupts the scientific process and leads to redundant efforts and misconceptions about clinical evidence, undermining not only the trial in question but also the evidence available in the medical literature that forms the basis for systematic reviews and meta-analyses, evidence based clinical and policy decisions, and even institutional review board assessments of risks and benefits associated with future research studies. In addition, non-publication violates the commitment made to trial participants, all of whom took part in the research and possibly placed themselves at some risk in an effort to contribute to clinical science. International guidelines on protecting participants in research, such as the Declaration of Helsinki, also recognise the ethical obligation in publishing or otherwise making publicly available “negative and inconclusive as well as positive results.”32
There are many studies “competing” for trial volunteers, yet we found that data from about 30% of participants in trials have not contributed to the publicly available knowledge base available through Medline 30 or more months after study completion. Given the concerns raised about insufficient numbers of volunteers for clinical trial research,33
a key step should be to ensure that data from all those who volunteer are published and widely used to inform future research and practice.
ClinicalTrials.gov and its results database offer a unique opportunity and venue for sponsors and investigators to report results from NIH funded clinical trials, as well as all other studies, to the broader public, regardless of whether or not the results are published in a peer reviewed journal. As a central repository already being used for trial registration,24
the site will be familiar to researchers and regulators. Although the legal requirements pertain only to certain interventional studies of drugs and devices, all studies registered in ClinicalTrials.gov are eligible to use the database for reporting summary results. Finally, using the results database ensures that summary data from trials are reported with tabular data forms that can be easily searched and do not limit reporting only to specific results relevant only to particular target audiences.
The central limitation to our study is that we focused our search for relevant publications from NIH funded research to peer reviewed biomedical journals indexed by Medline and did not search other databases, such as Embase or research conference proceedings (abstracts). Our search for publications was extensive, however, involving two independent investigators using systematic methods. In addition, some trials might have been made publicly available elsewhere, such as through investigator or institutional websites. Another factor to consider is that even though Medline initiated efforts to index publications using ClinicalTrials.gov NCT numbers in July 2005, the practice didn’t become common until later, perhaps making it more difficult to identify publications from older trials.
There are additional limitations to consider. This study relied on data in ClinicalTrials.gov that are reported by the trial sponsor or responsible investigator. The National Library of Medicine cannot verify the validity of the trial information, and there might be errors in some records. In addition, there is some uncertainty in our estimates of trial enrolment as for some trials within ClinicalTrials.gov only the target enrolment number was reported, which might differ from the actual enrolment number. Moreover, our study was limited to NIH funded trials and cannot be generalised to other international government agencies that fund clinical trial research.
Finally, there are several reasons why our overall observation that 32% of NIH funded trials were not published in a peer reviewed biomedical journal indexed by Medline is a conservative estimate of non-publication. Firstly, we limited our analysis to trials that were registered and had updated their status to completed, indicating that they were no longer recruiting or observing participants. We would expect that some proportion of initiated trials would be registered and have difficulty with recruitment or follow-up but continue to pursue these goals without formally terminating or completing the trial. Secondly, we limited our analysis to trials that were registered and provided a study completion date. Our previous work has shown that completed trials that do not provide study completion dates within ClinicalTrials.gov were less likely to be published than those providing the information.18
In addition, some institutes, such as the National Cancer Institute, were less likely to provide trial completion dates and were thus under-represented in our sample. Finally, we limited our analysis to trials that were completed by December 2008, providing a minimum of two and a half years for investigators to publish the trial findings in a peer reviewed biomedical journal indexed by Medline. We expect lower publication rates among trials that have had fewer than two years to publish the findings.
In conclusion, despite recent improvement in timely publication, overall we found that fewer than half of NIH funded trials in our sample registered within ClinicalTrials.gov were published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion. Moreover, after a median of 51 months after trial completion, we found that a third of trials remained unpublished. Although there might be many reasons for lack of publication, the results database at ClinicalTrials.gov offers a complementary method of providing timely public access to study results, although the peer reviewed literature is likely to remain the principal method of communicating with clinicians and policy makers. Steps must be taken to ensure the timely dissemination of publicly funded research so that data from all those who volunteer are available to inform future research and practice.
What is already known on this topic
- Previous work, largely focused on industry sponsored studies, suggests that between 25% and 50% of clinical trials remain unpublished even several years after completion
- Little is known about the patterns of publication of publicly funded trials
What this study adds
- Fewer than half of NIH funded trials registered after September 2005 within ClinicalTrials.gov and completed by December 2008 were published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion
- After a median of 51 months after study completion, a third of NIH funded trials in our sample remained unpublished