Publication bias can occur in several ways, including not publishing data at all, selectively reporting data, or framing data. We found evidence of both lack of publication and selective reporting of data. Seventy-eight percent of the trials submitted to the FDA were published, and trials with active controls or statistically significant outcomes favoring the test drug were more likely to be published. In a multivariate model, trials with favorable primary outcomes (OR = 4.7, 95% CI 1.33 to 17.1, p = 0.018) and active controls (OR = 3.4, 95% CI 1.02 to 11.2, p = 0.047) were more likely to be published. In addition, reporting sometimes differed between trials as they were described in FDA reviews and their corresponding publications. These changes included the addition or deletion of outcomes, changes in statistical significance of reported outcomes, and changes in overall trial conclusions. Papers included 155 outcomes that were also in the NDAs, 15 additional outcomes that favored the test drug, and two other neutral or unknown additional outcomes. Excluding outcomes with unknown significance, there were 43 outcomes in the NDAs that did not favor the NDA drug. Of these, 20 (47%) were not included in the papers. The statistical significance of five of the remaining 23 outcomes (22%) changed between the NDA and the paper, with four changing to favor the test drug in the paper (p = 0.38). Excluding unknowns, 99 conclusions were provided in both NDAs and papers, nine conclusions (9%) changed from the FDA review of the NDA to the paper, and all nine did so to favor the test drug (100%, 95% CI 72% to 100%, p = 0.0039). All of these changes in reporting led to more favorable presentations of the NDA drug in the published articles.
Our findings extend those of others by demonstrating that reporting bias occurs across a variety of drug categories and that statistical significance of reported primary outcomes sometimes changes to give a more favorable presentation in the publications [10
]. These changes occur primarily in peer-reviewed, moderate impact factor journals that disclose funding sources and other financial ties. Thus, publication of trial data in peer-reviewed publications appears to be inadequate, supporting the need for reporting of full protocols and findings in a trial registry [31
]. There have been several advances in trial registration, including The World Health Organization International Clinical Trials Registry Platform, a US register (http://www.clinicaltrials.gov
), and the International Committee of Journal Editors requirement for pre-registration of all trials published in participating journals from late 2005, and the FDA Amendments Act requirements for study results posting. Our findings suggest that these registries should contain, at a minimum, full reporting of results for all primary outcomes that are reported for trials submitted for regulatory approval.
Responses from investigators to our inquiries about unpublished studies suggest that studies were not published because they were not submitted to journals. Several other studies, based on self-reports from authors with unpublished studies, suggest that authors' decisions not to submit manuscripts account for the majority of unpublished studies [2
]. A prospective cohort study of 1107 manuscripts submitted to three major medical journals found that having statistically significant results did not improve the chance of a study being published, although studies of higher methodological rigor were more likely to be published [36
]. In our opinion, investigators have an ethical obligation to submit the results of their research for publication.
Our study has several limitations. We were unable to determine why results were changed from the FDA review of the trial in the NDA to publication. Possible explanations for changes in the primary outcome(s) include: there was a problem with the measurement of the primary outcome as identified by the sponsor or the FDA; the primary outcome as submitted to the FDA did not have a favorable result; or the primary outcome was criticized by the FDA. Changes in the reporting of the significance of primary outcome(s) may have been due to either changes in analysis initiated by the sponsor in order to improve the results or changes in analysis due to criticism by the FDA of the initial analysis used. We could not investigate predictors of change to more favorable results because almost all of the results started out as favorable, therefore leaving only a small subset of results that could change to more favorable reporting in the publication. Another limitation was the quality of our data sources from the FDA. Although clinical trial data submitted to the FDA are publicly available, they are not in a format that is easily accessible, and the documents are often incomplete. Trials that are included in the efficacy or safety analysis are not clearly indicated. They could, for example, be listed in a table. In addition, the data are available only in the FDA review, and not as originally submitted by the sponsor. Thus, the primary outcomes as identified in the FDA review could differ from those listed in the original trial protocol. Lastly, it is possible that sponsors may not submit all their data to the FDA.
The goal of this study was to determine whether information that is available to the FDA is readily accessible to clinicians, and whether it is presented in the same way. As we hypothesized, not all data submitted to the FDA in support of a new drug approval were published, and there were discrepancies between original trial data submitted to the FDA and data found in published trials. Thus, the information that is readily available in the scientific literature to health care professionals is incomplete and potentially biased.