This review provides a narrative summary of the guidelines that are available from major organisations across the world and UK charities that fund RCTs. We found that funders of clinical trials have not yet incorporated recent concerns about ORB into their recommendations for researchers. There is a need to provide detailed guidance for those conducting and reporting clinical trials to help prevent the selective reporting of outcomes.
In this review we focussed on guidelines for good research practice for clinical trials. The main limitation of this review is that the search of organisations was not exhaustive. We concentrated mainly on organisations from Europe, USA, Australia and South Africa, and only UK charities have been investigated. However, the ICH guidelines, discussed below, are international.
The majority of guidelines provided no references to the problem for several of the questions in the questionnaire (see Additional file 1
, Appendix 3); these were mainly questions specifically regarding outcomes. This is likely to be due to the fact that awareness of the problem of ORB has been limited over the past decade. As we have found, the importance of this issue has not filtered through to these guidelines.
Many of the guidelines state that a trial should be registered before it begins. The purpose of registration is to inform other researchers of trials that are being conducted; the aims include reducing publication bias and the duplication of research [6
Several guidelines that we obtained discuss the importance of adherence to the protocol and state that amendments to the protocol should be reported. This issue is related to ORB as outcomes that are specified as primary or secondary in the protocol should be written up as such in the final report. Insertion of new outcomes and deletion of old outcomes should not occur unless an official amendment to the protocol was made, and such changes should be reported in journal articles. The ICH E3 [25
] guideline states that "if the protocol did not identify the primary variables, the study report should explain how these critical variables were selected (e.g., by reference to publications, guidelines or previous actions by regulatory authorities) and when they were identified (i.e., before or after the study was completed and unblinded)." The Medicines and Healthcare products Regulatory Agency (MHRA) state that "new questions based on the observed data may well emerge during the analysis. This additional work should be strictly distinguished in the report from the work which was planned in the protocol." No such reports were found in the cohort of 102 trials studied by Chan et al [11
] despite over half having changed their primary outcomes in some way.
Publication is often mentioned in funders' guidelines, stating that work should be published in a peer reviewed journal or at least stating that it should be disseminated. The Association of the British Pharmaceutical Industry states that work should be published and refers to guidelines on good publication practice [26
], which aim to ensure that clinical trials sponsored by pharmaceutical companies are published in a responsible and ethical manner.
Many RCTs are carefully monitored by trial steering committees and this requirement is stated in several of the guidelines/terms and conditions. Monitoring tends to focus on whether trialists are following correct procedures. Therefore, issues such as protocol adherence and trial registration should be noticed and highlighted if they are not done correctly. Once a trial has been completed, such oversight may diminish. Ensuring that trial results are published may also be a concern of the data monitoring committee [23
These four issues – registration of a trial, protocol adherence/amendment, publication and monitoring against guidelines – are important with regard to ORB, as the more rigorous the guidelines are in these areas, adherence makes ORB less likely.
Statements found in the guidelines generally refer to publication of negative and positive findings which could be interpreted as referring to study publication bias (i.e. non-publication of whole studies) or ORB (selective non-publication of outcomes). The empirical evidence of publication bias was first seen in the early 1990's [2
] and much work has now been done in the area of publication bias since [6
]. By contrast, studies providing the empirical evidence for ORB were published more recently between 2002 and 2008 [2
]. Some of the more detailed and most often referred to guidelines, such as ICH [17
] and MRC Good Research Practice guidelines [29
], were written before 2000, when ORB first began to be recognised [7
]. However, the Department of Health's Research Governance Framework [30
] was updated in 2005 and provides no guidelines aimed specifically at preventing ORB.
Organisations tended to have their own guidelines whereas the charities mostly had terms and conditions but referred to some of the organisations' guidelines. This may reflect some of the charities being small and that some had only recently begun to fund RCTs.
The CONSORT Statement [20
] addresses the write up of the final report and does not link this explicitly to the trial protocol. It does not specifically refer to ORB as it only states that outcome measures should be specified in the methods section and fully reported in the results section. Although this recommendation attempts to prevent partial reporting of outcomes (e.g. p
-value only), and in addition may help identify outcomes that have not been reported, these recommendations alone will not prevent the selective reporting of outcomes. The CONSORT statement does not state that all
outcomes that were included in the protocol should be stated in the methods section and reported in the results section of the final report. It also does not state that outcomes should not be upgraded or downgraded between the protocol and final report, or that new outcomes should not be added or old outcomes deleted based on their results. Therefore even when trialists follow the CONSORT Statement, it would be impossible to determine whether all outcomes in the protocol are reported in the final report. However, these concerns will be addressed in the revision of CONSORT that is under way. (DG Altman, personal communication)
Since this work has been completed, the members of the WHO Registry Platform Working Group on the Reporting of Findings of Clinical Trials have proposed that "the findings of all clinical trials must be made publicly available [31
In the USA legislation is in place so that legally researchers are required to make their findings publicly available within a specific timeframe [32
]. The legislation states that "The expansion of the registry data bank to include results of clinical trials; beginning not later than 90 days after the date of the enactment of the Food and Drug Administration Amendments Act of 2007, for those clinical trials that form the primary basis of an efficacy claim or are conducted after the drug involved is approved or after the device involved is cleared or approved, the Secretary shall ensure that the registry data bank includes links to results information." It also includes specifics on the reporting of outcomes: "The primary and secondary outcome measures as submitted and a table of values for each of the primary and secondary outcome measures for each arm of the clinical trial, including the results of scientifically appropriate tests of the statistical significance of such outcome measures." It is hoped that these initiatives will reduce the problem of ORB.
This review forms part of the Outcome Reporting Bias in Trials (ORBIT) project
]. It has been funded to investigate how prevalent outcome reporting bias is in trials within systematic reviews and whether this has a large impact on the conclusions of the reviews, as well as trying to understand why trialists do not report all outcomes and investigating the prevalence of ORB in individual patient data reviews on the Cochrane Collaboration Individual Patient Data Methods Working Group register. The identification of this gap in the guidelines is part of a suite of initiatives to reduce the problem of ORB.