PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jrsocmedLink to Publisher's site
 
J R Soc Med. Apr 2007; 100(4): 187–190.
PMCID: PMC1847744
Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report
Mike Clarke,1 Sally Hopewell,1 and Iain Chalmers2
1 UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK
2 James Lind Initiative, Summertown Pavilion, Middle Way, Oxford OX2 7LG, United Kingdom
Correspondence to: Professor Mike Clarke E-mail: mclarke/at/cochrane.co.uk
Objective Scientific and ethical justification for new clinical trials requires them to have been designed in the light of scientifically defensible assessments of relevant previous research. Reliable interpretation of the results of new clinical trials entails setting them in the context of updates of the reviews upon which they were deemed scientifically and ethically justifiable. We have shown previously that most reports of randomized trials published in five general medical journals in May 1997 and in May 2001 failed to set their results in the context of the findings from similar research. In the current study, we assess whether there had been progress in this respect in 2005 and also investigate the extent to which reports begin by referring to systematic reviews providing the justification for the new research reported.
Design Assessment of the Introduction and Discussion sections in all reports of randomized trials published during May 2005 in five general medical journals.
Setting Reports of randomized trials in five general medical journals.
Participants Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine.
Interventions None.
Main outcome measures The inclusion or mention of one or more systematic reviews in the Introduction or Discussion section of each report assessed.
Results We found 18 reports of randomized trials. The Introduction sections referred to systematic reviews in five (27%) of these reports. None of the Discussion sections of the 15 reports of trials that were not the first published trials to address the question studied placed the results of the new trial in the context of an updated systematic review of other research. Although reference was made to relevant systematic reviews in five of these 15 reports, there was no integration—quantitative or qualitative—of the results of the new trials in an update of these reviews. In the remaining ten reports there was no evidence that any systematic attempt had been made to set the new results in the context of previous trials.
Conclusions There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial. Researchers and journal editors do a disservice to the interests of the public and others involved in healthcare decision-making by acquiescing in this situation.
Scientific and ethical justification for new clinical trials requires that they be designed in the light of scientifically defensible assessments of relevant previous research.1 Then, reliable interpretation of the results of new trials entails setting them in the context of an updated systematic review of similar studies.2
In May 19973 and again in May 20014 we found and assessed 26 and 33 reports, respectively, of randomized trials in Annals of Internal Medicine, BMJ, JAMA, Lancet and the New England Journal of Medicine. We identified reports of apparently similar trials for 55 of these index trials, yet in only two were new results placed in the context of up-to-date systematic reviews of other relevant studies. In seven reports, systematic reviews were referred to in the Discussion sections of the reports, but in the other 46 reports no apparently systematic attempt had been made to set the results of the new trials in context. Thus, in both of our previous samples of trials, only a small proportion of the reports provided sufficient information to assess what contribution the new results had made to the totality of information, and thus permit reliable interpretation of their significance.
We repeated our studies in May 2005 to assess whether there had been any detectable improvement. In this study, as in the previous two, it was not our aim to assess the overall quality of the Discussion sections of trial reports, but simply to assess how well they had placed the results of the new trials in the context of other relevant research. Furthermore, we now also investigated the extent to which reports began by referring to systematic reviews to make clear the scientific and ethical justification for the design and conduct of the new research.
Eligibility
A report was eligible for inclusion as a ‘trial’ if it met the following three criteria:
  • It was published during May 2005 as a full report or paper (that is, not in the editorials, news, research letters, short reports or correspondence sections of the journals), in Annals of Internal Medicine, BMJ, JAMA, Lancet, or the New England Journal of Medicine.
  • On the basis of information in the report, ‘the individuals (or other units) followed in the trial were assigned prospectively to one of two (or more) alternative forms of health-care using random allocation or some quasi-random method of allocation (such as alternation, date of birth, or case record number)’—that is, randomized and quasi-randomized trials, as defined by The Cochrane Collaboration.
  • The report was principally concerned with the outcomes studied in the trial.
Identification of eligible reports and assessment of the Introduction and Discussion sections of the reports
We each searched the relevant issues of the journals in different random orders. Any report judged to be eligible by at least one of us was considered for inclusion.
We independently assessed the Introduction section of each eligible report to decide whether it referred to a systematic review (done by the authors or other people) that could have been used in planning the trial. We did not attempt to identify whether relevant systematic reviews were available at the time of planning those trials that did not refer to such a review in their Introduction. To do so would have required more knowledge than was available about when each trial was planned. Furthermore, the nonavailability of a review by others should not preclude the researchers from performing their own systematic review and, if they had done so, we might expect that they would have described this in their Introduction section.
We also independently assessed the Discussion section of each eligible report to decide whether an attempt had been made to integrate the results of the new trial within a systematic review, either qualitatively or quantitatively.
If a trial claimed to be the first or only trial of a topic, we searched for similar trials in the Cochrane Central Register of Controlled Trials (CENTRAL). We did not systematically search for all such trials or judge whether there was sufficient similarity between the new trial and other trials to combine them in a formal meta-analysis. We simply tried to find studies that might be considered for inclusion in a systematic review of the topic.
We resolved disagreements by discussion.
Eighteen reports of randomized trials (listed in Appendix A) were identified in the issues of these five journals published in May 2005. We did not include two economic analyses based on randomized trials.
Systematic reviews were referred to in the Introduction sections of five reports, and these could have been used in planning the newly reported trials. Reports of the remaining 13 trials did not provide any information on whether they had used systematic reviews done by themselves or other people to justify or inform the design of the index trial, or whether they had searched for systematic reviews.
In five reports, the authors claimed that their study was the only one to have addressed the question concerned, but we found apparently similar trials for two of these. None of the 15 reports for which there appear to be similar trials contained a discussion of the trial's results in the context of an up-to-date systematic review of earlier trials. Reference was made to relevant systematic reviews in the Discussion sections of five reports, but without integrating the results of the new trial—either qualitatively or quantitatively—into an update of the review. Four of these five had referred to a systematic review in their Introduction sections, while one had not done so. The fifth report in which a systematic review was referred to in the Introduction was categorized as a first and only trial when we assessed its Discussion section.5 It was a report of the 30-year follow-up of the first trial of a treatment, and none of the subsequent trials are old enough to evaluate this outcome. In ten reports, there was no evidence that any systematic attempt had been made to set the results of the new trials in context (Table 1).
Table 1
Table 1
Classification of Discussion sections in reports of RCTs published in May 1997, May 2001 and May 2005 in five general medical journals
There is no evidence of progress between 1997 and 2005 in the proportion of reports of trials published in general medical journals which discussed new results within the context of up-to-date systematic reviews of relevant evidence from other controlled trials. Although the proportion of trials referring to systematic reviews in Discussion sections has increased, the majority of reports continued to fail even to do this. Similarly, most researchers appear not to have considered a systematic review when designing their trial.
It is now a decade since the first CONSORT statement required that data from a new trial should be interpreted ‘in the light of the totality [emphasis added] of the available evidence’,6 and more than 40 years since Austin Bradford Hill suggested that the structure of a scientific paper could usefully be conceptualized in terms of four questions: ‘Why did you start? What did you do? What answer did you get? And what does it mean anyway?’.7 This is reflected in a common structure for scientific reports (sometimes referred to by the acronym ‘IMRaD’) comprising an ‘Introduction’; a description of the ‘Materials and Methods’; the ‘Results’; and a ‘Discussion’ of the findings. The Introduction should make clear why the study was worth starting and the Discussion should indicate the contribution of the new findings to the evidence available at the time of reporting.
The expectation that the results of a new randomized trial will be reported in the context of an up-to-date systematic review of earlier trials does not imply that the Discussion section of every report of a randomized trial should contain a full account of the material, methods and findings of such a review. The technology already exists to enable a brief review to be included in the Discussion section, with links to a relevant, up-to-date systematic review published elsewhere. And, now, with 3000 Cochrane reviews published in The Cochrane Library and protocols published for 1600 more, the availability and ready accessibility of systematic reviews has never been greater.
Randomized trials should be a key piece of the evidence for the public and others making decisions about health care, but to serve that purpose they need to be designed and reported in the light of other similar research. Systematic reviews provide the means to do this: they reduce unwanted duplication of research, help ensure that new research builds on lessons from earlier research, and place the findings of the new research in proper context.
In our three studies over the last eight years, we have repeatedly shown that the results of new randomized trials published in five general medical journals have not been presented within the context of updated systematic reviews of other relevant studies (Table 1). We have searched for but have not found any other reports of empirical research assessing the extent to which this issue has been addressed. We would welcome the conduct and reporting of such research.
In 2002, when we reported the second of our three studies, we wrote the following:
‘Because our expectations imply radical changes in the way that research is done and reported, we expect that not all researchers, journal editors, or publishers will agree with them. However, science is cumulative, and everyone, including the public, has a right to expect that this principle will be reflected more effectively in the way that science is conducted and reported. We feel that this imposes a duty on researchers to present their results in proper context and on journal editors to require them to do so.’4
A few years on, the findings we present here show that editors and authors—in these five high impact journals, at least—continue to fail to serve the needs of those who wish to use the results of randomized trials to make decisions about health care. Others have also shown how researchers designing new trials have not made proper use of systematic reviews, even where these are known to exist.8
Some years ago, the BMJ took the ground-breaking step of including a summary with each report of new research to show what is already known on a topic and what the new study adds. Unfortunately, this has not been followed by many other journals, nor extended to provide links to the evidence, such as systematic reviews, upon which these summaries are based. On a positive note, though, we are heartened to see that things might be changing in at least one of the journals we studied, with the announcement by the Lancet in mid 2005 that it will require all reports of clinical trials to include a clear summary of previous research findings and to explain how the new trial's findings affect this summary.9 Examples of this policy's implementation have appeared.10
Notes
Competing interests Work such as that reported in this paper relating to the quality of reporting research is relevant to the employment of all authors.
Funding There was no dedicated funding for this project.
Ethical approval This project did not require ethical approval. It did not involve the care of any people or animals.
Guarantor Mike Clarke.
Contributorship All authors were involved in the design, conduct and interpretation of this project and the preparation and approval of this manuscript. The manuscript was prepared within the context of the authors' employment by the NHS and the MRC, and is subject to Crown Copyright. The views expressed are not necessarily those of these organizations or The Cochrane Collaboration.
1. Clarke M. Doing new research? Don't forget the old. Nobody should do a trial without reviewing what is known. PLoS Medicine 2004;1: 100-2.
2. Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 1999;353: 490-3. [PubMed]
3. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 1998;280: 280-2. [PubMed]
4. Clarke M, Alderson P, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals. JAMA 2002;287: 2799-801. [PubMed]
5. Dalziel SR, Walker NK, Parag V, et al. Cardiovascular risk factors after antenatal exposure to betamethasone: 30-year follow-up of a randomised controlled trial. Lancet 2005;365: 1856-62. [PubMed]
6. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276: 637-9. [PubMed]
7. Hill AB. The reasons for writing. BMJ 1965;4: 870.
8. Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews when designing studies. Clinical Trials 2005;2: 260-4. [PubMed]
9. Young C, Horton R. Putting clinical trials into context. Lancet 2005;366: 107-8. [PubMed]
10. ESPRIT Study Group. Aspirin plus dipyridamole versus aspirin alone after cerebral ischaemia of arterial origin (ESPRIT): randomised controlled trial. Lancet 2006;367: 1665-73. [PubMed]
Bill-Axelson A, Holmberg L, Ruutu M, et al. Scandinavian Prostate Cancer Group Study No. 4. Radical prostatectomy versus watchful waiting in early prostate cancer. N Engl J Med 2005;352: 1977-84. [PubMed]
Brain EG, Bachelot T, Serin D, et al. RAPP-01 Trial Investigators. Life-threatening sepsis associated with adjuvant doxorubicin plus docetaxel for intermediate-risk breast cancer. JAMA 2005;293: 2367-71. [PubMed]
Ciliberto H, Ciliberto M, Briend A, Ashorn P, Bier D, Manary M. Antioxidant supplementation for the prevention of kwashiorkor in Malawian children: randomised, double blind, placebo controlled trial. BMJ 2005;330: 1109-11. [PMC free article] [PubMed]
Dalziel SR, Walker NK, Parag V, et al. Cardiovascular risk factors after antenatal exposure to betamethasone:30-year follow-up of a randomised controlled trial. Lancet 2005;365: 1856-62. [PubMed]
DuPont HL, Jiang ZD, Okhuysen PC, et al. A randomized, double-blind, placebo-controlled trial of rifaximin to prevent travelers' diarrhea. Ann Intern Med 2005;142: 805-12. [PubMed]
Fairbank J, Frost H, Wilson-MacDonald J, Yu LM, Barker K, Collins R. Spine Stabilisation Trial Group. Randomised controlled trial to compare surgical stabilisation of the lumbar spine with an intensive rehabilitation programme for patients with chronic low back pain: the MRC spine stabilisation trial. BMJ 2005;330: 1233-8. [PMC free article] [PubMed]
Gardner CD, Coulston A, Chatterjee L, Rigby A, Spiller G, Farquhar JW. The effect of a plant-based diet on plasma lipids in hypercholesterolemic adults:a randomized trial. Ann Intern Med 2005;142: 725-33. [PubMed]
Grant AM, Avenell A, Campbell MK, et al. for the RECORD Trial Group. Oral vitamin D3 and calcium for secondary prevention of low-trauma fractures in elderly people (Randomised Evaluation of Calcium or vitamin D, RECORD): a randomised placebo-controlled trial. Lancet 2005;365: 1621-8. [PubMed]
Guillou PJ, Quirke P, Thorpe H, et al. for the MRC CLASICC trial group. Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (MRC CLASICC trial): multicentre, randomised controlled trial. Lancet 2005;365: 1718-26. [PubMed]
Hakonarson H, Thorvaldsson S, Helgadottir A, et al. Effects of a 5-lipoxygenase-activating protein inhibitor on biomarkers associated with risk of myocardial infarction:a randomized trial. JAMA 2005;293: 2245-56. [PubMed]
Jokhio AH, Winter HR, Cheng KK. An intervention involving traditional birth attendants and perinatal and maternal mortality in Pakistan. N Engl J Med 2005;352: 2091-9. [PubMed]
Korzenik JR, Dieckgraefe BK, Valentine JF, Hausman DF, Gilbert MJ. Sargramostim in Crohn's Disease Study Group. Sargramostim for active Crohn's disease. N Engl J Med 2005;352: 2193-201. [PubMed]
Linde K, Streng A, Jurgens S, et al. Acupuncture for patients with migraine: a randomized controlled trial. JAMA 2005;293: 2118-25. [PubMed]
MacMillan HL, Thomas BH, Jamieson E, et al. Effectiveness of home visitation by public-health nurses in prevention of the recurrence of child physical abuse and neglect: a randomised controlled trial. Lancet 2005;365: 1786-93. [PubMed]
Penny ME, Creed-Kanashiro HM, Robert RC, Narro MR, Caulfield LE, Black RE. Effectiveness of an educational intervention delivered through the health services to improve nutrition in young children: a cluster-randomised controlled trial. Lancet 2005;365: 1863-72. [PubMed]
Singh BN, Singh SN, Reda DJ, et al. for the Sotalol Amiodarone Atrial Fibrillation Efficacy Trial (SAFE-T) Investigators. Amiodarone versus sotalol for atrial fibrillation. N Engl J Med 2005;352: 1861-72. [PubMed]
Smith AD, Cowan JO, Brassett KP, Herbison GP, Taylor DR. Use of exhaled nitric oxide measurements to guide treatment in chronic asthma. N Engl J Med 2005;352: 2163-73. [PubMed]
Valgimigli M, Percoco G, Malagutti P, et al. for the STRATEGY Investigators. Tirofiban and sirolimus-eluting stent versus abciximab and bare-metal stent for acute myocardial infarction: a randomized trial. JAMA 2005;293: 2109-17. [PubMed]
Articles from Journal of the Royal Society of Medicine are provided here courtesy of
Royal Society of Medicine Press