Substantive criticism was raised against about one third of research papers in the BMJ
over the two year period from October 2005 to September 2007. This may seem surprising, as the BMJ
is a high impact factor general journal with an acceptance rate of submitted manuscripts that went down from 4% to 2% in the period we studied.6
However, this finding possibly reflects that it is easy to submit a rapid response and have it published in the BMJ
, in contrast with most other journals that publish only a few selected letters in their print issues.7
As the perfect scientific study does not exist, we believe this type of post publication peer review is a valuable asset that other journals should consider implementing. A huge advantage of electronic publishing is that letters can appear the next day, whereas it usually takes months before letters become available in print journals. The drawback of the online response system is that some papers receive so much feedback that important criticism may be difficult to find.
Although we defined substantive criticism as a problem that could invalidate research or reduce its reliability, the authors responded to it in only about half of the cases. In contrast with what we had expected, the criticism was not more severe in those cases where the authors did not reply. Furthermore, for those criticisms that had replies, the replies were not less adequate when the criticism was severe.
It was also unexpected that editors differed noticeably from the critics in their assessment of the adequacy of authors’ replies. The editors were much more prone to accept a reply as being adequate than the critics. It is unclear who should be trusted. On the one hand, editors might be too positive, in a subconscious attempt to protect their own work and the journal’s prestige, whereas critics who bother to communicate their concerns might not be typical of the average reader and less likely to be satisfied whatever the replies. On the other hand, critics on average would be expected to be more knowledgeable about the subject and the methodological issues related to the research than the editors. The criticisms raised were generally appropriate, and the replies less so, and we therefore think that the critics’ assessments are likely to be closer to the truth than those of the editors. Criticisms were, on average, only partly addressed (table 3).
Limitations of the study
The web based rapid response facility in the BMJ
makes it easy and quick to post criticisms, and practically all submitted letters are accepted. Many more letters are therefore published in the BMJ
than in journals without online response systems: the mean and median number of responses for all 350 research papers in our study were 4.9 and 3, respectively, whereas a survey showed that the mean in print issues in 2007 was only 1.2 for BMJ
, 1.0 for JAMA
, 1.3 for Lancet
, 2.1 for New England Journal of Medicine
, and 0.6 for Annals of Internal Medicine
It is therefore not possible to extrapolate our results directly to other journals, although it is likely that important criticism could be raised equally often in other high impact journals, and probably much more often in other journals.
To investigate this further, we identified letters in the print version of the BMJ using PubMed and the BMJ website. We found that 20 of the 105 rapid responses with substantive criticism (19%) were also published in the print issue and five of these (25%) had a reply from the author in the print issue. The severity of the 20 criticisms in the print issue was similar to that of the 85 criticisms that were only published online (2.05 v 2.25, P=0.12). This is reassuring, although one might have expected the severity to be higher, as editors would generally select the harshest criticisms to give them the widest possible attention.
Our checks also showed that only one study had an author’s reply in the print version of the journal but no reply in the online rapid response section. In some cases, rapid responses are not posted in conjunction with the article itself on the BMJ website but in relation to commentaries or editorials describing the study. Our strategy for identifying studies could therefore have missed some criticisms and underestimated prevalence, although we judge the problem to be minor.
The corresponding author of every BMJ article receives an automated email when the paper is published and an automated reminder whenever a rapid response is posted. Authors are not told that responding is a requirement and not all rapid responses need a response. We, however, believe authors have an academic duty to respond to substantive criticism of their work and therefore cannot be excused because they were not asked to do so by an editor.
Implications for editors
Editors should encourage authors to respond adequately—for example, by making a contract between the author and journal on acceptance of the article, alerting authors to substantive criticisms, and highlighting substantive criticisms that have not been responded to adequately. Editors should post an editorial note if, despite requests for a proper response, the authors refuse to address criticism that could invalidate the paper or important parts of it. As editors may lack sufficient knowledge about the particular research area or the methods used, they should also consider sending substantive criticisms and authors’ replies for peer review, and it could be particularly valuable to use the critics as peer reviewers of authors’ replies. Finally, editors should consider not having time limits for letters that highlight important flaws.1
Science has no “use before” date but evolves through open debate. Editors of journals that do not have online response systems should consider introducing these to improve post publication peer review of their papers, as we found that only one fifth of rapid responses with substantive criticisms were published in the print journal.
Implications for readers
Readers should be aware that flaws in medical research are common and that the peer review system cannot ensure that flawed studies do not get published, even in the best journals. If a paper carries an important message, it is prudent to look for subsequent letters in the same journal and for commentaries in other journals. The journal Trials
, for example, contributes to correcting the scientific record by welcoming submissions that discuss trials published elsewhere, and there is no time limit for such commentaries. Sometimes the flaws are so well hidden that it requires time consuming detective work to unravel them.8 9 10
As only a minor part of substantive criticism is published in the print version of a journal and as only these letters are indexed in PubMed, readers should also check out whether online responses have been posted to papers in journals with this possibility. Conversely, papers may seem more flawed than they really are, as relevant authors’ replies may have been only published online.
Implications for authors
Authors who consider quoting research papers should be careful. Flawed results and conclusions are often propagated in subsequent papers by authors who might have only read the abstract of the papers they quote, or even only the abstract’s conclusion.
What is already known on this topic
- Letters to the editor about research papers serve a useful role as post publication peer review and for advancement of science
What this study adds
- Substantive criticism was raised against a third of research papers published in the BMJ yet authors responded to the criticisms in only half of these cases
- No relation was found between the severity of the criticism and the adequacy of the reply, neither as judged by the editors nor by the critics
- The critics were much more critical of the replies than the editors