PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jrsocmedLink to Publisher's site
 
J R Soc Med. 2007 January; 100(1): 38–39.
PMCID: PMC1761683

Bufala spotting, part one: assessing research papers

QUICK INSTRUMENT (FIGURE 1)

Figure 1
Quick pathway to follow to minimize the risk of being taken for a bufala ride when scanning an original research paper. Reproduced with permission from Jefferson T. Attenti Alle Bufale. Rome: Pensiero Scientifico Editore, Second edition 2006

If the topic interests you, go to the bottom of the Introduction section. If you cannot find a clearly defined objective, bin the study. Think twice about reading the journal again. If the study has a clear objective, go to a results table, add up a column at random and match the results with the total. If they do not match, bin the paper. Remember that if you are adding percentages, the total may be slightly over or under 100% because of rounding. If you have a little more time, read the study backwards, starting with the Discussion. If the bits do not fit logically, bin the study. Alternatively, match the content of the abstract (the shop window of the paper) with the content of the text. Again, bin the study if they do not fit. Crazy? Some wag found errors and discrepancies in 68% of studies in one sample1.

FULL INSTRUMENT

There are some very useful full instruments to help in this case. The CONSORT statement is perhaps the most famous (http://www.consort-statement.org/Statement/revisedstatement.htm). CONSORT was developed to make reporting of randomized trials clearer, but it is a useful checklist to guide you through the intricacies of trials2. There is also a separate statement covering cluster-randomized trials.3

For cohorts and case-control studies, use the Newcastle-Ottawa scale, or NOS (http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm). The NOS takes a while to get used to but it is a good, relatively quick instrument to use4.

To assess the quality of diagnotic studies try using the QUADAS instrument (http://www.biomedcentral.com/1471-2288/3/25)5

The rules for their reporting are addressed in the STARD instrument (http://www.consort-statement.org/Initiatives/newstard.htm).6

RATIONALE

Research papers should be written according to IMRAD (Introduction, Methods, Results and Discussion). They should accurately reflect what the authors intended doing, what really did happen and what they found. However, the majority of research papers one reads while undertaking a systematic review are unreliable for a variety of reasons, regardless of who wrote them or where they were published. The quick instrument is the fastest way of spotting bufale lurking between the folds of the paper. Also beware of perfectly written, immaculate papers, especially those reporting large trials. It is likely that these have had the input of one or more ghost authors—in other words, that they are marketing instruments of one kind or another.

This apparently conflicting advice (bin what is badly written and beware of what is written too well) demonstrates how difficult it can be to spot these bufale. The golden rule is beware at all times. If something appears too good or too bad to be true, it probably is.

Notes

This is the second in a series of articles on making evidence-based medicine work for you. The series is based on the book ‘Attenti Alle Bufale’ by Tom Jefferson (www.attentiallebufale.it).

References

1. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA 1999;281: 1110-1 [PubMed]
2. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomised trials. Lancet 2001;357: 1191-4 [PubMed]
3. Campbell MK, Elbourne DR, Altman DG, for the CONSORT Group. CONSORT statement: extension to cluster randomised trials. BMJ 2004;328: 702-8 [PMC free article] [PubMed]
4. Wells GA,Shea B, O'Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of non randomised studies in meta-analyses. http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm
5. Whiting P, Utjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Leijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 2004;8: 59-65 [PubMed]
6. Bossuyt P, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003;49: 7-18 [PubMed]

Articles from Journal of the Royal Society of Medicine are provided here courtesy of Royal Society of Medicine Press