PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of bmjLink to Publisher's site
 
BMJ. 2004 October 16; 329(7471): 868–869.
PMCID: PMC523097

The scandal of poor epidemiological research

Reporting guidelines are needed for observational epidemiology
Erik von Elm, research fellow
(vonelm/at/ispm.unibe.ch)
Department of Social and Preventive Medicine, University of Berne, Finkenhubelweg 11, CH-3012 Berne, Switzerland
Matthias Egger, professor

Something surely must be wrong with epidemiology when the new editors of a leading journal in the field entitle their inaugural offering, “Epidemiology—is it time to call it a day?”1 Observational epidemiology has not had a good press in recent years. Conflicting results from epidemiological studies of the risks of daily life, such as coffee, hair dye, or hormones, are frequently and eagerly reported in the popular press, providing a constant source of anxiety for the public.2,3 In many cases deeply held beliefs, given credibility by numerous observational studies over long periods of time, are challenged only when contradicted by randomised trials. In the most recent example, a Cochrane review of randomised trials shows that antioxidant vitamins do not prevent gastrointestinal cancer and may even increase all cause mortality.4,5

Now Pocock et al describe the quality and the litany of problems of 73 epidemiological studies published in January 2001 in general medical and specialist journals (p 883).6 Perhaps the most relevant findings relate to how investigators dealt with confounding, multiple comparisons, and subgroup analyses. Confounding, the situation in which an apparent effect of an exposure on risk is explained by its association with other factors, is probably the most important cause of spurious associations in observational epidemiology. For example, a recent meta-analysis of observational studies shows how confounding could have been responsible for the belief that hormone replacement therapy provides protection against cardiovascular disease.7 A protective effect of hormone replacement therapy was evident in studies that did not control for socioeconomic status, but not in studies that did (figure).7 Higher socioeconomic position is strongly associated with both more frequent use of hormone replacement therapy and lower risk of coronary heart disease. In the large (unconfounded) Women's Health Initiative randomised trial hormone replacement therapy had no beneficial effect on cardiovascular disease.8

Figure 1
Meta-analysis of cohort studies and case-control studies of hormone replacement therapy and coronary heart disease. There is little evidence for a protective effect when analyses are adjusted for, in contrast to studies not adjusted for, socioeconomic ...

Worryingly, Pocock et al find that the rationale behind the choice of confounders is usually unclear, and that the extent of adjustment varies greatly. They also confirm that observational studies often consider several exposures, outcomes, and subgroups. This results in multiple statistical tests of hypotheses and a high probability of finding associations that are statistically significant but spurious. In many studies 20% or more of the findings may be erroneous, rather than the expected 5% false positive associations (P < 0.05).9

Epidemiology is not alone: and there is hope. As Altman pointed out 10 years ago, much medical research is of poor quality.10 Efforts so far to remedy what Altman described as the “scandal of poor medical research” have focused on controlled clinical trials. Empirical research has shed light on aspects of methodological quality that are crucial to prevent bias,11 and Consolidated Standards for Reporting Trials (CONSORT) have been developed and endorsed widely.12

Clearly, such guidelines are also required for observational epidemiology. A month ago a group of epidemiologists, statisticians, and editors met in Bristol to work on a first draft of STROBE, the Standards for the Reporting of Observational Studies in Epidemiology. STROBE will provide guidance on the reporting of cohort studies, case-control studies, and cross sectional studies; a supporting document will give explanations and examples. Outcomes from the Bristol workshop will be posted on a dedicated website (www.strobe-statement.org), and you and everyone else will be invited to comment and suggest improvements before a revised version is published some time next year.

Can STROBE do for observational epidemiology what CONSORT does for randomised clinical trials? Not exactly. Both guidelines aim to promote comprehensive reporting of important methodological detail and facilitate appraisal of study quality. In the case of a large high quality randomised trial, this means that the results can be assumed to provide an unbiased estimate of the treatment effect in the population studied. This is not so in observational epidemiology. A well conducted case-control or cohort study might still produce misleading results if, for example, important confounders were not known, not measured, or imprecisely measured.

Importantly for observational studies, STROBE will also pay considerable attention to what investigators should write in the discussion section of their paper by suggesting the inclusion of statements on why methodological approaches were chosen, and why results are interpreted the way they are. The skilful interpretation of epidemiological evidence, bearing in mind theoretical considerations, resisting being seduced by possible mechanisms, and suspending beliefs to allow healthy scepticism will remain the great challenge and joy of working in epidemiology.8 More transparent and complete reporting of epidemiological studies, coupled with more thoughtful interpretation of their results, will help restore the reputation of a discipline that has contributed importantly to improving the health of the public, and will continue to do so in the future.

Notes

Papers p 883

Competing interests: EvE and ME are members of the STROBE group.

References

1. Davey Smith G, Ebrahim S. Epidemiology—is it time to call it a day? Int J Epidemiol 2001;30: 1-11. [PubMed]
2. Taubes G. Epidemiology faces its limits. Science 1995;269: 164-9. [PubMed]
3. Bartlett C, Sterne J, Egger M. What is newsworthy? Longitudinal study of the reporting of medical research in two British newspapers. BMJ 2002;325: 81-4. [PMC free article] [PubMed]
4. Bjelakovic G, Nikolova D, Simonetti RG, Gluud C. Antioxidant supplements for prevention of gastrointestinal cancers: a systematic review and meta-analysis. Lancet 2004;364: 1219-28. [PubMed]
5. Lawlor DA, Davey SG, Kundu D, Bruckdorfer KR, Ebrahim S. Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence? Lancet 2004;363: 1724-7. [PubMed]
6. Pocock SJ, Collier TJ, Dandreo KJ, de Stavola BL, Goldman MB, Kalish LA, Kasten LE, McCormack VA. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 2004;329: 883-7. [PMC free article] [PubMed]
7. Humphrey LL, Chan BK, Sox HC. Postmenopausal hormone replacement therapy and the primary prevention of cardiovascular disease. Ann Intern Med 2002;137: 273-84. [PubMed]
8. Petitti D. Commentary: hormone replacement therapy and coronary heart disease: four lessons. Int J Epidemiol 2004;33: 461-3. [PubMed]
9. Ottenbacher KJ. Quantitative evaluation of multiplicity in epidemiology and public health research. Am J Epidemiol 1998;147: 615-9. [PubMed]
10. Altman DG. The scandal of poor medical research. BMJ 1994;308: 283-4. [PMC free article] [PubMed]
11. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001;323: 42-6. [PMC free article] [PubMed]
12. Moher D, Schulz KF, Altman DG, for the CONSORT Group. The CON SORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. JAMA 2001;285: 1987-91. [PubMed]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Group