|Home | About | Journals | Submit | Contact Us | Français|
The online version of this article has been published under an open access model. Users are entitled to use, reproduce, disseminate, or display the open access version of this article for non-commercial purposes provided that: the original authorship is properly and fully attributed; the Journal and Oxford University Press are attributed as the original place of publication with the correct citation details given; if an article is subsequently reproduced or disseminated not in its entirety but only in part or as a derivative work this must be clearly indicated. For commercial re-use, please contact email@example.com
The Lancet of August 27, 2005 featured a cluster of articles highly critical of homeopathy which attracted considerable media attention. The media reports echoed The Lancet's press release: ‘homeopathy is no better than placebo’. The centerpiece was a meta-analysis of clinical trials of homeopathy compared with clinical trials of allopathy (conventional medicine) (1). The first author is Aijing Shang, but the leader of the research group is Prof. Matthias Egger of the Department of Social and Preventive Medicine, University of Berne, Switzerland.
The meta-analysis formed part of the Complementary Medicine Evaluation Programme (Programm Evaluation Komplementärmedizin, PEK) financed by the Swiss Federal government. The international review board of PEK has publicly protested at political interference in the scientific process: ‘There is a consensus among the review board members that the final PEK process deviated from what would have been expected by conventional standards. Especially disconcerting was the fact that the products of the PEK process—health technology assessment (HTA) reports, single description of studies, manuscripts for publication and the condensed final report—were sent to the board members but no discussion, comment, or review was solicited by the responsible agencies’ (2).
The meta-analysis was accompanied by a short, anonymous editorial entitled ‘The end of homoeopathy’ calling for ‘doctors to be bold and honest with their patients about homeopathy's lack of benefit, and with themselves about the failings of modern medicine’ (3); and a more thoughtful commentary from the Dutch epidemiologist Jan Vandenbroucke, reflecting on the ‘growth of truth’, including the relationship between bias, background knowledge and the concordance of clinical results with laboratory science findings (4). Vandenbroucke concludes that the proof of the pudding is in the eating: the ultimate proof of the validity of a scientific or medical idea is extent to which it changes reality.
The same issue of The Lancet featured a leak of the World Health Organisation's (WHO) draft report on homeopathy. The WHO document was apparently leaked to The Lancet by Dutch and Belgian doctors hostile to homeopathy; their comments and the (hostile) comments of Prof. Edzard Ernst of the University of Exeter were published. Dr Xiaorui Zhang, Traditional Medicine Coordinator of WHO, who is responsible for the report, was also interviewed, but declined to comment on a leaked, confidential draft. This leak came only 2 days after The Times of London published, as its front page lead, a remarkably similar story: a leak of the Smallwood Enquiry on The Role of Complementary and Alternative Medicine in the NHS commissioned by The Prince of Wales' Foundation for Integrated Health. It is ironic that the editor of The Lancet, Dr Richard Horton, wrote to The Times accusing Prof. Ernst of having ‘broken every code of scientific behaviour’ for leaking the draft report of the Smallwood Enquiry (and incidentally describing complementary medicine as ‘a largely pernicious influence… preying on the fears and uncertainties of the sick’), while simultaneously doing the same to the WHO report in his own journal!
Dr Horton also wrote an open letter to the UK Secretary of State for Health, Patricia Hewitt and the Chairman of the National Institute for Clinical Excellence (NICE) Prof. Sir Michael Rawlings, calling for the use of homeopathy in the NHS to be reviewed in light of this publication.
The meta-analysis at the centre of the controversy is based on 110 placebo-controlled clinical trials of homeopathy and 110 clinical trials of allopathy (conventional medicine), which are said to be matched. These were reduced to 21 trials of homeopathy and 9 of conventional medicine of ‘higher quality’ and further reduced to 8 and 6 trials, respectively, which were ‘larger, higher quality’. The final analysis which concluded that ‘the clinical effects of homoeopathy are placebo effects’ was based on just the eight ‘larger, higher quality’ clinical trials of homeopathy. The Lancet's press release did not mention this, instead giving the impression that the conclusions were based on all 110 trials.
The criteria for the matching of the homeopathic and conventional trials were not clearly stated, and it is evident from the numbers above that the clinical trials of homeopathy and conventional medicine were not, as claimed, well matched—the homeopathic trials were generally of better quality. And this is a crucial parameter [as Shang et al. (1) acknowledge]; it is well established that high quality trials are less likely to be positive than those of lower quality. Because of the individualization involved, it is difficult to do large-scale trials of homeopathy (again this is evident from the figures above), so the final sample is unlikely to be representative of trials of homeopathy. An example is a study that may have been included in the final eight, which looked at the use of a homeopathic medicine for prophylaxis of influenza, an indication for which it is not recommended, and obscured the identity of the homeopathic medication (5). However, a Cochrane Review has concluded that it is probably effective for treatment (as opposed to prevention) of flu-like illnesses (6). A number of similar instances could be cited, but since we do not know which studies are under discussion, there is little point.
One of the most serious criticisms is the complete lack of transparency: we have no idea which eight trials were included in the final, damning, analysis. The literature references are not given, nor any information on the diagnoses, numbers of patients, etc., nor can these be deduced from the article. Prof. Egger has refused several requests to disclose the identity of the eight trials. This is not even a matter of scientific method, but of natural justice: the accused has the right to know the evidence against him.
Meta-analysis should incorporate sensitivity analysis. In this case the obvious sensitivity analysis is to look at the 21 trials of ‘higher quality’, particularly since it appears that the criterion ‘larger’ appears to have been added retrospectively to ‘higher quality’. But the result of this analysis was not published.
This meta-analysis assessed quality purely in terms of internal validity. Internal validity is the extent to which a trial measures what it purports to measure; external validity the extent to which what it measures is meaningful in the real world. This is an important concept, best explained by an example. A clinical trial of homeopathy in childhood asthma by White et al. (7), published in Thorax in 2003, was reported as showing that homeopathy is ineffective. This trial was of high internal validity because, among other things, it used a predefined primary outcome measure: quality of life as measured by the childhood asthma questionnaire.
The results showed no difference in the quality of life scores, although the secondary outcome measures (such as severity and time lost from school), consistently favored homeopathy over placebo. However, because quality of life was the primary outcome, the authors reached a negative conclusion. But, as was pointed out several times in the ensuing correspondence, this conclusion was meaningless because of a ‘ceiling effect’: the children had normal quality of life when they entered the study, and this could not have been expected to improve further! (8–11) This study was of high internal validity but very low external validity, but would have been classified ‘high quality’ in The Lancet meta-analysis.
Shang et al. (1) state that ‘eight trials of homoeopathic remedies in acute infections of the upper respiratory tract … indicated a substantial beneficial effect … sensitivity analyses might suggest that there is robust evidence that the treatment under investigation works. However, the biases that are prevalent in these publications, as shown by our study, might promote the conclusion that the results cannot be trusted’. They state that eight studies is too few to question their conclusion about the whole set of publications. Their conclusion about the whole set, however, was also based on eight studies. Is eight enough for a conclusion or not? Or does it depend on what that conclusion is?
But perhaps the most telling single criticism of this meta-analysis is that it fails, on multiple counts, to meet the generally accepted standards for meta-analysis—the QUOROM statement (Quality of Reports of Meta-Analyses of Randomised Controlled Trials), published in The Lancet itself in 1999 (12). The main failings are those outlined above, although there are others.
The implausibility of homeopathy is an important Bayesian prior for Shang et al. (1), but they fail to quote emerging evidence for in vitro activity of ultramolecular dilutions (13,14) which has important implications for the implausibility of the claims made for homeopathy.
This meta-analysis is subject to fundamental criticisms. Regrettably, the media have already reported The Lancet's version of the story. Homeopathy's popularity is growing worldwide despite many such attacks. To paraphrase Mark Twain, reports of the death of homeopathy are much exaggerated: the facts simply are incompatible with The Lancet's claim that the end of homeopathy is nigh. Regrettably, this attack will only widen the divisions. The way forward is open, transparent science, not opaque, biased analysis and rhetoric.