The use of evidence-based approaches to therapeutic decision making can frequently raise the problem of how to make decisions when evidence is in limited supply. Often systematic reviews limit trial inclusion in an attempt to generate clinical homogeneity and allow sensible meta-analysis. The problem, though, is that other, useful, information is omitted. An example of this is the popular combination of paracetamol with codeine for treatment of acute and, more frequently, chronic pain. For the combination of 1000 mg paracetamol plus 60 mg codeine, for instance, there was information on only 127 patients in two placebo-controlled acute pain studies [1
Systematic reviews should seek unbiased evidence, which may limit the number of studies available for analysis. One approach to resolving the problem of apparently insufficient evidence may be to assess evidence of lower methodological quality. The amount of bias that could result from this approach would be a concern.
Schultz et al [3
] demonstrated that lack of randomisation is the major source of bias in trials; studies which are not randomised can lead to overestimation of treatment effects by up to 40%. Restricting systematic reviews to include only randomised studies therefore makes sense for reviews of effectiveness. A classic example is a review of transcutaneous nerve stimulation for post-operative pain relief. Randomised studies overwhelmingly showed no benefit over placebo, while non-randomised studies did show benefit [4
Non-blinded studies over-estimate treatment effects by about 17% [3
]. In a review of acupuncture for back pain [5
], the inclusion of both blinded and non-blinded studies changed the overall conclusion. The blinded studies showed 57% of patients improved with acupuncture and 50% with control, a non-significant relative benefit of 1.2 (95% confidence interval 0.9 to 1.5). Five non-blinded studies showed a difference from control, with 67% improved with acupuncture and 38% with control. Here the relative benefit was significant at 1.8 (1.3 to 2.4).
Trials of poor reporting quality consistently overestimate the effect of treatment. Using a validated scoring system for methodological quality [6
], studies of lower quality are likely to overestimate treatment effects [7
]. Other sources of bias may include small trials [9
], covert duplication [12
], and geography [13
]. Vickers and colleagues [13
] showed that trials of acupuncture conducted in east Asia were universally positive, while those conducted in Australasia, north America or western Europe were positive only about half the time. Randomised trials of therapies other than acupuncture conducted in China, Taiwan, Japan or Russia/USSR were also overwhelmingly positive.
There is also the issue of the overall validity of a randomised trial. In some areas, like acute pain, valid methods for the conduct of clinical studies have been set out for many years, and are well understood [14
]. There are therefore many trials that are randomised and double blind, and conducted on patients with the same initial severity of pain under similar conditions and assessing identical or similar outcomes over the same time periods. Trials with low validity are more likely to have a positive result than those with higher validity [15
], seen in acupuncture for head and neck pain.
It is obviously sensible to avoid bias where it is likely to occur. That means avoiding studies with designs or features where bias is possible. What other strategies remain when faced with apparently inadequate information from the literature? There are four that could be applied to the particular combination of paracetamol 1000 mg plus codeine 60 mg. These are:
1. The number of patients required for the number needed to treat (NNT) to be within ± 0.5 of the true value varies with the efficacy of an analgesic [11
]. We can therefore calculate how confident we can be in an NNT given the number of patients in the analysis, or calculate the number of patients needed to reach a required level of certainty.
2. We can assess evidence from other dose combinations of paracetamol and codeine and see whether there is a dose response relationship in placebo-controlled trials.
3. We can assess results for paracetamol 1000 mg plus codeine 60 mg against results for active comparators in high quality active controlled and placebo controlled studies. This can then be compared with systematic reviews done in similar clinical settings.
4. We can assess paracetamol 1000 mg plus codeine 60 mg in studies which, though randomised and double blind, had designs that did not allow their inclusion in a meta-analysis.
We applied these approaches to paracetamol 1000 mg plus codeine 60 mg through systematic review of the published literature.