What assumptions are being made in this analysis? One way to conceptualise this is to imagine that all 36 trials had examined all seven treatments but that in each trial results for all but two or three treatments had been lost at random. The key assumption for the fixed effect analysis is that the relative effect of one treatment compared with another is the same across the entire set of trials.3,4
This means that the true odds ratio comparing A with B in trials of A v
B is exactly the same as the true odds ratio for A v
Binthe A v
C, B v
C, and indeed E v
F trials, even though A and B were not included in those studies. In a random effects model, where it is assumed that the odds ratios in each trial are different but from a single common distribution, the assumption is that this common distribution is the same across all sets of trials.
These assumptions are remarkably similar to those that underlie a standard pair-wise meta-analysis. The only additional assumption is that the similarity of the relative effects of treatment holds across the entire set of trials, irrespective of which treatments were actually evaluated. Indeed, one way to question this assumption is to imagine all trials had compared the same two treatments and to judge whether they are sufficiently similar to be combined in a meta-analysis.
It may be helpful to consider how the target population relates to the patient groups in the studies. If we are attempting to find the single best treatment with a view to recommending it to all the various patient groups represented in previous trials, a single combined analysis is clearly indicated. If our target is a specific subgroup of patients who were represented in studies around a specific set of treatment comparisons, it may be more appropriate to consider restricting the analysis to a subset of the trials.
The assumptions behind comparisons of multiple treatments are unlikely to be statistically verifiable, and it seems reasonable to rely on expert clinical and epidemiological judgment, both for multiple treatment comparison and for standard pair-wise meta-analysis. This is, after all, the criticism that clinicians often make of meta-analyses—that like is not being compared with like. Multiple treatment meta-analysis is no different from pair-wise meta-analysis in requiring such a judgment. Poor judgments in both cases may induce heterogeneity of effects. Poor judgments in meta-analyses of multiple treatment comparisons may, in common with subgroup analyses and meta-regression of pair-wise comparisons, lead to confounding (if, for example, trials of A v B and A v C are systematically different from trials of B v C). However, bias would not be expected generally to operate in any particular direction.
A further assumption relates to scale of measurement. In common with studies based on indirect comparisons,3-5,20
multiple comparison models are based on the assumption that treatment effects add together so that the relative effect of A v
C can be predicted from the effects of A v
B and B v
C. This assumes that the appropriate measure of effect (log odds ratios, relative risk, or risk difference) has been chosen.21
Healthcare decisions often involve choosing from a selection of treatment options
Most systematic reviews and meta-analyses focus on pair-wise comparisons, forcing reliance on indirect comparisons
Statistical methods for comparing multiple treatments that combine direct and indirect evidence in a single analysis are available
These methods make the similar assumptions to standard pair-wise meta-analyses but require that they hold over the entire set of trials
Multiple treatment comparisons should be more frequently used to inform healthcare decisions
Both of the pair-wise meta-analyses described earlier were built on the above assumption.15,16
In our multiple analysis we made the further assumption that the two sets of studies are homogeneous regarding the relative effects of all the treatments, or (in the random effects analysis) that the relative effects are from the same common distribution. This was generally supported by the similarity between the direct evidence and the estimates obtained by combining direct and indirect evidence. This observation has been made in previous analyses.4
Another common feature of our proposed multiple comparison methods and standard pair-wise meta-analyses is a lack of any assumptions about baseline risks across studies. Thus, our analyses are based only on randomised comparisons. Methods that compare patient outcomes in one treatment arm of one trial directly with those in a treatment arm in another trial break randomisation and have been rightly criticised.6