Indirect comparison is being increasingly used for the evaluation of a wide range of healthcare interventions. In this study, 16 of the 88 included reviews were health technology assessment reports. In many such reports, indirect comparison had not been done for clinical effectiveness but was used in the economic evaluation.
In the literature, several related but different assumptions underlying adjusted indirect comparison (figure) have not been clearly distinguished, resulting in methodological and practical problems in the interpretation of indirect or mixed treatment comparison. The problems include unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence.
Indirect comparison was explicit but informal in 13 reviews—neither relative effects nor statistical significance were calculated. Since the use of indirect comparison is often inevitable, a more explicit and formal approach is preferable. In six reviews, the results from individual arms of different trials were compared naively as if they were from one controlled trial. This approach is flawed because the strength of randomisation is disregarded.2
The strength of randomisation could be preserved in adjusted indirect comparison. The most common scenario was the indirect comparison of two competing interventions adjusted by common comparators using classic frequentist methods (including simple metaregression). The advantages of the simple methods include ease of use and transparency. However, when there are several alternative interventions to be compared, the simple adjusted indirect comparison may become inconvenient. More complex methods, such as network meta-analysis, are being increasingly used to make simultaneous comparisons of multiple interventions.10 12 13
These methods treat all included interventions equally rather than focusing on one particular comparison of two interventions.
Subgroup analysis and metaregression are commonly used to assess or improve trial similarity for adjusted indirect comparison (see bmj.com). Their usefulness may be limited because the number of trials involved in adjusted indirect comparison was usually small and it was uncertain whether the important study level variables were reported in all relevant trials.
Trial similarity was often assessed by examining heterogeneity across trials and by a narrative comparison of trial characteristics for the different treatment comparisons being included, which may be deemed informal and subjective.
When data from head to head comparison trials are available, consideration needs to be given to whether an indirect comparison is justified when direct comparison trials are available; any discrepancies between direct and indirect evidence need to be sensibly interpreted; and could direct evidence be combined with the results of indirect comparison.
It is controversial whether indirect evidence needs to be considered when there is evidence from direct comparison trials.5 9
Indirect comparison was considered helpful by authors of the 40 reviews in which both direct and indirect evidence were available. Such evidence was less likely to be explicitly compared and more likely to be combined in reviews that used complex rather than simple methods (see bmj.com). Since the evidence consistency is usually assessed informally and subjectively,9
transparency is important to allow others to make their own judgment.
Reviews may include trials with three or more arms. Some reviews separately compared two active treatments with placebo within the same trial, and then the results of two separate comparisons were used in adjusted indirect comparison. This downgrades direct evidence to indirect evidence, reduces precision, and uses data from the same placebo arm twice.
In nine reviews, direct comparison trials were excluded or not searched for systematically. In reviews that included only placebo controlled trials, it was often unclear whether there were other active treatment controlled trials that could also be used for adjusted indirect comparison. Some indirect comparisons seemed to be done on an ad hoc basis, using data from existing systematic reviews and meta-analyses.
Reviews were included in this survey only if the indirect comparison was explicit in their titles and abstracts, and if they were indexed in PubMed. Thus we may have missed reports with indirect comparisons. Missed reviews may have been less explicit and less formal than included ones, therefore not mentioned in the abstract.
Empirical evidence on the validity of indirect and mixed treatment comparison is still limited and many questions remain unanswered. In addition, there is only limited empirical evidence to show that improved trial similarity is associated with improved validity of indirect and mixed treatment comparison.
What is already known on this topic
- Indirect comparisons can be valid if some basic assumptions are fulfilled
- The related but different methodological assumptions have not been clearly distinguished
What this study adds
- Certain methodological problems may invalidate the results of evaluations using indirect comparison approaches
- Understanding basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these problems
- A framework can help clarify homogeneity, similarity, and consistency assumptions underlying adjusted indirect comparisons