Exercise, a component of many physiotherapy interventions, provides a case in point. Exercise science provides us with some insights into how the design of an exercise programme influences physiological responses to exercise. A dose-response relation exists, as with drugs. Physiological responses to exercise are determined by the mode, frequency, intensity, and duration of exercise.⁷ In practice, the dose is also determined by adherence to the exercise programme. The effects of exercise programmes observed in clinical trials are likely to vary because trials use different training doses and inspire different levels of adherence.⁸

An example of statistical heterogeneity that could be attributed to trials administering the interventions differently comes from studies of the effects of training the pelvic floor muscle. We identified four randomised trials of the effects of pelvic floor training to prevent urinary incontinence during pregnancy.^9–12 Three presented enough data to permit meta-analysis.^10–12 The studies were heterogeneous with respect to intervention. Two showed significant and clinically important effects of antenatal training,^11,12 whereas one study reported non-significant and clinically trivial effects.¹⁰ In the two trials with positive effects, training was supervised regularly by a physiotherapist, whereas in the study with negative effects women saw the physiotherapist only once.

The pooled estimate of effect obtained from a meta-analysis of all three trials did not show an effect of pelvic floor training on risk of urinary incontinence (odds ratio 0.67, 95% confidence interval 0.39 to 1.16; figure). When we excluded the large trial of a low intensity intervention¹⁰ from the meta-analysis, we found a clinically worthwhile effect of antenatal training (0.50, 0.34 to 0.75). The largest trial may have reported a smaller effect because of its size. Resource limitations often mean that large trials provide less intensive interventions, and in large trials it may be logistically difficult to provide well supervised interventions. Yet large trials are most heavily weighted in meta-analyses. If large studies with less intense interventions show smaller effects, they will tend to dilute the effects of smaller studies that show larger effects. An uncritical synthesis of these data suggests that the intervention is ineffective, but a more accurate interpretation may be that the intervention is effective only if administered intensively.

Formal assessment of the quality of the intervention in systematic reviews is potentially problematic. It will always be difficult to assess the quality of the intervention before it is known if the intervention is effective. However, the findings of other research can sometimes guide considerations of quality. When other research strongly suggests an intervention should be administered in a particular way, that research can be used as a benchmark for assessing the quality of intervention. For example, studies of the dose-responsiveness of strength training clearly indicate that strength training programmes produce the greatest increases in muscle strength when the training load is high.⁷ So systematic reviews of interventions designed to increase muscle strength should assess whether the training load was adequate.

Assessment of the quality of the intervention relies on sufficient detail in trial reports, but many reports provide only superficial descriptions of complex interventions. The CONSORT statement on reporting of clinical trials recommends that reports of clinical trials include “precise details of the interventions intended for each group and how and when they were actually administered.”¹³ Interventions should be described in sufficient detail to enable readers to assess if the intervention was administered well.

In some reviews of complex interventions it may be possible to formally incorporate considerations of the quality of the interventions into the analysis. At the simplest level, the systematic reviewer could separately analyse trials with high and low quality interventions—that is, stratify by intervention quality. A more sophisticated approach is to quantify interactions between measures of the quality of an intervention and its effect with meta-regression techniques.¹⁴

Whatever approach is used, the analysis is prone to statistical errors that are similar to the errors that arise in subgroup analysis of clinical trials.^15–17 Stratified analyses are more likely to miss real effects within strata (because each stratum is smaller than the set of all trials). They are also exposed to an increased risk of finding spurious effects within strata (because the risk of falsely significant findings accrues with each test of each stratum) and, consequently, they risk finding spurious differences in effects across strata.¹⁷ Several strategies can be used to minimise statistical errors. Most importantly, the strata that are to be tested should be specified in the review protocol before the review is started and the analysis should focus on the differences between effects across strata—that is, the analysis should examine interactions between the quality and effect of an intervention; claims of stratum specific effects should not be based on tests of effects within strata.

Even when reviewers take care to minimise statistical errors, the effects of intervention quality may still generate false conclusions. One reason is that such analyses almost always rely on comparisons between studies—that is, they compare the effects of an intervention in trials in which the intervention was administered well with those in trials in which the intervention was administered poorly. Such comparisons are prone to confounding because studies with high quality interventions may differ from studies with low quality interventions in other respects, such as the types of participants or their methodological quality. In the pelvic floor muscle example above, the study by Hughes et al¹⁰ provided the lowest intensity of intervention, but there were other differences between this study and the other two: the women who participated in the study by Reilly et al¹¹ had no symptoms at baseline, and fewer of the women in the study by Mørkved et al¹² had symptoms of urinary incontinence at baseline than in the study by Hughes et al. These factors, rather than the quality of intervention, might have produced the apparent differences in effect. Analyses based on comparisons between studies should always be considered exploratory rather than definitive.