The different analysis methods (CC, LOCF, MM and MI) applied to the FREE trial data produced similar results, with only minor differences in standard error sizes observed between the groups. This suggests that the FREE trial data are not dependent on the ITT analysis method used. Indeed, the information contained within the missing data would seem to be of a similar nature to the actual documented information. This implies that the conclusions made about the efficacy of BKP treatment in the FREE trial are robust.
The literature shows that use of the ITT principle in orthopaedic randomized clinical trials (RCT) is limited; in a survey of eight leading orthopaedic journals only 35% of RCT used the ITT method [5
]. Thus, lack of ITT use could be a potential source of bias for a comparatively large proportion of orthopaedic RCT. Similar investigations have previously been performed for the general medical literature: an analysis of four leading medical journals demonstrated that 119 out of 249 (48%) RCT published in 1997 used the ITT principle [13
], while a later analysis of ten medical journals showed that 249 out of 403 (62%) RCT published in 2002 used the ITT method [14
]. Therefore, while the ITT principle appears to be more commonly used in general medical RCT compared with orthopaedic RCT, there is still scope for improvement.
In the FREE study, the missing response level was 19% for the entire 24-month follow-up period. This proportion was similar to that observed for other long-term studies; in a survey of RCT published in eight orthopaedic journals the mean rate of patients lost to follow up was 17% for thirty studies with a follow-up period longer than 1 year [5
]. Indeed, increased missing response levels in RCT may be expected over longer time periods; in the same survey of orthopaedic RCT a significant increase in the proportion of patients with missing data was observed over longer follow-up periods [5
While each imputation method provided similar results, there are certain characteristics related to each analysis method that need to be acknowledged. For example, the CC analysis (which is at variance with the ITT principle) is generally considered insufficient for evaluating data from clinical trials. Data can be missing in a sporadic manner across several different covariates and this can lead to the omission of a high proportion of patients for a CC analysis. Indeed, in the FREE trial almost half of all randomised patients were not available for the CC analysis. It is possible that only including patients with complete information may not provide a representative randomly selected subset of the total randomised population. The higher MSE for the CC analysis can also be interpreted as a sign of bias in the treatment effect estimate. However, in the FREE study the results obtained for the CC analysis were reasonably consistent with the MM, not being an imputation method, and MI suggesting those patients with complete information were representative of the randomised population.
The Cochrane Musculoskeletal Group recommends that imputations based on methods such as LOCF are acceptable in both 'platinum' and 'gold' level publications included in Cochrane systematic reviews [15
]. Implicit in the LOCF method is the assumption that the outcome remains constant from the last observed value after drop out and that no measurement errors exist, otherwise there is a risk for bias particularly over longer follow-up periods. In the FREE trial treatment effects were actually observed to change during the follow-up period [7
]. At 1 month follow-up, patients treated with BKP showed significantly (p < 0.0001) greater improvements in SF-36 PCS score compared with those who received non-surgical treatment. However, at 12-months follow-up the difference between the two groups had diminished likely due to fracture healing in patients who had received non-surgical treatment. Therefore, use of LOCF to analyze the FREE trial data could be a potential source of bias. However, the present analysis showed that the LOCF method produced similar results to the MM and MI methods suggesting that any potential effects on bias would likely be minimal.
The MI method does not have the disadvantages associated with single imputation methods: variance is not underestimated and treatment effect changes are preserved over time. The MI method does rely to a great extent on the imputation model being correct, but providing the data are normally distributed or can be transformed to normality there is a sound theoretical background to how the imputations are generated [16
]. However, if some of the imputed variables cannot be treated as normally distributed then other MI methods, such as chained equations, need to be used. The chained equations method has produced accurate imputations in various settings [9
], but the overall theory that proves its correctness is currently being developed. The inclusion of more variables in the imputation model than in the analysis model increases the probability that MAR assumptions hold, although this may lead to an increase in the standard errors of the final estimates [16
]. In case the imputation model and the analysis models differ it is crucial to ensure that the models are congenial, i.e. that all variables used in the analysis model are included in the imputation model [17
When surgical intervention in one group is compared with a non-surgical treatment in another, the proportions of patients lost to follow up could be expected to differ between the groups. The phenomenon has been described earlier [5
], and it occurs also in this trial. The higher rate of missing data in the control group may undermine a MAR assumption, which would affect both the MM and CC analysis. However, if the MAR assumption holds for the analysis model used then MI should converge to MM as the number of imputations goes to infinity [18
]. For the FREE data there were no substantial differences observed between the MM and MI analyses despite the fact that the imputation model included more variables than the analysis model. This indicates that the MAR assumption holds for the analysis model and that the MM analysis method appears to be optimal.
In the present analysis of the FREE data, the MM method showed the highest statistical precision without the unsubstantiated assumptions required for LOCF. This was particularly evident when the three methods (LOCF, MM and MI) were used to examine the percentage reduction in standard error compared with CC for overall treatment effect. Only MM provided a reduction in variance across all six outcomes measured (including 'days in bed'). The MI method also provided low variance but was not as precise as the MM method across all outcomes considered. This suggests that application of the MM method is probably the most accurate approach for analysing these data. This suggestion is also supported by a simulation study [18