The present study demonstrates the use of risk-benefit analysis to perform indirect comparisons of multiple competing interventions. We believe that this approach might help to guide clinical decisions in areas where tension between risk and benefit exist and there are multiple therapeutic options. This approach provides a way to indirectly compare several interventions in the absence of randomized trials involving multiple arms.
This method assumes that the use of certain therapy has a clinical risk, something particularly important when potentially harmful therapies are used. Clinicians opt for these options because the expected benefit outweighs the risk; however, choosing the best therapeutic option can be difficult when the rates of major complications (risk) or clinical effectiveness (benefits) from different options vary. Furthermore, clinicians' willingness-to-accept the risks might differ depending on a number of issues. If for a clinical situation there are numerous therapeutic options it is unlikely that all options will be compared in a randomized trial usually because of sample size and research cost constraints. The approach described herein takes advantage of the risk-benefit analysis framework and provides a way to indirectly compare several interventions in order to determine the one with the best risk-benefit profile using information available from randomized trials to estimate costs and benefits.
Using the proposed approach we conducted an indirect comparison of five common anticoagulant drugs used to prevent the development of thrombotic complications in patients undergoing major orthopedic surgery. This intervention is particularly suited to exemplify this approach given that anticoagulants are associated with bleeding episodes as their major (and almost only) complication and with a palpable benefit, namely prevention of thrombosis. Furthermore, they are usually given for a short period of time which facilitates defining the time horizon for the study. The analysis showed that compared to placebo, all agents are likely risk-beneficial (Figures ), a finding that is not surprising. The problem is then choosing the one(s) with the best risk-benefit profile. By calculating the net clinical benefit for each anticoagulant at a particular value of the RBAT it can be easily determined the probability of obtaining the highest net clinical benefit for each competing anticoagulant at that particular RBAT value. These probabilities are then calculated for a range of RBAT values and used to create a net clinical benefit probability curve (Figures ). Furthermore, since the ultimate clinical consequences of major bleeding and venous thromboembolic events might be different (i.e. their associated mortality), we incorporated in the analysis a reference range of RBAT values derived from the case fatality rate-ratios of thrombosis and bleeding. The final plots then incorporate the key elements of a decision, namely risk, benefit, willingness to accept the risk and their uncertainties.
It can be argued that when analyzing and pooling information obtained from randomized trials, the use of proportions for comparison purposes -as opposed to effect sizes- supposes the loss of the randomization effect because groups are treated independently. Nevertheless, if properly conducted and reported, information derived from randomized trials has usually a high quality [30
], and although we recognize that there might be a concern regarding the generalizability of the results from randomized trials, pooled estimates of event proportions will most likely approach reality as the population in the included studies increases. Although non-randomized studies could potentially also be used we prefer randomized trials which are less prone to bias. However, since a number of issues could influence meta-analysis results, systematic reviews should be methodologically sound and incorporate a priori all pertinent subgroup and sensitivity analyses, an evaluation of the homogeneity of outcomes' definitions used across different studies, and an assessment of study quality using validated scales [25
]. This issue can be better appreciated in our study by analyzing the results for all patients and the subgroup analyses which showed that the risk-benefit profiles of the different anticoagulants were different in patients undergoing hip or knee replacement
Some conditions are necessary to conduct study using the approach proposed herein: First, outcomes used to measure risks and benefits must be defined similarly across studies. Incorporating studies using different outcome definitions would result in a heterogeneous result difficult to interpret. It is entirely possible that the results of the study example could be different if all and each one of the potential side effects of the drugs were incorporated in the analysis. Second, if clinically acceptable, risks and benefits should ideally be defined by a single outcome; if this is not possible then a composite outcome could be used, such as mortality. Third, events defining risks and benefits should be clinically relevant. Fourth, if the systematic review spans several years, it is particularly important to test for secular trends since outcome definitions might change over time as a result of new knowledge, which might affect the evaluation of older therapies still in use. In addition, changes in medical or surgical techniques or in health policy are likely to influence the results and therefore populations might not be similar. It can be appreciated that the systematic review must be stringently rigorous and adhere to standardized requirements such as the PRISMA (formerly QUOROM) statement and exploring heterogeneity becomes essential. A thorough sensitivity analysis should be conducted prior to incorporating the information into the Monte Carlo simulation, to assure to the maximum extent that studies include comparable populations. If these conditions are met, it can be argued that this approach provides a pragmatic panorama of the situation studied.
Our approach does not estimate effect sizes; instead it provides the clinician with information regarding the probability of having the best risk-benefit profile that certain intervention will have at a given value of risk acceptance. If risks and benefits can be measured in equivalent terms (e.g. deaths induced or prevented by the treatment), the preferred choice should be that conferring the highest probability of being beneficial at the chosen value for risk acceptance. However, if surrogate endpoints are used case fatality rates might be different and so will be the clinical relevance of costs and benefits relative to one another. Our method allows for an adjustment by changing the value of the RBAT to one that better suits the clinical situation being studied. Since the level of risk acceptance is pivotal to making the best choice, a reference value for the RBAT can be obtained from expert consensus or surveys, or by using case-fatality rate ratios. In any case a reference RBAT should be a guideline for the clinician to be applied on an individual basis.
It is important to note that randomized trials are usually powered to detect differences in benefit-defining events and inadequately powered to detect differences in risk and only some of them include risk-defining events in a composite outcome. In this regard, an advantage of our approach is that it allows larger sample sizes; hence more power to detect differences in less common, more clinically meaningful outcomes.
The potential limitations of our approach arise from three issues: 1) the fact that the information is obtained retrospectively from a systematic review and meta-analysis with their inherent caveats; 2) the methodological issues regarding the pooling of single proportions; and 3) the limitations regarding modeling techniques and their application to clinical risk-benefit ratio and incremental risk-benefit ratio analysis. With respect to the first point we considered that an ample sensitivity analysis should establish the robustness of the conclusions. If the conclusions are not robust, the conduction of a study using such information should probably be questioned.
In regard to the second point there are examples of the application of the proposed pooling techniques [32
] and a major strength of this method is the fact that the estimates are the result of a comprehensive review and meta-analysis including all major studies. A potential problem with the weighting method of Laird and Mosteller is that it uses a normal approximation to proportions which might be problematic if the proportions are very small because they might not have a normal distribution anymore [33
]. In a similar fashion, the statistical properties of a χ2 test to determine heterogeneity might be adversely affected when proportions are very small. If this is the case, robustness might be assessed by using alternate pooling methods or comparing fixed versus random approaches. Finally, although the modeling techniques used in this study are not commonly found in clinical medicine and their application to clinical decision making was described only very recently hey are frequent in the economic literature and have been well validated. Additionally, other concerns have been recently raised regarding the use of cost-effectiveness analysis (in our case risk-benefit analysis), namely the degree of discrepancy between probability-based and expectation-based methods, as well as non-transitivity in pair-wise comparisons [34
]. These problems could potentially arise in economic studies and their potential effects on studies using the approach proposed herein deserves further evaluation. Finally, the performance of the present approach compared to other methods for indirect comparisons still remains to be tested.