The direct translation of results of trials to individual patients in clinical practice is often difficult because not all respond to treatment similar to the average patient enrolled in a trial. This is because the effect of treatment often depends on the characteristics of individual patients. In the present study we have shown how data from randomised clinical trials can be used to predict absolute treatment effects for individual patients, taking patient characteristics into account. In addition, we have assessed the added value of such predictions for medical decision making.
Implementation of an individualised prediction of treatment effect in clinical practice is not necessarily complicated. Several prediction rules are already available for estimating baseline risk for vascular events in primary prevention—for example, the Framingham risk score and Reynolds risk score. The example from the Justification for the Use of Statins in Prevention trial shows that estimation of an individual treatment effect can be as easy as multiplying the individual baseline risk, as estimated from the Framingham risk score or the Reynolds risk score, by the average relative treatment effect from the trial report. If, however, risk scores are not yet available in a certain area of medicine, a new prediction model to estimate individual treatment effect can be developed from the trial data. The methods described in this paper can thus be applied to various medical specialties. Online calculators and integration of prediction models in electronic patient record systems could facilitate the widespread use of prediction of treatment effect in clinical practice. The trial example used in this article also shows that even when discrimination and calibration of a prediction model are moderate, the net benefit of treatment assignment according to prediction can still be superior to both treating all patients within the study domain and treating no one for a certain range of NWT (in this example between about 15 and 50).
Prediction of treatment effect for individual patients may enable doctors to practise individualised medicine in an evidence based manner. It could help to make better informed treatment decisions and perhaps motivate patients to adhere to treatment. Presentation of the net benefit of all possible strategies of treatment assignment for a spectrum of NWT is useful in this respect because the NWT possibly varies with patient and provider preferences. This is especially true when treatment is associated with important adverse reactions. For example, treatment with tissue plasminogen activator for acute myocardial infarction is associated with an increased risk for intracranial haemorrhage that also varies according to individual patient characteristics.6 13
If patients have difficulties understanding the concept of risk, the predicted individual treatment effect (expressed in terms of absolute risk reduction) can be expressed as a NNT (the number of similar patients that needs to be treated to prevent one outcome event; table 2), which might be more intuitive, and this can be compared to the appropriate NWT.
Prediction of treatment effect for individual patients might also facilitate the work of practice guideline committees that aim to make well informed decisions about indications for treatment on a group level. When the trial results are presented using the methods presented in this paper, the remaining issue that guideline committees need to focus on is the appropriate NWT. The NWT is estimated by weighing the total harms of treatment (for example, adverse reactions, monetary costs, discomfort of sustaining treatment) against the harms of the outcome event of interest (cardiovascular event). For any given NWT three possible treatment strategies must be considered: treat everyone, treat no one, or treat based on prediction (selective treatment of patients whose predicted treatment effect exceeds a decision threshold). When the NWT is agreed on, the trial results can be used to estimate the net benefit of each strategy (table 3 and fig 4) and to determine the optimal treatment strategy (fig 5). The treatment strategy with the highest net benefit for the appropriate value of NWT results in the most favourable trade-off between treatment rate and event rate. Applying this strategy in clinical practice leads to more selective treatment of patients who will benefit from treatment.
Previously, risk stratified reporting of trial results was proposed as a method for presenting heterogeneity of treatment effects in trials.3 7
In line with this, the relative risk and NNT for participants of the Justification for the Use of Statins in Prevention trial within subgroups of estimated baseline risk were published earlier.22 23 24
Stratified analysis of treatment effects in subgroups of the total study cohort may, however, lead to imprecision owing to loss of statistical power. Moreover, existing risk scores are not available for many diseases, invalidating the risk stratified approach. Also, risk based stratification may still obscure important modification of relative treatment effect that can be discovered or excluded (as in the Justification for the Use of Statins in Prevention trial example) by a multivariate model for predicting treatment effect based on trial data. Also, the cut-off values for defining subgroups of estimated baseline risk are usually predefined, whereas the methods shown in this paper allow searching for the treatment threshold that is associated with maximum net benefit.
Although data from clinical trials have been used before to predict treatment effects for individual patients, evidence supporting the added value of individualised prediction of treatment effect for clinical practice has been sparse.8 9 10 11 12 13
Expensive and long lasting impact trials were needed to show the benefit of prediction based treatment.25
In this article we show that the net benefit assessment methods, described previously, provide a more efficient and readily available opportunity for evaluating the potential net benefit of prediction based treatment and for determining implications of contemporary trial results for clinical practice.5
This report also shows that the added value of individualised prediction of treatment effect for medical decision making may not be universal but instead is conditional on the NWT.
Limitations and challenges of the study
Limitations of using trial data for individualised predictions of treatment effect generally include short and variable follow-up times, whereas meaningful predictions of cardiovascular event risk usually comprise a 10 year period. This is particularly true for the Justification for the Use of Statins in Prevention trial because the study was discontinued early, but few clinical trials have a follow-up period as long as 10 years either. Thus the predictions and observations usually need to be extrapolated. Furthermore, similar to conventional trial reports, generalisability of the results may be problematic. Trial participants were often selected on the basis of strict eligibility criteria and are healthier and more compliant to treatment than are patients in clinical practice.6
In this example, the results apply to patients without manifest vascular disease or diabetes, but additional eligibility criteria of the trial were low levels of low density lipoprotein cholesterol and increased levels of high sensitivity C reactive protein. Hence application of trial based predictions of treatment effect to the general population may be suboptimal. This is especially true for newly fit models because important risk factors (such as low density lipoprotein cholesterol and high sensitivity C reactive protein in our example) may not be included in the prediction model if all trial participants had similar characteristics.
Apart from these practical constraints, many feel reluctant to interpret the implications of subgroup analyses let alone multivariate prediction of treatment effect, because over-accuracy and chance findings may occur.26
Predictions of treatment effect should therefore be based on existing risk scores developed in external data when possible.2 7
Yet even when validated risk scores are available, as in our example, developing a new prediction model fit to the trial data can help to confirm the assumption that treatment effect increases linearly with baseline risk (fig 1). Moreover, it should be stressed that the estimated treatment effects in prediction models originating from randomised trials are not subject to confounding bias, because treatment was allocated randomly in the study population. Over-fitting can be minimised by careful and preferably prespecified selection of candidate predictors and shrinkage of the model coefficients when needed. Web extra appendix 3 summarises considerations that need to be taken into account when applying the methods described in this paper to other trial datasets.
Data from randomised trials can be used to predict treatment effect in terms of absolute risk reduction for individual patients before the start of intended treatment. Predictions could be based on existing risk scores, if available, or a newly developed model. The value of such prediction of treatment effect for medical decision making is conditional on the NWT to prevent one outcome event. Prediction based treatment may result in positive net benefit for a range of NWT, even when model calibration and discrimination are moderate. The methods shown in this paper could therefore become a routine part of reporting clinical trials and be used in everyday clinical practice.
What is already known on this topic
- In clinical practice some patients benefit more than average from treatment, whereas others do not or may even be harmed
- Implementing trial results by treating all or no patients, expecting the treatment effect for everyone to be similar to the average treatment effect in the original trial, may not lead to optimal benefit
What this study adds
- Data from randomised trials can be used to predict treatment effect in terms of absolute risk reduction for individual patients
- Predictions could be based on existing validated risk scores, if available, or a new prediction model fit to the trial data
- The value of such prediction of treatment effect for medical decision making is conditional on the number willing to treat (NWT) to prevent one outcome event