Suppose it is possible to specify a pt
, the threshold probability of disease for taking some action, such as biopsying a man for prostate cancer: if a patient's estimated probability of disease is greater than pt
he will opt for biopsy; if it is less than pt
, he will not opt for biopsy. By definition, when the probability of disease is equal to the threshold probability pt
, the benefits of opting for biopsy or no biopsy are equal. Thus:
Now b00 – b10
is the benefit of true negative result compared to a false positive result; in clinical terms, the benefit of avoiding unnecessary treatment such as a negative biopsy. Comparably, b11 – b01
is the benefit of a true positive result compared to a false negative result; in other words, the benefit of treatment where it is indicated. Equation (1)
therefore tells us that the threshold probability at which a patient will opt for treatment is informative of how a patient weighs the relative benefit of appropriate treatment compared to the benefit of avoiding unnecessary treatment (Pauker and Kassirer 1980
We can rearrange (1) to obtain:
This states that the harm of a false positive compared to a true negative, is equal to the benefit of a true positive compared to a false negative, multiplied by the odds at pt
. A “net benefit” is benefit minus harm, thus the theoretical relationship in (2) allows us to define a net benefit (first described by CS Peirce (Baker and Kramer 2007
This expression is equivalent to:
There are three advantages to using the threshold probability pt
in place of the benefit parameters bxy
. First, only a single parameter needs to be chosen. Second, the units of the parameter are more intuitive: patients and clinicians understand the concept of risk much more easily than the idea of a health state value on a scale of 0 to 1. Indeed, threshold probability is closely related to a widely-used statistic, positive predictive value. For example, it has been argued that the positive predictive value of a screening test for ovarian cancer needs to be at least 10%, because clinicians would be unwilling to conduct more than 10 surgeries to find a single case of ovarian cancer(Skates et al., 1995
). Accordingly, we might therefore use a pt
of 10% in a decision analysis of ovarian cancer. Third, a threshold probability can be used both for weighting true and false positive test results and for determining the cut-off for a positive test result: instead of arbitrarily choosing a free-to-total PSA ratio cut-off of 0.18, 0.15 or 0.20, we calculate probabilities of cancer by logistic regression and use the threshold probability as the cut-off.
Following Vergouwe et al. (2002)
, a straightforward decision analytic method for determining the value of a diagnostic test, predictive model or molecular marker is as follows:
- Obtain a threshold probability (pt) for treatment
- If necessary, use logistic regression to convert the results of the test, marker or model into a predicted probability of disease
- Define patients as test positive if ≥ pt and negative otherwise. For a binary diagnostic test, is 1 for positive and 0 for negative
- Calculate net benefit of the test, marker or model using the formula for net benefit in equation (3) or (4)
- Calculate clinical net benefit for the strategy of treating all patients. As sensitivity is 100% and specificity 0%, (4) simplifies to:
- The net benefit for the strategy of treating no patients is defined as zero.
- The optimal strategy is that with the highest clinical net benefit.
Note that the unit for net benefit is the number of true cases found per patient and therefore has a maximum value at the prevalence π: all cases found, with no false positives.
To illustrate calculation of a net benefit, we will use a pt of 20%. To calculate the net benefit for free-to-total ratio, we first have to convert values of the marker into predicted probabilities of cancer by logistic regression. shows that of the total of 753 patients, there were 369 who, on the basis of the predictive model using free-to-total PSA ratio, had a predicted probability of cancer of 20% or more. Of these, 149 had cancer and 220 did not. This gives a net benefit of [149 + 220 × (0.2 ÷ 0.8)] ÷ 753 = 0.1248. In comparison, the net benefit for a strategy of biopsying all men is 0.0687; the net benefit for biopsying no men is, by definition, zero.
Net benefit at a threshold probability pt of 20%.
As was for the case for expected value in a traditional decision analysis, we take the strategy with the highest net benefit, irrespective of the size of the difference. Hence for men who would accept a biopsy if their risk of prostate cancer was 20% or more, but not if their risk was less than 20%, the optimal strategy is to calculate their probability of cancer from a logistic model using free-to-total ratio as the predictor and then biopsy those with predicted risk from the model of 20% or more.