We illustrate the methodology using data from the placebo arm of the Prostate Cancer Prevention Trial (

Thompson et al., 2006). Almost all subjects, 5519, in the cohort underwent prostate biopsy at the end of the study and thus had prostate cancer disease status available. 21.9% of men were found to have prostate cancer. The marker, PSA value prior to biopsy, is available for every subject. Our goal is to evaluate PSA as a risk prediction marker for diagnosis of prostate cancer from the biopsy. To illustrate application of our methodology to a nested case-control sample, we randomly sampled 250 cases and 250 controls from the cohort and pretend that PSA is only measured for these 500 subjects but not for the rest of the cohort.

Given a low risk threshold of 10% and a high risk threshold of 30%, our main interest is to estimate

*R*^{−1}(0.10) and 1 −

*R*^{−1}(0.30), the proportions of subjects falling into the low and high risk ranges. We considered two ways to fit the partial binormal ROC curve, that prioritize precision and flexibility respectively. First, we fit one concave partial ROC curve to provide estimates for both

*R*^{−1}(0.10) and 1 −

*R*^{−1}(0.30). To include most of the available data in the interval while avoiding problems due to sparse data at the boundary, we choose the FPF range

*t* (0.05, 0.95). This guarantees at least 12 controls with data beyond the corresponding risk threshold estimate. The estimated partial ROC curves are displayed in . Also displayed is the empirical ROC curve (

Obuchowski, 2003). The two in general agree well with each other.

We then estimate the corresponding partial predictiveness curve for PSA plugging in

estimated from the parent cohort. The partial predictiveness curve with its 95% confidence intervals (taking variablity in

into account) are displayed in , Also displayed is the nonparametric predictiveness estimate,

(

*v*) vs

*v*, under the monotone increasing risk assumption. This curve is generated by estimating

(

*D* = 1|

*Y*) using isotonic regression as described below, estimating

*F*_{D} and

*F*_{} empirically with

_{D} and

_{}, calculating

=

_{D} +(1−

)

_{}, and

(

*v*) =

{

*D* = 1|

*Y* =

^{−1}(

*v*)}. Again, the partial predictiveness curve derived from the partial binormal ROC models appears to be similar to the nonparametric curve.

presents corresponding estimates of

*R*^{−1}(0.1) and 1 −

*R*^{−1}(0.3). First, based on

estimated from the phase-one cohort (

= 21.9%), 20.5% subjects in the population are classified as low risk, 26.4% classified as high risk. Confidence intervals are constructed either treating disease prevalence as fixed or taking variability in

into account. In this example, the parent cohort is much larger than the case-control sample (around 10 times larger), so variability in

has a very small impact on the width of these CIs. Observe that the widths of the confidence intervals are around 18% for estimating

*R*^{−1}(0.10) and 15% for estimating 1 −

*R*^{−1}(0.30), reasonably tight from a clinical point of view.

| **Table 4**Estimates of the predictiveness for PSA. Estimates are based on partial binormal ROC models. Confidence intervals are constructed using percentiles of bootstrap distribution. |

When flexibility of the curve is of major concern, a second strategy can be employed for estimating the predictiveness of PSA at the low and high risk thresholds. Two concave binormal partial predictiveness curves are fitted separately at the low and high ends of the domains for *v*. The ranges of false positive fractions (FPF) for the corresponding partial ROC curve are (0.38, 0.95) and (0.10, 0.32). Our strategy for choosing these ranges is described as follows.

For the high risk threshold,

*p*_{H}, and analogously for the low risk threshold,

*p*_{L}, we require that the FPF corresponding to the risk threshold is an interior point of the domain of the partial ROC curve. As described below, we first fit a non-parametric risk model to the data, and use

to denote the fitted value for the

*i*^{th} subject. We then choose as the upper limit of the ROC domain the estimated FPF corresponding to a slightly lower risk value,

. Similarly, we chose as the lower limit of the ROC domain the estimated FPF corresponding to a slightly higher risk values,

. We used Δ = 0.05, although larger values could certainly be employed. To avoid problems due to sparse data at the boundary, if the lower FPF limit computed above was smaller than 0.05, it was changed to 0.05, and if the upper limit calculated above was larger than 0.95, it was changed to 0.95.

The estimated partial ROC curves are displayed in . They agree fairly well with the empirical ROC curve and the partial ROC curve with

*t* (0.05, 0.95). The corresponding partial predictiveness curve for PSA plugging in

estimated from the parent cohort, with its 95% percentile bootstrap confidence interval (with variability in

taken into consideration) are displayed in . Compared to , confidence intervals are wider with reduced range of FPF. The width of CIs for

*R*^{−1}(0.10) and 1 −

*R*^{−1}(0.30) are around 22% (). Compare the results for the two partial ROC curve fitting strategies in . Estimates of

*R*^{−1}(0.10) and 1 −

*R*^{−1}(0.30) shift upward by 3% and 6% respectively when the more flexible strategy is employed. Still, they fall into the 95% CIs based on the curve with the wider FPF range. In practice, if desired precision is of major concern, a wide FPF range is favored. To further increase precision while maintaining flexibility requires a larger sample size.

Note that in practice, when we don’t have a large cohort available for precise estimation of the disease prevalence, sensitivity analysis with varying *ρ* becomes important. Using our data as an example, the impact of varying perturbations of *ρ* are examined (). First we plug in *ρ* = 20.87% and 23.05%, which correspond to the lower and upper bounds of the 95% confidence interval for disease prevalence based on the cohort. When a bigger *ρ* is entered, we get a larger estimate of 1−*R*^{−1}(0.30) and smaller estimate of *R*^{−1}(0.10), although changes in these estimates are not big here (around 3% for both partial ROC curve fitting strategies). Next we increase the perturbation in *ρ* and plug in *ρ* = 18.53% and 25.78%, which correspond to lower and upper bounds of the 95% confidence interval for disease prevalence if a cohort with 500 subjects is used for estimating prevalence. This time the magnitude of the predictiveness estimates changes quite a lot (around 5–10%). This demonstrates the value of a two-phase design for biomarker evaluation when *ρ* is estimated precisely from a large cohort while the novel marker *Y* is measured only for a case-control subset. We also see the value of a sensitivity analysis when an accurate estimate of prevalence is not available.

Finally we describe the algorithm used to fit the non-parametric risk model that incorporates the monotone increasing risk assumption. It involves two steps: (a) We compute

*P* (

*D* = 1|

*Y*,

*Sampled*) from the case-control sample using isotonic regression with the pool-adjacent-violators algorithm (

Barlow et al., 1972). Specifically, let

*y*_{1}, …,

*y*_{U} be the unique values for

*Y* in the case-control sample in increasing order. These comprise the initial blocks of the data. The unrestricted MLE of

*P* (

*D* = 1|

*Y* =

*y*,

*Sampled*) within each block is computed as the observed proportion of diseased subjects in that block. Next, estimators within adjacent blocks are compared. If estimators from a pair of adjacent blocks do not increase, the two blocks are pooled and the estimator is recomputed. This procedure of comparison and blocking continues until the sequence of proportions is non-decreasing. At the conclusion of the procedure, the restricted MLE,

(

*D* = 1|

*Y* =

*y*_{j},

*Sampled*), is the proportion of diseased subjects within the block containing

*y*_{j}; (b) We estimate the population risk function according to Bayes’ theorem

(

*D* = 1|

*Y*)/

(

*D* = 0|

*Y*) = {

(

*D* = 1|

*Y*,

*sampled*)/

(

*D* = 0|

*Y*,

*sampled*)} (

*n*_{}/

*n*_{D}) {

/(1 −

)}, and obtain a nonparametric risk estimate for every subject in the case-control sample

,

*i*=1,…,

*n*_{D}+

*n*_{}.