Home | About | Journals | Submit | Contact Us | Français |

**|**Int J Biostat**|**PMC2827895

Formats

Article sections

- Abstract
- 1. Background
- 2. Summary Indices Involving Risk Thresholds
- 3. Threshold Independent Summary Measures
- 4. Estimation OF Summary Measures
- 5. Asymptotic Distribution Theory
- 6. Simulation Studies
- 7. The Cystic Fibrosis Data
- 8. Concluding Remarks
- REFERENCES

Authors

Related links

Int J Biostat. 2009 January 1; 5(1): 27.

Published online 2009 October 1. doi: 10.2202/1557-4679.1188

PMCID: PMC2827895

Wen Gu and Margaret Pepe

Copyright © 2009 The Berkeley Electronic Press. All rights reserved

This article has been cited by other articles in PMC.

The predictive capacity of a marker in a population can be described using the population distribution of risk (Huang et al. 2007; Pepe et al. 2008a; Stern 2008). Virtually all standard statistical summaries of predictability and discrimination can be derived from it (Gail and Pfeiffer 2005). The goal of this paper is to develop methods for making inference about risk prediction markers using summary measures derived from the risk distribution. We describe some new clinically motivated summary measures and give new interpretations to some existing statistical measures. Methods for estimating these summary measures are described along with distribution theory that facilitates construction of confidence intervals from data. We show how markers and, more generally, how risk prediction models, can be compared using clinically relevant measures of predictability. The methods are illustrated by application to markers of lung function and nutritional status for predicting subsequent onset of major pulmonary infection in children suffering from cystic fibrosis. Simulation studies show that methods for inference are valid for use in practice.

The predictive capacity of a marker in a population can be described using the population distribution of risk (Huang et al. 2007; Pepe et al. 2008a; Stern 2008). Virtually all standard statistical summaries of predictability and discrimination can be derived from it (Gail and Pfeiffer 2005). The goal of this paper is to develop methods for making inference about risk prediction markers using summary measures derived from the risk distribution. We describe some new clinically motivated summary measures and give new interpretations to some existing statistical measures. Methods for estimating these summary measures are described along with distribution theory that facilitates construction of confidence intervals from data. We show how markers and, more generally, how risk prediction models, can be compared using clinically relevant measures of predictability. The methods are illustrated by application to markers of lung function and nutritional status for predicting subsequent onset of major pulmonary infection in children suffering from cystic fibrosis. Simulation studies show that methods for inference are valid for use in practice.

Let *D* denote a binary outcome variable, such as presence of disease or occurrence of an event within a specified time period and let *Y* denote a set of predictive markers used to predict a bad outcome, *D* = 1, or a good outcome, *D* = 0. For example, elements of the Framingham risk score (age, gender, total and high-density lipoprotein cholesterol, systolic blood pressure, treatment for hypertension and smoking) are used to predict occurrence of a cardiovascular event within 10 years (http://hp2010.nhlbihin.net/atpiii/calculator.asp). We write the risk associated with marker value *Y* = *y* as *risk*(*y*) = *P*[*D* = 1*|Y* = *y*].

Huang et al. (2007) proposed the predictiveness curve to describe the predictive capacity of *Y*. It displays the population distribution of risk via the risk quantiles, *R*(*ν*) versus *ν*, where

$$P[\mathit{\text{risk}}(Y)\hspace{0.17em}\le \hspace{0.17em}R(\nu )]=\nu .$$

The inverse of the predictiveness curve is simply the cumulative distribution function (cdf) of *risk*(*Y*)

$${R}^{-1}(p)=P[\mathit{\text{risk}}(Y)\hspace{0.17em}\le p]={F}_{\mathit{\text{risk}}}(p)$$

and correspondingly

$$R(\nu )={F}_{\mathit{\text{risk}}}^{-1}(\nu ).$$

Gail and Pfeiffer (2005) noted that standard statistical measures used to quantify the predictive capacity of a risk prediction model can be calculated from the risk distribution function, *F _{risk}*(

Summary indices are often used to compare prediction models. The area under the ROC curve is widely used in practice for this purpose. However there is controversy about its use, particularly in the cardiovascular research community (Cook 2007; Pencina et al. 2008). This has motivated another approach to evaluating risk prediction markers that relies on defining categories of risk that are clinically meaningful. Several summary indices based on this notion have been proposed. The reclassification percent and the net reclassification index (NRI) are such summary measures derived from reclassification tables and they have recently gained popularity in the applied literature (Ridker et al. 2008; D’Agostino et al. 2008).

In this paper, we explicitly relate existing and new summary measures of prediction to the risk distribution, i.e. to the predictiveness curve. We contrast them qualitatively, paying particular attention to their clinical interpretations and relevance. We then derive distribution theory that can be used for making statistical inference. Note that rigorous methods for inference have not been available heretofore for several of the existing summary measures. Rather the measures are used informally in practice. Small sample performance is investigated for the new and existing summary measures with simulation studies.

The methods are illustrated with data from 12,802 children with cystic fibrosis disease. We describe the data and risk modelling methods in detail later in section 7. Briefly, we compare the capacities of lung function and nutritional measures made in 1995 to predict onset of a pulmonary exacerbation event during the following year. Overall, 41% of children had a pulmonary exacerbation in 1996. Figure 1 displays predictiveness curves (estimated using methods described in Section 7) for two risk models, one based on lung function (FEV_{1}) and one based on weight. We see from Figure 1 that lung function is more predictive in the sense that more subjects have lung function based risks that are at the high and low ends of the risk scale than is true for weight based risks. Since a good risk marker is one that is helpful to individuals making medical decisions, and because decisions are more easily made when an individual’s risk is high or low than if it is in the middle, we conclude informally from the curves that lung function is a superior predictor than weight. We next define formal summary indices that can be used for descriptive and comparative purposes and illustrate them with the cystic fibrosis data.

In clinical practice, a subject’s risk is calculated to assist in medical decision making. If his risk is high, he may be recommended for diagnostic, treatment or preventive interventions. If his risk is low, he may avoid interventions that are unlikely to benefit him. In certain clinical contexts, explicit treatment guidelines exist that are based on individual risk calculations. For example, the Third Adult Treatment Panel recommends that if a subject’s 10 year risk of a cardiovascular disease exceeds 20% he should consider low density lipoprotein (LDL)-lowering therapy (Adult Treatment Panel III 2001). The risk threshold that leads one to opt for an intervention depends on anticipated costs and benefits. These may vary with individuals’ perceptions and preferences (Vickers and Elkin 2006; Hunink et al. 2006). The choice of threshold may also vary with the availability of health care resources. In this section we discuss summary indices that depend on specifying a risk threshold. To be concrete we suppose that the overall risk in the population is high, *ρ* = *P*[*D* = 1], and that the goal of the risk model is to identify individuals at low risk, *risk*(*Y*) *< p _{L}*, where

For illustration with the cystic fibrosis data, we choose the low risk threshold *p _{L}* = 0.25 which contrasts with the overall incidence

A simple compelling summary measure is the proportion of the population deemed to be at low risk according to the risk model. This is *R ^{−}*

Another important perspective from which to evaluate risk prediction markers is classification accuracy (Pepe et al. 2008* _{a}*, Janes, Pepe and Gu 2008). This is characterized by the risk distribution in cases, subjects for whom

$$\text{TPR}({p}_{L})=P(\mathit{\text{risk}}(Y)\ge {p}_{L}|D=1);\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{FPR}({p}_{L})=P(\mathit{\text{risk}}(Y)\ge {p}_{L}|D=0).$$

Higher TPR(*p _{L}*) and lower FPR(

Figure 2 shows cumulative distributions of *risk*(*Y*) in cases and controls separately. From this, TPR(*p*) and FPR(*p*) can be gleaned for any value of *p*. We see that the proportion of controls in the low risk stratum is much larger when using lung function as the risk prediction marker than for weight, 1-FPR(*p _{L}*) = 46% for lung function as opposed to 15% for weight. However the proportion of cases whose risks exceed

Cumulative distributions of risk based on FEV_{1} and weight in predicting the risk of having at least one pulmonary exacerbation in the following year in children with cystic fibrosis. Distributions are shown separately for subjects who had events (cases, **...**

Observe that TPR(*p _{L}*) and FPR(

Another pair of summary measures is the event rates in the two risk strata. These can be thought of as predictive values, PPV(*p _{L}*) and 1-NPV(

$$\text{PPV}({p}_{L})=P(D=1|\mathit{\text{risk}}(Y)>{p}_{L}),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}1-\text{NPV}({p}_{L})=P(D=1|\mathit{\text{risk}}(Y)<{p}_{L}).$$

(1)

PPV(*p _{L}*) is the event rate in the high risk stratum and 1-NPV(

By applying Bayes theorem to (1), PPV and NPV can be written in terms of TPR and FPR:

$$\text{PPV}(p)=\frac{\rho}{1-\rho}\frac{\text{TPR}(p)}{\text{FPR}(p)};\text{NPV}(p)=\frac{1-\rho}{\rho}\frac{1-\text{FPR}(p)}{1-\text{TPR}(p)},$$

(2)

These expressions facilitate estimation of PPV(*p*) and NPV(*p*), which we discuss in section 4.

Event rates are also functions of the predictiveness curve. Specifically they average the curve over the ranges (*ν _{L},* 1) and (0

$$\text{PPV}({p}_{L})={\int}_{{\nu}_{L}}^{1}R(u)du/(1-{\nu}_{L});\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}1-\text{NPV}({p}_{L})={\int}_{0}^{{\nu}_{L}}R(u)du/{\nu}_{L}.$$

For the cystic fibrosis example, estimates of the event rates, 1-NPV(*p _{L}*) and PPV(

In the applied literature, variables are often categorized using quantiles. In this vein, categories of risk are sometimes defined using risk quantiles for which we have used the notation *R*(*ν*). For example, Ridker et al. (2000) used quartiles of risk and noted that high sensitivity c-reactive protein (hs-CRP) was more predictive of cardiovascular risk than standard lipid screening because the level of hs-CRP in the highest versus lowest quartile was associated with a much higher relative risk for future coronary events than was the case for standard lipid measurements.

Another context in which *R*(*ν*) is well motivated is when availability of medical resources is limited. Suppose resources are available to provide an intervention to a fraction 1 *−* *ν* of the population, those 1 *−* *ν* at highest risk. Since *R*(*ν*) is the corresponding risk quantile, subjects given the intervention have risks *≥ R*(*ν*). A marker or risk model for which *R*(*ν*) is larger is preferable because it ensures that those receiving intervention are at greater risk of a bad outcome in the absence of the intervention.

In the cystic fibrosis example, suppose the 10% of the population deemed to be at highest risk will be treated. If lung function is used to calculate risk, subjects with risks at or above 0.76 receive treatment. On the other hand if weight is used to calculate risk, subjects whose risks are as low as 0.52 will be offered treatment.

In a diagnostic setting, it may be important to flag most people with disease as high risk so that people with disease get necessary treatment. In other words, we may require that the TPR exceed a certain minimum value, TPR=*t*. The corresponding risk threshold is an important entity to report. We denote it by *R*(*ν _{T}* (

In screening healthy populations for a rare disease such as ovarian cancer, the false positive rate must be very low in order to avoid large numbers of subjects undergoing unnecessary medical procedures. The risk threshold that yields an acceptable FPR must also be acceptable for individuals as a threshold for deciding for or against medical procedures. To maintain a very low FPR, the risk threshold may be very high in which case the decision rule would not be ethical. Reporting the risk threshold that yields specified FPR=*t* is therefore often important in practice and we denote the threshold by *R*(*ν _{F}* (

Unlike other predictiveness summary measures, *R*(*ν _{T}* (

Several summary measures that rely on defined risk categories have been proposed recently. The context for their definition has been when comparing a baseline risk model with one that adds a novel marker to the baseline predictors using risk reclassification tables that involve 3 or more categories of risk. It is illuminating to consider these measures in our much simplified context, where only 2 risk categories defined by a single risk threshold *p _{L}* are of interest and when the baseline model involves no covariates at all so that the baseline risk is equal to

Cook (2007) proposes the reclassification percent to summarize predictive information in a model. In our context, all subjects are considered high risk under the baseline model because *ρ > p _{L}*. The reclassification percent is therefore the proportion of subjects classified as low risk according to the risk model involving

Pencina et al. (2008) criticize the reclassification percent because it does not distinguish between desirable risk reclassifications (up for cases and down for controls) and undesirable risk reclassifications (down for cases and up for controls). They propose the net reclassification improvement (NRI) summary statistic as an alternative. We use“up” and “down” to denote changes of one or more risk categories in the upward and downward directions, respectively, for a subject between their baseline and augmented risk values. The NRI is defined as

$$\text{NRI}=[P(\text{up}|D=1)-P(\text{down}|D=1)]-[P(\text{up}|D=0)-P(\text{down}|D=0)].$$

In our simple context it is easy to see that

$$\text{NRI}=\text{TPR}({p}_{L})-\text{FPR}({p}_{L})$$

where TPR(*p _{L}*) and FPR(

A key use of summary measures is to compare different risk models. One can quantify the difference in performance between two risk models by taking the difference between summary measures derived from the two models. In the cystic fibrosis example discussed here, the two risk models involve completely different markers. However, one could also entertain two models that involve some common predictors. The setting in which risk reclassification ideas have emerged, is where one model involves standard baseline predictors and the other includes a novel marker in addition to the baseline predictors. Taking the difference in summary measures for the two models is a sensible way of assessing improvement in performance in this context too.

Recall that when only 2 risk categories (low versus high) exist, Cook’s reclassification percent is equal to *R ^{−}*

We represented the NRI statistic as TPR(*p _{L}*)-FPR(

Summarizing data is difficult when more than two risk categories are involved. Statistics such as the NRI have been criticized because they do not distinguish between changes of one risk category and more than one risk category (Pepe et al. 2008b). In a similar vein, when 3 risk categories exist with specific treatment recommendations for each, misclassifying a case as being in the lowest risk level may be more serious than misclassifying him as in the middle category. Similarly, misclassifying a control as being in the highest risk level may be more serious than misclassifying him as being in the middle category. Without specifying utilities associated with different types of misclassifications, any accumulation of data across risk categories is difficult to justify. For these settings we propose use of a vector of summary statistics distinguished by the risk thresholds. For example, suppose we consider three risk categories for the cystic fibrosis study defined by two thresholds *p _{L}* = 0.25 and

Although statistical summaries that depend on clinically meaningful risk thresholds are appealing, the choice of risk thresholds is often uncertain. Different clinicians or policy makers may choose different risk categorizations. This argues for displaying the risk distributions as continuous curves since one can then read from them summary indices described here using any risk threshold of interest to the reader.

Classic measures that describe the predictive strength of a model can be interpreted as summary indices for the predictiveness curve. We describe the relationships next. These measures can compliment the display of risk distributions for several models when no specific risk thresholds are of key interest. In addition, formal hypothesis tests to compare predictiveness curves can be based on them.

The proportion of explained variation, also called *R*^{2}, is the most popular measure of predictive power for continuous outcomes and is popular for binary outcomes too. It is most commonly defined as

$$\text{PEV}\equiv \frac{\text{var}(D)-E(\text{var}(D|Y))}{\text{var}(D)}.$$

But it can also be written as

$$\text{PEV}=\text{var}(\mathit{\text{risk}}(Y))/\rho (1-\rho ),$$

because var(*D*)=*E*(var(*D|Y*))+var(*E*(*D|Y*)) and *E*(*D|Y*) = *P*(*D* = 1*|Y*) = *risk*(*Y*). PEV is a standardized measure of the variance in *risk*(*Y*) since *ρ*(1 *− ρ*) in the denominator is the risk variance for an ideal marker that predicts *risk*(*Y*) = 1 for cases and *risk*(*Y*) = 0 for controls. Hu et al. (2006) noted that PEV can also be written as the correlation between *D* and *risk*(*Y*).

An unintuitive but interesting and simple interpretation for PEV is as the difference between the averages of *risk*(*Y*) for cases and controls(Pepe, Feng and Gu 2008* _{b}*),

$$\text{PEV}=E[\mathit{\text{risk}}(Y)|D=1]-E[\mathit{\text{risk}}(Y)|D=0].$$

In summary for the cystic fibrosis data, PEV, calculated as 0.22 for the lung function measure and 0.05 for weight, can be interpreted as variances of risk distributions displayed in Figure 1 standardized by the ideal variance of 0.41 *×* (1 *−* 0.41) = 0.24, or as differences in means of distributions shown in Figure 2. In Figure 1, var(*risk*(*Y*)) = 0.053 for lung function and 0.012 for weight yielding 0.22 and 0.05 respectively when divided by 0.24. On the other hand in Figure 2, case and control mean risks are 0.54 and 0.32 for lung function while they are 0.44 and 0.39 for weight, again yielding 0.54–0.32=0.22 and 0.44–0.39=0.05 for the PEV values calculated as the differences in means.

Pencina et al. (2008) employ the PEV summary measure to gauge the improvement in risk prediction when clinically relevant risk thresholds do not exist. They do not recognize it as the proportion of explained variation but call it integrated discrimination improvement (IDI) and note that it has another interpretation as Youden’s index integrated uniformly over (0,1):

$$\text{PEV}=\int Y\hspace{0.17em}I(p)dp,$$

where *Y I*(*p*) = *P*(*risk*(*Y*) *> p|D* = 1) *− P*(*risk*(*Y*) *> p|D* = 0) is Youden’s index for the binary decision rule that is positive when *risk*(*Y*) *> p*. In other words, PEV can also be interpreted as the difference between integrated TPR(*p*) and FPR(*p*) functions defined earlier.

In a commentary on the Pencina et al. (2008) paper, Ware and Cai (2008) suggest that IDI, denoted here by PEV, does not depend on the overall event rate, *ρ* = *P*(*D* = 1). We disagree. To illustrate, suppose we have a single marker with risk function *risk*(*Y*) increasing in *Y*. Then

$$\begin{array}{l}\text{PEV}=\int (P(\mathit{\text{risk}}(Y)>p|D=1)-P(\mathit{\text{risk}}(Y)>p|D=0))dp\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\int (P(Y>y|D=1)-P(Y>y|D=0))\frac{\partial \mathit{\text{risk}}(Y)}{\partial Y}{|}_{Y=y}dy\end{array}$$

where *risk*(*y*) = *p*. Here, the conditional probabilities, *P*(*Y > y|D* = 1) and *P*(*Y > y|D* = 0), are independent of prevalence, *ρ*, but
$\frac{\partial \mathit{\text{risk}}(Y)}{\partial Y}$ is a function of *ρ*. To demonstrate, consider a simple linear logistic regression model,

$$\text{logit}P(D=1|Y)={\theta}_{0}+{\theta}_{1}Y.$$

(3)

And note that

$$\frac{\partial \mathit{\text{risk}}(Y)}{\partial Y}=\frac{\partial P(D=1|Y)}{\partial Y}={\theta}_{1}P(D=1|Y)\{1-P(D=1|Y)\}.$$

Since
$P(D=1|Y)=\frac{P(Y|D=1)}{P(Y|D=0)}\frac{\rho}{1-\rho}/\{1+\frac{P(Y|D=1)}{P(Y|D=0)}\frac{\rho}{1-\rho}\}$, clearly varies with *ρ*, so does its derivative. Figure 4 shows the relationship between PEV and *ρ* for a marker *Y* that is standard normally distributed in controls and normally distributed with mean 1 and variance 1 in cases. The risk is a simple linear logistic risk function (equation (3)). As *ρ* increases from 0 to 1, we see that PEV increases then decreases with maximum occurring at *ρ* = 0.5. Janssens et al. (2006) also demonstrated dependence of PEV on *ρ* through a simulation study.

Relationship between the proportion of explained variation, PEV, and the prevalence. A linear logistic risk model with controls standard normally distributed and cases normally distributed with mean 1 and variance 1 was used to generate the data. Maximum **...**

The proportion of explained variation has been defined in other ways, notably based on notions of log likelihood (deviance). Gail and Pfeiffer (2005) note that these can also be calculated from the risk distribution. However, Zheng and Agresti (2000) make the point that these summary measures are difficult to interpret and we concur wholeheartedly. Therefore, we do not pursue them further in this paper but note that methods for inference could be developed in analogy with those we develop here for PEV.

Total gain, proposed by Bura and Gastwirth (2001) is defined as,

$$TG={\int}_{0}^{1}|R(\nu )-\rho |dv.$$

(4)

This is the area sandwiched between the predictiveness curve and the horizontal line at *ρ*, which is the predictiveness curve for a completely uninformative marker assigning *risk*(*Y*) = *ρ* to all subjects. TG is appealing because it can be visualized directly from the predictiveness curve. For a perfect risk prediction model, the predictiveness curve is a step function rising from 0 to 1 at *ν* = 1 *− ρ*. The corresponding TG is 2*ρ*(1 *− ρ*).

Other interpretations can be made for TG. Huang and Pepe (2008b) have shown that TG is equivalent to the Kolmogorov-Smirnov measure of distance between risk distributions for cases and controls. This is an ROC summary index (Pepe 2003, page 80):

$$\text{TG}=2\rho (1-\rho ){\text{sup}}_{t}\{\text{ROC}(t)-t\}$$

(5)

$$\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=2\rho (1-\rho )\hspace{0.17em}{\text{max}}_{p}\{\text{TPR}(p)-\text{FPR}(p)\}$$

(6)

In fact we can write this more simply.

*Result*

$$\text{TG}=2\rho (1-\rho )\{\text{TPR}(\rho )-\text{FPR}(\rho )\}$$

(7)

Proof

Let *ν ^{*}* be the point where

$$\text{TG}={\int}_{0}^{v*}(\rho -R(\nu ))d\nu +{\int}_{\nu *}^{1}(R(\nu )-\rho )d\nu .$$

Furthermore, because $\rho ={\int}_{0}^{\nu *}R(\nu )d\nu +{\int}_{\nu *}^{1}R(\nu )d\nu $ and $\rho ={\int}_{0}^{1}\rho d\nu $, setting these two terms equal it follows that

$${\int}_{0}^{v*}(\rho -R(\nu ))d\nu ={\int}_{\nu *}^{1}(R(\nu )-\rho )d\nu .$$

Therefore TG can be written as

$$\begin{array}{l}\text{TG}=2{\int}_{\nu *}^{1}(R(\nu )-\rho )d\nu \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=2{\int}_{\nu *}^{1}(R(\nu )d\nu -2\rho (1-\nu *)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=2\rho \hspace{0.17em}\text{TPR}(\rho )-2\rho (1-{R}^{-1}(\rho ))\end{array}$$

because
$\text{TPR}(\rho )={\int}_{\nu *}^{1}R(\nu )d\nu /\rho $. Moreover, since 1 *− R ^{−}*

This representation of TG is useful for estimation and for deriving asymptotic distribution theory. Interestingly, by equating (6) and (7), we find that the maximum value of TPR(*p*)-FPR(*p*) occurs at the risk threshold *p* = *ρ*. Another short proof follows by taking its derivative. In particular, since

$$\text{TPR}(R(\nu ))-\text{FPR}(R(\nu ))=\frac{{\int}_{\nu}^{1}R(u)du}{\rho}+\frac{{\int}_{\nu}^{1}(1-R(u))du}{1-\rho},$$

(8)

taking the derivative of the right side with respect to *ν* and setting it to 0, we have

$$\{1-\frac{R(\nu )}{\rho}\}-\{1-\frac{1-R(\nu )}{1-\rho}\}=0$$

at the solution. That is, the solution is at *R*(*ν*) = *ρ*. In the same illustrative setting used above, where *ρ* = 0.2, *Y* is standard normal in controls and normal with mean 1 and variance 1 in cases, we see from Figure 5 how TPR(*p*)-FPR(*p*) varies with *p*. The maximum value, 0.39, is achieved at *p* = 0.2, i.e. at *p* = *ρ*.

Association between and *TPR*(*p*)*− FPR*(*p*) and *p*. A linear logistic risk model with controls standard normally distributed and cases normally distributed with mean 1 and variance 1 was used to generate the data. Overall prevalence of event *ρ* **...**

Another appealing feature of TG is that after it is standardized by 2*ρ*(1*−ρ*), the total gain for a perfect marker, it is functionally independent of *ρ*. Let’s use
TG to denote standardized total gain

$$\overline{\text{TG}}\equiv \frac{\text{TG}}{2\rho (1-\rho )}$$

so that
TG [0, 1]. We will focus on
TG; here. It is independent of disease prevalence because of it’s interpretation as the Kolmogorov-Smirnov ROC summary index. Moreover, based on the results above,
TG is simply interpreted as the difference between the proportions of cases and controls with risks above the average, *ρ* = *P*(*D* = 1) = *E*(*risk*(*Y*)).

In the cystic fibrosis example, TG based on lung function is 0.20, while TG based on weight is 0.09. Since the overall event rate is *ρ*=41%, the corresponding standardized TG values are
TG=0.42 for lung function and
TG=0.20 for weight.

The area under the ROC curve is widely used to summarize and compare predictive markers and models. It can be interpreted simply as the probability of correctly ordering subjects with and without events using *risk*(*Y*):

$$\text{AUC}=P(\mathit{\text{risk}}({Y}_{1})>\mathit{\text{risk}}({Y}_{2})|{D}_{1}=1,{D}_{2}=0)$$

However, it has been criticized widely for having little relevance to clinical practice (Cook 2007; Pepe and Janes 2008; Pepe et al. 2007). In particular, the task facing the clinician in practice is not to order risks for two individuals. Part of the appeal of the AUC, however, lies in the fact that it depends neither on prevalence, *ρ*, nor on risk thresholds. Yet in the context of risk prediction within a specific clinical population, these attributes may be weaknesses. In particular, when specific risk thresholds are of interest, the ROC curve hides them. In Figure 3, we plot the ROC curves for risk based on lung function and on weight. The AUC values are 0.771 and 0.639, respectively.

Interestingly all of the measures discussed here can be thought of as the mathematical distance between risk distributions for cases and controls (Figure 2) measured in different ways. The PEV is the difference in the means of case and control risk distributions. The
TG is the Kolmogorov-Smirnov measure and we have shown that this is equal to the difference between the proportions of cases and controls with risks larger than *ρ*. The AUC is equivalent to the Wilcoxon measure of distance between risk distributions for cases and controls.

We now turn to estimation of summary indices from data. We focus on the scenario where *Y* is a single continuous marker. We also allow *Y* to be a predefined combination of multiple markers. For example, the score may be derived from a training dataset and our task is to evaluate the combination score using a test dataset.

We use the following notation: *Y*, *Y _{D}* and

Suppose the risk model is *risk*(*Y*) = *P*(*D* = 1*|Y*) = *G*(*θ, Y*), where

$$\text{logit}\{G(\theta ,Y)\}={\theta}_{0}+h({\theta}_{1},Y),$$

and *h* is some monotone increasing function of *Y*. This is a very general formulation. As a special case, logit*{G*(*θ, Y*)*}* could be as simple as *θ*_{0} + *θ*_{1}*Y* with *θ*_{1} *>* 0, the ordinary linear logistic model. We consider estimation first under a cohort or cross sectional design and later discuss case-control designs for which the logistic regression formulation is particularly helpful.

Suppose we have *n* independent identically distributed observations (*Y _{i}*

We plug and into *G* to get estimators of *R*(*ν*), and *R ^{−}*

$$\begin{array}{c}\widehat{R}(\nu )=G\{\widehat{\theta},\widehat{F}-1(\nu )\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\nu \in (0,1),\\ {\widehat{R}}^{-1}(p)=\widehat{F}\{{G}^{-1}(\widehat{\theta},p)\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}p\in \{R(\nu ):\nu \in (0,1)\}.\end{array}$$

Estimates of cases and controls with risks above *p* are:

$$\begin{array}{l}\text{T}\widehat{\text{P}}\text{R}(p)=1-{\widehat{F}}_{D}\{{G}^{-1}(\widehat{\theta},p)\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}p\in \{R(\nu ):\nu \in (0,1)\},\\ \text{F}\widehat{\text{P}}\text{R}(p)=1-{\widehat{F}}_{\overline{D}}\{{G}^{-1})(\widehat{\theta},p)\}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}p\in \{R(\nu ):\nu \in (0,1)\}.\end{array}$$

We write the event rates in risk strata in terms of TPR(*p*) and FPR(*p*) to facilitate their estimation:

$$\begin{array}{cc}\text{P}\widehat{\text{P}}\text{V}(p)=\frac{\widehat{\rho}}{1-\widehat{\rho}}\frac{\text{T}\widehat{\text{P}}\text{R}(p)}{\text{F}\widehat{\text{P}}\text{R}(p)}& \text{for}\hspace{1em}p\in \{R(\nu ):\nu \in (0,1)\}\\ 1-\widehat{\text{N}}\text{PV}(p)=1-\frac{1-\widehat{\rho}}{\widehat{\rho}}\frac{1-\text{F}\widehat{\text{P}}\text{R}(p)}{\text{T}\widehat{\text{P}}\text{R}(p)}& \text{for}\hspace{1em}p\in \{R(\nu ):\nu \in (0,1)\}\end{array}$$

In a cohort study, these estimates are equal to the empirical proportions of cases amongst those with estimated risks above and below *p*. However, the formulations here are valid in a case-control study too. Finally risk thresholds yielding specified TPR or FPR are obtained by first calculating the corresponding quantile of *Y* and then plugging it into the fitted risk model:

$$\begin{array}{l}\widehat{R}({\nu}_{T}(t))=G\{\widehat{\theta},{\widehat{F}}_{D}^{-1}({\nu}_{T}(t))\}\hspace{1em}\text{for}\hspace{1em}\text{TPR}=1-{\nu}_{T}(t),\\ \widehat{R}({\nu}_{F}(t))=G\{\widehat{\theta},{\widehat{F}}_{\overline{D}}^{-1}({\nu}_{F}(t))\}\hspace{1em}\text{for}\hspace{1em}\text{FPR}=1-{\nu}_{F}(t).\end{array}$$

Summary measures that do not involve specific risk thresholds are proportion of explained variation, PEV, standardized total gain, TG, and area under the ROC curve, AUC. Recall that PEV is the difference between mean risk in cases and in controls. Sample means of estimated risks yield an estimator of PEV:

$$\text{P}\widehat{\text{E}}\text{V}=\int G(\widehat{\theta},Y)d{\widehat{F}}_{D}(Y)-\int G(\widehat{\theta},Y)d{\widehat{F}}_{\overline{D}}(Y).$$

On the other hand,
TG, can be expressed as the difference between the proportion of cases and controls with risks less than *ρ*. We write:

$$\widehat{\overline{\text{TG}}}=\{{\widehat{F}}_{\overline{D}}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))\}.$$

Finally AUC is estimated as the proportion of case-control pairs where the estimated risk for the case exceeds that of the control

$$\text{A}\widehat{\text{U}}\text{C}=\frac{1}{{n}_{D}{n}_{\overline{D}}}\sum _{i=1}^{{n}_{\hspace{0.17em}D}}\sum _{j=1}^{{n}_{\hspace{0.17em}\overline{D}}}I(G(\widehat{\theta},{Y}_{D\hspace{0.17em}i})>G(\widehat{\theta},{Y}_{\overline{D}\hspace{0.17em}j})).$$

Since *G*(*θ, Y*) is increasing in *Y*, this is the same as the standard empirical estimator of the AUC based on *Y*,

$$\text{A}\widehat{\text{U}}\text{C}=\frac{1}{{n}_{D}{n}_{\overline{D}}}\sum _{i=1}^{{n}_{\hspace{0.17em}D}}\sum _{j=1}^{{n}_{\hspace{0.17em}\overline{D}}}I({Y}_{D\hspace{0.17em}i}>{Y}_{\overline{D}\hspace{0.17em}j}).$$

Case-control studies are often conducted in the early phases of marker development (Pepe et al. 2001; Baker et al. 2002). Compared to cohort studies, they are smaller and more cost efficient. Since early phase studies dominate biomarker research, it is crucial that estimates of statistical measures of performance accommodate case-control designs. In this section, we describe estimation under a case-control design assuming that an estimate of prevalence, is available. The value may be derived either from a cohort which is independent from the case-control sample, or from the parent cohort within which the case-control sample is nested. As a special case one can assume *ρ* is known or fixed without sampling variability. In determining populations where risk markers may or may not be useful, predictiveness curves could be evaluated for various specified fixed values of *ρ*.

In case-control studies, we sample fixed numbers of cases and controls, *n _{D}* and

$$\text{logit}\{G({\theta}_{S},Y)\}={\theta}_{0S}+h({\theta}_{1S},Y),$$

where
${\theta}_{0S}={\theta}_{0}-\text{log}\frac{{n}_{\overline{D}}}{{n}_{D}}\frac{\rho}{1-\rho}$ and *θ*_{1}* _{S}* =

The marker distribution in the population, *F*, cannot be estimated directly because of the case-control sampling design. However, since case and control samples are representative, empirical estimates of *F _{D}* and

Estimates of the predictiveness summary measures can then be obtained by plugging corresponding values for , , * _{D}*,

In this section, we present asymptotic distribution theory for all of the summary measures defined in previous sections. Results for pointwise estimators of *R*(*ν*) and *R ^{−}*

Assume the following conditions hold:

*G*(*s, Y*) is a differentiable function with respect to*s*and*Y*at*s*=*θ*,*Y*=*F*^{−}^{1}(*ν*).*G*^{−}^{1}(*s, p*) is continuous, and*G*^{−}^{1}(*s, p*)*/s*exists at*s*=*θ*.

**Theorem** As *n → ∞*, each of the following random variables converges to a mean zero normal random variable: (i)
$\sqrt{n}({\widehat{R}}^{-1}(p)-{R}^{-1}(p))$, with variance

$$\begin{array}{l}{\sigma}_{1}{(p)}^{2}=\mathit{\text{var}}(\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p))))+{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial {R}^{-1}(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p))));\end{array}$$

(ii) $\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p))$, with variance

$$\begin{array}{l}{\sigma}_{2}{(p)}^{2}=\mathit{\text{var}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))+{(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2{(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))));\end{array}$$

(iii) $\sqrt{n}(F\widehat{P}R(p)-FPR(p))$, with variance

$$\begin{array}{l}{\sigma}_{3}{(p)}^{2}=\mathit{\text{var}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))))+{(\frac{\partial FPR(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial FPR(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2{(\frac{\partial FPR(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))));\end{array}$$

(iv) $\sqrt{n}(P\widehat{P}V(p)-PPV(p))$, with variance

$$\begin{array}{l}{\sigma}_{4}{(p)}^{2}=PPV{(p)}^{2}{(1-PPV(p))}^{2}\{(\frac{{\sigma}_{2}{(p)}^{2}}{\mathit{\text{TPR}}{(p)}^{2}}+\frac{{\sigma}_{3}{(p)}^{2}}{FPR{(p)}^{2}}+\frac{{\sigma}_{\rho}^{2}}{{\rho}^{2}{(1-\rho )}^{2}})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{{\mathit{\text{cov}}}_{1}}{\mathit{\text{TPR}}(p)FPR(p)}-\frac{{\mathit{\text{cov}}}_{2}}{\mathit{\text{TPR}}(p)\rho (1-\rho )}+\frac{{\mathit{\text{cov}}}_{3}}{FPR(p)\rho (1-\rho )})\}\end{array}$$

where ${\sigma}_{\rho}^{2}$ is the asymptotic variance of $\sqrt{n}(\widehat{\rho}-\rho )$ and

$${\mathit{\text{cov}}}_{1}=\mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(F\widehat{P}R)(p)-FPR(p)))$$

(9)

$${\mathit{\text{cov}}}_{2}=\mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(\widehat{\rho}-\rho ))$$

(10)

$${\mathit{\text{cov}}}_{3}=\mathit{\text{cov}}(\sqrt{n}(F\widehat{P}R(p)-FPR(p)),\sqrt{n}(\widehat{\rho}-\rho ));$$

(11)

(v) $\sqrt{n}(N\widehat{P}V(p)-NPV(p))$, with variance

$$\begin{array}{l}{\sigma}_{5}{(p)}^{2}=NPV{(p)}^{2}{(1-NPV(p))}^{2}\{(\frac{{\sigma}_{5}{(p)}^{2}}{(1-TPR{(p))}^{2}}+\frac{{\sigma}_{6}{(p)}^{2}}{(1-FPR{(p))}^{2}}+\frac{{\sigma}_{\rho}^{2}}{{\rho}^{2}{(1-\rho )}^{2}})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{{\mathit{\text{cov}}}_{1}}{(1-\mathit{\text{TPR}}(p))\hspace{0.17em}(1-FPR(p))}+\frac{{\mathit{\text{cov}}}_{2}}{(1-\mathit{\text{TPR}}(p))\rho (1-\rho )}-\frac{{\mathit{\text{cov}}}_{3}}{(1-FPR(p))\rho (1-\rho )})\};\end{array}$$

(vi) $\sqrt{n}(\widehat{R}(\nu )-R(\nu ))$, with variance

$$\begin{array}{l}{\sigma}_{6}{(\nu )}^{2}={(\frac{\partial R(\nu )}{\partial \nu})}^{2}\mathit{\text{var}}(\sqrt{n}(\widehat{F}({F}^{-1}(\nu ))-\nu ))+{(\frac{\partial R(\nu )}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial R(\nu )}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R\nu}{\partial \nu})\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}({F}^{-1}(\nu ))-\nu ))(\frac{\partial R(\nu )}{\partial \theta});\end{array}$$

(vii)
$\sqrt{n}(\widehat{R}({\nu}_{T}(t))-R({\nu}_{T}(t)))$, where TPR=1−*ν _{T}* (

$$\begin{array}{l}{\sigma}_{7}{(t)}^{2}={(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{2}\mathit{\text{var}}(\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R({\nu}_{T}(t))}{\partial t})\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta});\end{array}$$

(viii)
$\sqrt{n}(\widehat{R}({\nu}_{F}(t))-R({\nu}_{F}(t)))$, where FPR=1*−**ν _{F}* (

$$\begin{array}{l}{\sigma}_{8}{(t)}^{2}={(\frac{\partial R({\nu}_{F}(t))}{\partial t})}^{2}\mathit{\text{var}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({F}_{\overline{D}}^{-1}({\nu}_{F}(t)))-t))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{(\frac{\partial R({\nu}_{F}(t))}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial R({\nu}_{F}(t))}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R({\nu}_{F}(t))}{\partial t})\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{\overline{D}}({F}_{\overline{D}}^{-1}({\nu}_{F}(t)))-t))(\frac{\partial R({\nu}_{F}(t))}{\partial \theta});\end{array}$$

(ix) $\sqrt{n}(P\widehat{E}V-PEV)$, with variance

$$\begin{array}{l}{\sigma}_{9}^{2}=\frac{\mathit{\text{var}}(G(\theta ,{Y}_{D}))}{{n}_{D}/n}+\frac{\mathit{\text{var}}(G(\theta ,{Y}_{\overline{D}}))}{{n}_{\overline{D}}/n}+(\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{D}(y)-\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{\overline{D}}{(y))}^{T}\times \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))\times (\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{D}(y)-\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{\overline{D}}(y))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2(\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{D}(y)-\int \frac{\partial G(\theta ,y)}{\partial \theta}d{F}_{\overline{D}}{(y))}^{T}\times \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\{\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}\int G(\theta ,Y)d({\widehat{F}}_{D}(Y)-{F}_{D}(Y)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}\int G(\theta ,Y)d({\widehat{F}}_{\overline{D}}(Y)-{F}_{\overline{D}}(Y)))\};\end{array}$$

(x) $\sqrt{n}(\widehat{\overline{TG}}-\overline{TG})$, with variance

$${\sigma}_{10}^{2}={\mathrm{\Sigma}}_{1}-2{\mathrm{\Sigma}}_{1,2}+{\mathrm{\Sigma}}_{2},$$

where

$${\mathrm{\Sigma}}_{1}=\mathit{\text{var}}(\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho ))),$$

(12)

$${\mathrm{\Sigma}}_{2}=\mathit{\text{var}}(\sqrt{n}(F\widehat{P}R(\rho )-FPR(\rho ))),$$

(13)

$${\mathrm{\Sigma}}_{1,2}=\mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho )),\sqrt{n}(F\widehat{P}R(\widehat{\rho})-FPR(\rho ))).$$

(14)

We performed simulation studies to investigate the validity of using large sample theory for making inference in finite sample studies, and to compare it with inference using bootstrap resampling. Data were simulated under a linear logistic risk model. Specifically we employed a population prevalence of *ρ* = 0.2 and generated marker data according to *Y _{D̄}*

Simulation studies were conducted for case-control study designs as well as for cohort study designs. For the case-control scenario, we simulated nested case-control samples within the main study cohort employing equal numbers of cases and controls and with size of the cohort equal to 5 times that of the case-control study. The estimator is calculated from the main study cohort and sampling variability in summary estimates due to is acknowledged in making inference. Separate resampling of cases and controls was done for the nested case-control scenarios.

A full reporting of our simulation results can be found in Gu and Pepe (2009a). Tables 1 and and22 display results for a subset of the summary indices under case-control study designs. Huang and Pepe (2008a) report extensive simulation results for estimates of points on the predictiveness curve, *R*(*ν*) and *R ^{−}*

Cystic fibrosis is an inherited chronic disease that affects the lungs and digestive system of people. A defective gene and its protein product cause the body to produce unusually thick, sticky mucus which clogs the lungs and leads to life-threatening lung infections, and also obstructs the pancreas and stops natural enzymes from helping the body break down and absorb food. The main culminating event that leads to death is acute pulmonary exacerbations, i.e. lung infection requiring intravenous antibiotics.

The data for analysis here is from the Cystic Fibrosis Registry, a database maintained by the Cystic Fibrosis Foundation that contains annually updated information on over 20,000 people diagnosed with CF and living in the USA. In order to illustrate our methodology, we consider FEV_{1}, a measure of lung function, and weight, a measure of nutritional status, as measured in 1995 to predict occurrence of pulmonary exacerbation in 1996. Data from 12,802 patients 6 years of age and older are analyzed. 5,245 subjects (41%) had at least one pulmonary exacerbation. A child’s weight is standardized for age and gender by reporting his/her placement value, which is equal to 1 minus his/her percentile value, in a healthy population of children of the same age and gender (Hamill et al. 1977), while FEV_{1} is standardized for age, gender and height in a different way, explicit formulae were provided by Knudson et al. (1983). We modelled the risk functions using logistic regression models with weight and FEV_{1} entered the model as linear terms, and both are negated to satisfy the assumption that increasing values are associated with increasing risk. Figure 1 shows the predictiveness curves for the entire cohort and Figure 2 shows the risk distributions separately for cases (those who had a pulmonary exacerbation) and for controls.

First, we use the entire cohort to estimate predictiveness summary measures for weight and lung function. Table 3 shows the point estimates discussed earlier in sections 2 and 3. Here we provide confidence intervals based on asymptotic distribution theory and on bootstrap resampling. Observe that standard deviations are all small and that corresponding confidence intervals are very tight. Bootstrap confidence intervals are almost identical to those based on asymptotic theory.

Point estimates and 95% confidence intervals for the summary indices using FEV_{1} and weight as markers of risk for subsequent pulmonary exacerbation in patients with cystic fibrosis. Results based on the entire cohort.

We used the summary indices as the basis for hypothesis tests to formally compare the predictive capacities of FEV_{1} and weight. The difference between estimates of the indices was calculated and standardized using a bootstrap estimate of the variance of the difference. By comparing these test statistics with quantiles of the standard normal distribution, *p*-values were calculated. We see that differences between lung function and weight as predictive markers are statistically significant, no matter what summary index is employed. Note however that each test relates to a different question about predictive performance, depending on the particular summary index on which it is based. Asking if PEVs for weight and lung function are equal is not the same as asking if the proportion of subjects whose risks are less than 0.25, *R ^{−}*

Next, we randomly sampled 1,280 cases and 1,280 controls from the entire cohort to form a nested case-control study sample that is about 1/5 th the size of the cohort. Table 4 presents results that use data on FEV_{1} and weight from the case-control subset and the estimate of the overall incidence of pulmonary exacerbation from the entire cohort, = 0.41. Estimates of summary indices are very close to the cohort estimates but corresponding confidence intervals are much wider. Nevertheless conclusions remain the same. This demonstrates that in a study where predictive markers are costly to obtain, the nested case-control design could yield considerable cost savings.

Point estimates and 95% confidence intervals for the summary indices using FEV_{1} and weight as markers of risk for subsequent pulmonary exacerbation in patients with cystic fibrosis. Results based on prevalence estimated from the entire cohort and marker **...**

Predictiveness summary measures, such as *R*(*ν*) and *R ^{−}*

This paper presents some new clinically relevant measures that quantify the predictive performance of a risk marker. New measures formally defined include TPR(*p*), FPR(*p*), PPV(*p*), NPV(*p*), *R*(*ν _{T}*),

A second contribution of this paper is to provide distribution theory for estimators of the summary indices. Such has not been available for most of the measures heretofore, including the popular PEV measure. Our methods can now be used to construct confidence intervals for the summary indices. Simulation studies indicate that the methods are valid for application in practice with finite samples.

We also demonstrated in an example how summary indices can be used to make formal rigorous comparisons between markers. Such has only been possible previously for comparisons based on the AUC or on point estimates of the predictiveness curve, *R*(*ν*) and *R _{−}*

Our methods accommodate cohort or case-control study designs. The latter are particularly important in the early phases of biomarker development when biomarker assays are expensive or procurement of biological samples is difficult (Pepe et al. 2001). In such settings nested case-control studies are much more feasible (Baker et al. 2002; Pepe et al. 2008d). Our methodology is currently restricted to risk models that include a single marker or a predefined combination of markers. In practice studies often involve development of a marker combination and assessment of its performance. Compelling arguments have been provided in the literature for splitting a dataset into training and test subsets (Simon 2006; Ransohoff 2004). In these circumstances our methods apply to evaluation with the test data of the combination developed with the training data. It would be worthwhile however to explore use of cross validation techniques for simultaneous development and assessment of risk models using the summary indices we have described.

Which summary index should be recommended for use in practice? In our opinion, a summary index should not replace the display of the risk distributions (Figures 1 and and2)2) but should serve only to complement them. The choice of summary indices to report should be driven by the scientific objectives of the study. For example, if the objective is to risk stratify the population according to some risk threshold, below which treatment is not indicated and above which treatment is indicated, the corresponding proportions of the population that fall into the two risk strata, *R ^{−}*

The final stages of evaluating a model for use in practice should incorporate notions of costs and benefits (i.e. utilities) that may be associated with decisions based on *risk*(*Y*). However, specifying costs and benefits is typically very difficult in practice. Vickers and Elkin (2006) have recently proposed a standardized measure of expected utility for a prediction model that does not require explicit specifications of costs and benefits. The net benefit at risk threshold *p* is defined as *NB*(*p*) = *ρTPR*(*p*) + (1 *− ρ*)*FPR*(*p*)*p/*(1 *− p*), and their decision curve plots *NB*(*p*) versus *p*. This weighted average of true and false positive rates could complement descriptive plots of risk distributions. Moreover, the methods for inference that we have presented here give rise to methods for inference about decision curves too.

To simplify notation, we suppose the risk model is logistic linear in *Y* :

$$\text{logit}\{G(\theta ,Y)\}={\theta}_{0}+{\theta}_{1}Y.$$

In a cohort study the log likelihood function is

$$l(\theta |{Y}_{1},\dots ,{Y}_{n})=\sum _{i=1}^{{n}_{\hspace{0.17em}D}}\text{log}\frac{\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i})}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i})}+\sum _{i=1}^{{n}_{\hspace{0.17em}\overline{D}}}\text{log}\frac{1}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i})}.$$

(15)

Let _{0}, _{1} be the maximum likelihood estimators (MLE) of *θ*_{0}, *θ*_{1} based on (15). The following results are well known.

**Result 1** Let

$$\begin{array}{l}{A}_{0}(\theta ,t)={\int}_{-\infty}^{t}\frac{\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{{(1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y))}^{2}}dF(y)\\ {A}_{1}(\theta ,t)={\int}_{-\infty}^{t}\frac{y\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{{(1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y))}^{2}}dF(y)\\ {A}_{2}(\theta ,t)={\int}_{-\infty}^{t}\frac{{y}^{2}\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{{(1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y))}^{2}}dF(y)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}A(\theta ,t)=\left[\begin{array}{ll}{A}_{0}(\theta ,t)\hfill & {A}_{1}{(\theta ,t)}^{T}\hfill \\ {A}_{1}(\theta ,t)\hfill & {A}_{2}(\theta ,t)\hfill \end{array}\right],\end{array}$$

and *A*(*θ*) = *A*(*θ, ∞*). If *A*(*θ*)^{−}^{1} exists,

$$\sqrt{n}\left[\begin{array}{ll}{\widehat{\theta}}_{0}\hfill & -{\widehat{\theta}}_{0}\hfill \\ {\widehat{\theta}}_{1}\hfill & -{\widehat{\theta}}_{1}\hfill \end{array}\right]{\to}_{d}N(0,A{(\theta )}^{-1}).$$

We can also write
$\sqrt{n}(\widehat{\theta}-\theta )=\frac{1}{\sqrt{n}}{\sum}_{i=1}^{n}{\ell}_{\theta}({Y}_{i})+{o}_{p}(1)$, where * _{θ}*(

$${i}_{\theta}({Y}_{i})=\left[\begin{array}{l}\partial l(\theta |{Y}_{i})/\partial {\theta}_{0}\hfill \\ \partial l(\theta |{Y}_{i})/\partial {\theta}_{1}\hfill \end{array}\right]=\left[\begin{array}{c}{D}_{i}-\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i})/(1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i}))\\ {D}_{i}{Y}_{i}-{Y}_{i}\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i})/(1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}{Y}_{i}))\end{array}\right].$$

**Result 2** As *n → ∞,*

$$\begin{array}{l}\sqrt{n}(\widehat{\rho}-\rho ){\to}_{d}N(0,\rho (1-\rho )),\\ \sqrt{n}(\widehat{F}(t)-F(t)){\to}_{d}N(0,F(t)(1-F(t))),\\ \sqrt{n}({\widehat{F}}_{D}(t)-{F}_{D}(t)){\to}_{d}N(0,{F}_{D}(t)(1-{F}_{D}(t))/\eta ),\\ \sqrt{n}({\widehat{F}}_{\overline{D}}(t)-{F}_{\overline{D}}(t)){\to}_{d}N(0,{F}_{\overline{D}}(t)\hspace{0.17em}(1-{F}_{\overline{D}}(t))/(1-\eta )),\end{array}$$

where *η* *n _{D}*/

**Lemma 1** Let

$$\begin{array}{c}{B}_{D,0}(t)={\int}_{-\infty}^{t}\frac{1}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{D}(y)\\ {B}_{D,1}(t)={\int}_{-\infty}^{t}\frac{y}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{D}(y)\\ {B}_{\overline{D},0}(t)={\int}_{-\infty}^{t}\frac{\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{\overline{D}}(y)\\ {B}_{\overline{D},1}(t)={\int}_{-\infty}^{t}\frac{y\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{\overline{D}}(y),\end{array}$$

and use *B _{D,}*

Then we have

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}(t)-F(t))\hfill \\ =A{(\theta )}^{-1}\left[\begin{array}{l}\rho {B}_{D,0}(t)-(1-\rho ){B}_{\overline{D},0}(t)\hfill \\ \rho {B}_{D,1}(t)-(1-\rho ){B}_{\overline{D},1}(t)\hfill \end{array}\right]\hfill \\ \hfill \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}(t)-{F}_{D}(t))\hfill \\ =A{(\theta )}^{-1}\left[\begin{array}{l}{B}_{D,0}(t)-{F}_{D}(t){B}_{D,0}\hfill \\ {B}_{D,1}(t)-{F}_{D}(t){B}_{D,1}\hfill \end{array}\right]\hfill \\ \hfill \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{\overline{D}}(t)-{F}_{\overline{D}}(t))\hfill \\ =A{(\theta )}^{-1}\left[\begin{array}{l}-{B}_{\overline{D},0}(t)+{F}_{\overline{D}}(t){B}_{\overline{D},0}\hfill \\ -{B}_{\overline{D},1}(t)-{F}_{\overline{D}}(t){B}_{\overline{D},1}\hfill \end{array}\right]\hfill \end{array}$$

Proof:

We prove the first result and proofs of the other two results follow from similar arguments.

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}(t)-F(t))\\ =\mathit{\text{cov}}(\frac{1}{\sqrt{n}}\sum _{i=1}^{n}A{(\theta )}^{-1}{i}_{\theta}({Y}_{i}),\frac{1}{\sqrt{n}}\sum _{i=1}^{n}(I({Y}_{i}\le t)-F(t)))\\ =\mathit{\text{cov}}(A{(\theta )}^{-1}{i}_{\theta}(Y),I(Y\le t)-F(t)))\\ =A{(\theta )}^{-1}E({i}_{\theta}(Y)\times I(Y\le t))\\ =A{(\theta )}^{-1}\{\rho E({i}_{\theta}({Y}_{D})\times I({Y}_{D}\le t))+(1-\rho )E({i}_{\theta}({Y}_{\overline{D}})\times I({Y}_{\overline{D}}\le t))\}\\ =A{(\theta )}^{-1}\left[\begin{array}{l}\rho {B}_{D,0}(t)-(1-\rho ){B}_{\overline{D},0}(t)\hfill \\ \rho {B}_{D,1}(t)-(1-\rho ){B}_{\overline{D},1}(t)\hfill \end{array}\right]\end{array}$$

** Proof of Theorem items (i), (ii) and (iii)** We show the proof for item (i). The proofs for items (ii) and (iii) follow similar arguments.

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}({\widehat{R}}^{-1}(p)-{R}^{-1}(p))=\sqrt{n}(\widehat{F}({G}^{-1}(\widehat{\theta},p))-F({G}^{-1}(\theta ,p)))\\ =\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p)))+\sqrt{n}(F({G}^{-1}(\widehat{\theta},p))-F({G}^{-1}(\theta ,p)))+{R}_{n},\end{array}$$

where

$${R}_{n}=\sqrt{n}(\widehat{F}({G}^{-1}(\widehat{\theta},p))-\widehat{F}({G}^{-1}(\theta ,p)))-\sqrt{n}(F({G}^{-1}(\widehat{\theta},p))-F({G}^{-1}(\theta ,p)))={o}_{p}(1)$$

by equicontinuity of the process $\sqrt{n}(\widehat{F}-F)$. Earlier results and the delta method then imply:

$$\begin{array}{l}{\sigma}_{1}{(p)}^{2}=\mathit{\text{var}}(\sqrt{n}({\widehat{R}}^{-1}(p)-{R}^{-1}(p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{var}}(\sqrt{n})(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{var}}(\sqrt{n}(F({G}^{-1}(\widehat{\theta},p))-F({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2\mathit{\text{cov}}\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p))),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}(F({G}^{-1}(\widehat{\theta},p))-F({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={R}^{-1}(p)(1-{R}^{-1}(p))+{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}A{(\theta )}^{-1}(\frac{\partial {R}^{-1}(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}\rho {B}_{D,0}({G}^{-1}(\theta ,p))-(1-\rho ){B}_{\overline{D},0}({G}^{-1}(\theta ,p))\hfill \\ \rho {B}_{D,1}({G}^{-1}(\theta ,p))-(1-\rho ){B}_{\overline{D},1}({G}^{-1}(\theta ,p))\hfill \end{array}\right].\end{array}$$

(16)

The last equality follows from Result 2 (for variance of ), Result 1 (for variance of ) and Lemma 1 (for covariance of (*, *)).

*Proof of Theorem items (iv) and (v)*

We write

$$P\widehat{P}V(p)=\frac{\widehat{\rho}}{1-\widehat{\rho}}\frac{T\widehat{P}R(p)}{F\widehat{P}R(p)},$$

The asymptotic distribution of $\sqrt{n}(\widehat{\rho}-\rho )$ is given in Result 2 as are the distributions of $\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p))$ and $\sqrt{n}(F\widehat{P}R(p)-FPR(p))$ because:

$$\begin{array}{l}\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p))=\sqrt{n}((1-{\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},p)))-(1-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))).\end{array}$$

And similarly,

$$\sqrt{n}(F\widehat{P}R(p)-FPR(p))=-\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))).$$

In the following, we calculate the covariances between (
$\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(\widehat{\rho}-\rho )$), (
$\sqrt{n}(F\widehat{P}R(p)-FPR(p)),\sqrt{n}(\widehat{\rho}-\rho )$) and (
$\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(F\widehat{P}R(p)-FPR(p))$). The asymptotic variance of
$\sqrt{n}(P\widehat{P}V(p)-PPV(p))$, *σ*_{4}(*p*)^{2}, then follows from the delta method.

Consider the covariance between $\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p))$ and $\sqrt{n}(F\widehat{P}R(p)-FPR(p))$:

$$\begin{array}{l}{\mathit{\text{cov}}}_{1}=\mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(F\widehat{P}R(p)-FPR(p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p)))+{o}_{p}(1),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\sqrt{n}({F}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p)))+{o}_{p}(1)),\end{array}$$

Because * _{D}* and

$$\begin{array}{l}{\mathit{\text{cov}}}_{1}=\mathit{\text{cov}}(\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}({F}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))),\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}({F}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial FPR(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\theta}-\theta ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial \mathit{\text{FPR}}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\theta}-\theta ))\end{array}$$

(17)

$$\begin{array}{l}={(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}A{(\theta )}^{-1}(\frac{\partial FPR(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}-{B}_{\overline{D},0}({G}^{-1}(\theta ,p))+(1-FPR(p)){B}_{\overline{D},0}\hfill \\ -{B}_{\overline{D},1}({G}^{-1}(\theta ,p))+(1-FPR(p)){B}_{\overline{D},1}\hfill \end{array}\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial FPR(p)}{\partial \theta})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}{B}_{D,0}({G}^{-1}(\theta ,p))-(1-\mathit{\text{TPR}}(p)){B}_{D,0}\hfill \\ {B}_{D,1}({G}^{-1}(\theta ,p))-(1-\mathit{\text{TPR}}(p)){B}_{D,1}\hfill \end{array}\right]\end{array}$$

(18)

The last equality uses Result 1 (for variance of ) and Lemma 1 (for covariance of (_{D}*, *) and (_{D̄}*, *)).

The second covariance (equation (10)) is between $\sqrt{n}(\widehat{\rho}-\rho )$; and $\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p))$:

$$\begin{array}{l}{\mathit{\text{cov}}}_{2}=\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p)))+\sqrt{n}({F}_{D}({G}^{-1}(\theta ,\widehat{p}))-{F}_{D}({G}^{-1}(\theta ,p))))\hspace{0.17em}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\mathit{\text{cov}}\hspace{0.17em}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\equiv -({A}_{n}+{B}_{n}),\end{array}$$

where ${A}_{n}=\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))))$ and ${B}_{n}=\mathit{\text{cov}}(\sqrt{n}(\rho -\rho ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))$.

Observe that

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}(\widehat{\rho}-\rho )=\sqrt{n}(\int G(\widehat{\theta},Y)d\widehat{F}(Y)-\int G(\theta ,Y)dF(Y))\\ =\sqrt{n}(\int (G(\widehat{\theta}-Y)-G(\theta ,Y))dF(Y)+\sqrt{n}\int G(\theta ,Y)d(\widehat{F}(Y)-F(Y))+{H}_{n}\\ =(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )\sqrt{n}(\widehat{\theta}-\theta )+\sqrt{n}\int G(\theta ,Y)d(\widehat{F}(Y)-F(Y))+{H}_{n},\end{array}$$

where *R*(*v*) * G(θ, Y)* and
${H}_{n}\equiv \sqrt{n}\int (G(\widehat{\theta},Y)-G(\theta ,Y))d(\widehat{F}(Y)-F(Y))=\frac{1}{\sqrt{n}}\int \sqrt{n}(G(\widehat{\theta},Y)-G(\theta ,Y))d(\sqrt{n}(\widehat{F}(Y)-F(Y)))$. Both
$\sqrt{n}(G(\widehat{\theta},Y)-G(\theta ,Y))$ and
$\sqrt{n}(\widehat{F}(Y)-F(Y))$ are bounded in probability and therefore *H _{n}* converges to 0 as

We next derive expressions for *A _{n}* and

$$\begin{array}{l}{A}_{n}=\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}(\widehat{\theta}-\theta ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}\{\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))\int \frac{\partial R(\nu )}{\partial \theta}d\nu \hspace{0.17em}+\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}\int G(\theta ,Y)d(\widehat{F}(Y)-F(Y)),\sqrt{n}(\widehat{\theta}-\theta ))\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}\{\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))\int \frac{\partial R(\nu )}{\partial \theta}d\nu +\mathit{\text{cov}}(1/\sqrt{n}\sum _{i=1}^{n}G(\theta ,{Y}_{i}),1/\sqrt{n}\sum _{i=1}^{n}A{(\theta )}^{-1}{i}_{\theta}({Y}_{i}))\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}A{(\theta )}^{-1}(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )+(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})A{(\theta )}^{-1}\mathit{\text{cov}}(G(\theta ,Y),{i}_{\theta}(Y))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}A{(\theta )}^{-1}(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{(\frac{\partial (1-\mathit{\text{TPR}}(p))}{\partial \theta})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}\rho \int \frac{G(\theta ,y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{D}(y)+(1-\rho )\int \frac{-G(\theta ,Y)\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{\overline{D}}(y)\hfill \\ \rho \int \frac{yG(\theta ,y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{D}(y)+(1-\rho )\int \frac{-yG(\theta ,Y)\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}y)}d{F}_{\overline{D}}(y)\hfill \end{array}\right]\end{array}$$

(19)

And *B _{n}* is

$$\begin{array}{l}{B}_{n}=\mathit{\text{cov}}(\sqrt{n}(\widehat{\rho}-\rho ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\int G(\theta ,Y)d(\sqrt{n}(\widehat{F}(Y)-F(Y)),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{cov}}(1/\sqrt{n})\sum _{i=1}^{n}G(\theta ,{Y}_{i}),\sqrt{n}/{n}_{D}\sum _{i=1}^{{n}_{\hspace{0.17em}D}}I({Y}_{Di}\le {G}^{-1}(\theta ,p)))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\int \frac{\partial R(\nu )}{\partial \theta}d\nu )}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+({\int}_{-\infty}^{G-{1}_{(\theta ,p)}}G(\theta ,y)d{F}_{D}(Y)-{F}_{D}({G}^{-1}(\theta ,p))\int G(\theta ,Y)d{F}_{D}(Y)),\end{array}$$

(20)

where *cov* (
$\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p)))$) is given by Lemma 1.

Combining the two terms yields a value for *cov*_{2}. The derivation of *cov*_{3} follows from a similar argument.

The proof of item (v) of the Theorem uses exactly the same techniques.

** Proof of Theorem items (vi), (vii) and (viii)** We prove Theorem item (vii) in the following. Proofs of (vi) and (viii) are similar. The following proof is based on Huang and Pepe (2008

When the value of TPR is 1 *−* *ν _{T}* (

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}(\widehat{R}{\nu}_{T}(t))-R({\nu}_{T}(t)))=\sqrt{n}(G(\widehat{\theta},{\widehat{F}}_{D}^{-1}({\nu}_{T}(t)))-G(\theta ,{F}_{D}^{-1}({\nu}_{T}(t))))\\ =(\frac{\partial G(\theta ,{F}_{D}^{-1}({\nu}_{T}(t)))}{\partial {F}_{D}^{-1}({\nu}_{T}(t))})\sqrt{n}({\widehat{F}}_{D}^{-1}({\nu}_{T}(t))-{F}_{D}^{-1}({\nu}_{T}(t)))+{(\frac{\partial G(\theta ,{F}^{-1}({\nu}_{T}(t)))}{\partial \theta})}^{T}\sqrt{n}(\widehat{\theta}-\theta )+{o}_{p}(1)\\ =-(\frac{\partial R({\nu}_{T}(t))}{\partial t})\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t)+{(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})}^{T}\sqrt{n}(\widehat{\theta}-\theta )+{o}_{p}(1)\end{array}$$

It follows that the asymptotic variance is

$$\begin{array}{l}{\sigma}_{7}{(t)}^{2}=\mathit{\text{var}}(\sqrt{n}(\widehat{R}({\nu}_{T}(t))-R({\nu}_{T}(t))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{2}\mathit{\text{var}}(\sqrt{n}({F}_{\widehat{D}}({F}_{D}^{-1}({\nu}_{T}(t)))-(t))+{(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{T}\mathit{\text{var}}(\sqrt{n}(\theta -\widehat{\theta}))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R({\nu}_{T}(t))}{\partial t})\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta}).\end{array}$$

(21)

The variance of $\sqrt{n}(\widehat{\theta}-\theta )$ and of $\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t)$ are provided in Result 2, and their covariance is provided in Lemma 1. Putting them all together, we have the following result,

$$\begin{array}{l}{\sigma}_{7}{(t)}^{2}={(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{2}{\nu}_{T}(t)(1-{\nu}_{T}(t))/\eta +{(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})}^{T}A{(\theta )}^{-1}(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R({\nu}_{T}(t))}{\partial t}){(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}{B}_{D,0}({F}_{D}^{-1}({\nu}_{T}(t)))-{\nu}_{T}(t){B}_{\overline{D},0}({F}_{D}^{-1}({\nu}_{T}(t)))\hfill \\ {B}_{D,1}({F}_{D}^{-1}({\nu}_{T}(t)))-{\nu}_{T}(t){B}_{\overline{D},1}({F}_{D}^{-1}({\nu}_{T}(t)))\hfill \end{array}\right]\end{array}$$

*Proof of Theorem item (ix)*

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}(P\widehat{E}V-PEV)\\ =\sqrt{n}\{(\int G(\widehat{\theta},Y)d{\widehat{F}}_{D}-\int G(\widehat{\theta},Y)d{\widehat{F}}_{\overline{D}})-(\int G(\theta ,Y)d{F}_{D}-\int G(\theta ,Y)d{F}_{\overline{D}})\}\\ =\sqrt{n}\{(\int G(\widehat{\theta},Y)d{\widehat{F}}_{D}-\int G(\theta ,Y)d{F}_{D})-(\int G(\widehat{\theta},Y)d{\widehat{F}}_{\overline{D}}-\int G(\theta ,Y)d{F}_{\overline{D}})\}\\ =\{\sqrt{n}(\int G(\theta ,Y)d({\widehat{F}}_{D}-{F}_{D}))+\sqrt{n}(\int G(\widehat{\theta},Y)-G(\theta ,Y))d{F}_{D}\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\{\sqrt{n}(\int G(\theta ,Y)d({\widehat{F}}_{\overline{D}}-{F}_{\overline{D}}))+\sqrt{n}(\int (G(\widehat{\theta},Y)-G(\theta ,Y))d{F}_{\overline{D}})\}+{P}_{n}\\ \equiv ({A}_{n}+{B}_{n})-({C}_{n}+{D}_{n})+{P}_{n},\end{array}$$

where
${P}_{n}=\sqrt{n}\int (G(\widehat{\theta},Y)-G(\theta ,Y))d({\widehat{F}}_{D}-{F}_{D})+\sqrt{n}\int (G(\widehat{\theta},Y)-G(\theta ,Y))d({\widehat{F}}_{\overline{D}}-{F}_{\overline{D}})$. It is easy to see that *P _{n}* converges to zero as

Now we have,

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{var}}({A}_{n}+{B}_{n})=\mathit{\text{var}}({A}_{n})+\mathit{\text{var}}({B}_{n})+2\mathit{\text{cov}}({A}_{n},{B}_{n})\\ =\mathit{\text{var}}(\frac{1}{\sqrt{n}}\frac{1}{\sqrt{{n}_{D}}}\sum _{i=1}^{{n}_{\hspace{0.17em}D}}G(\theta ,{Y}_{Di}))+\mathit{\text{var}}({(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))}^{T}\sqrt{n}(\widehat{\theta}-\theta )\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2\hspace{0.17em}{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))}^{T}\mathit{\text{cov}}(\sqrt{n}(\int G(\theta ,Y)d({\widehat{F}}_{D}-{F}_{D})),\sqrt{n}(\widehat{\theta},\theta ))\\ =\mathit{\text{var}}(G(\theta ,{Y}_{D}))/\eta +(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}{(Y))}^{T}A{(\theta )}^{-1}(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}{(Y))}^{T}\mathit{\text{cov}}(G(\theta ,{Y}_{D}),A{(\theta )}^{-1}{i}_{0}({Y}_{D}))\\ =\mathit{\text{var}}(G(\theta ,{Y}_{D}))/\eta +{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))}^{T}A{(\theta )}^{-1}(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D}(Y))}^{T}A{(\theta )}^{-1}\left[\begin{array}{c}{P}_{1}\\ {P}_{2}\end{array}\right],\end{array}$$

(22)

where

$$\begin{array}{l}\left[\begin{array}{c}{P}_{1}\\ {P}_{2}\end{array}\right]\equiv \left[\begin{array}{l}\int G(\theta ,Y)(\partial l(\theta |{Y}_{D})/\partial {\theta}_{0})d{F}_{D}(Y)-\int G(\theta ,Y)d{F}_{D}(Y)\int (\partial l(\theta |{Y}_{D})/\partial {\theta}_{0})d{F}_{D}(Y)\hfill \\ \int G(\theta ,Y)(\partial l(\theta |{Y}_{D})/\partial {\theta}_{1})d{F}_{D}(Y)-\int G(\theta ,Y)d{F}_{D}(Y)\int (\partial l(\theta |{Y}_{D})/\partial {\theta}_{0})d{F}_{D}(Y)\hfill \end{array}\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\left[\begin{array}{l}\int \frac{G(\theta ,Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{D}(Y)-\int G(\theta ,Y)d{F}_{D}(Y)\int \frac{1}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{D}(Y)\hfill \\ \int \frac{YG(\theta ,Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{D}(Y)-\int G(\theta ,Y)d{F}_{D}(Y)\int \frac{Y}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{D}(Y)\hfill \end{array}\right].\end{array}$$

(23)

From a similar argument,

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{var}}({C}_{n}+{D}_{n})=\mathit{\text{var}}({C}_{n})+\mathit{\text{var}}({D}_{n})+2\mathit{\text{cov}}({C}_{n},{D}_{n})\\ =\mathit{\text{var}}(G(\theta ,{Y}_{\overline{D}}))/(1-\eta )+{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{\overline{D}})}^{T}A{(\theta )}^{-1}(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{\overline{D}})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{\overline{D}})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}{Q}_{1}\hfill \\ {Q}_{2}\hfill \end{array}\right],\end{array}$$

(24)

where

$$\begin{array}{l}\left[\begin{array}{l}{Q}_{1}\hfill \\ {Q}_{2}\hfill \end{array}\right]\equiv \left[\begin{array}{l}-\int G(\theta ,Y)(\partial l(\theta |{Y}_{\overline{D}})/\partial {\theta}_{0})d{F}_{\overline{D}}(Y)+\int G(\theta ,Y)d{F}_{\overline{D}}(Y)\int (\partial l(\theta |{Y}_{\overline{D}})/\partial {\theta}_{0})d{F}_{\overline{D}}(Y)\hfill \\ -\int G(\theta ,Y)(\partial l(\theta |{Y}_{\overline{D}})/\partial {\theta}_{1})d{F}_{\overline{D}}(Y)+\int G(\theta ,Y)d{F}_{\overline{D}}(Y)\int (\partial l(\theta |{Y}_{\overline{D}})/\partial {\theta}_{1})d{F}_{\overline{D}}(Y)\hfill \end{array}\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\left[\begin{array}{l}-\int \frac{G(\theta ,Y)\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{\overline{D}}(Y)+\int G(\theta ,Y)d{F}_{\overline{D}}(Y)\int \frac{\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{\overline{D}}(Y)\hfill \\ -\int \frac{YG(\theta ,Y)\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{\overline{D}}(Y)+\int G(\theta ,Y)d{F}_{\overline{D}}(Y)\int \frac{Y\hspace{0.17em}\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}{1+\mathit{\text{exp}}({\theta}_{0}+{\theta}_{1}Y)}d{F}_{\overline{D}}(Y)\hfill \end{array}\right].\end{array}$$

(25)

Because * _{D}* and

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}({A}_{n}+{B}_{n},{C}_{n}+{D}_{n})=\mathit{\text{cov}}({A}_{n},{D}_{n})+\mathit{\text{cov}}({B}_{n},{C}_{n})+\mathit{\text{cov}}({B}_{n},{D}_{n})\\ ={(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D})}^{T}A{(\theta )}^{-1}(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{\overline{D}})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{D})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}{Q}_{1}\hfill \\ {Q}_{2}\hfill \end{array}\right]+{(\int \frac{\partial G(\theta ,Y)}{\partial \theta}d{F}_{\overline{D}})}^{T}A{(\theta )}^{-1}\left[\begin{array}{l}{P}_{1}\hfill \\ {P}_{2}\hfill \end{array}\right]\end{array}$$

(26)

The asymptotic variance of
$\sqrt{n}(\text{P}\widehat{\text{E}}\text{V}-PEV)$,
${\sigma}_{9}^{2}$, can be obtained by combining *var*(*A _{n}* +

*Proof of Theorem item (x)*

$$\begin{array}{l}\sqrt{n}(\widehat{\overline{TG}}-\overline{TG})=\sqrt{n}\{(T\widehat{P}R(\widehat{\rho})-F\widehat{P}R(\widehat{\rho}))-(\mathit{\text{TPR}}(\rho )-FPR(\rho ))\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho ))-\sqrt{n}(F\widehat{P}R(\widehat{\rho})-FPR(\rho )).\end{array}$$

The result in the Theorem follows. Now we derive expressions for the variance components in a cohort study. Observe that

$$\begin{array}{l}-\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho ))\\ =\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{F}_{D}({G}^{-1}(\theta ,\rho )))\\ =\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{\widehat{F}}_{D}({G}^{-1}(\theta ,\rho )))-\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{F}_{D}({G}^{-1}(\theta ,\rho )))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,\rho ))-{F}_{D}({G}^{-1}(\theta ,\rho )))+\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta}-\widehat{\rho}))-{F}_{D}({G}^{-1}(\theta ,\rho )))\\ =\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,\rho ))-{F}_{D}({G}^{-1}(\theta ,\rho )))+{f}_{D}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \theta}\sqrt{n}(\widehat{\theta}-\theta )\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{f}_{D}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \rho}\sqrt{n}(\widehat{\rho}-\rho )+{o}_{p}(1)\\ \equiv {A}_{n}+{B}_{n}+{C}_{n}+{o}_{p}(1),\end{array}$$

where we define

$$\begin{array}{l}{A}_{n}\equiv \sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,\rho ))-{F}_{D}({G}^{-1}(\theta ,\rho ))),\\ {B}_{n}\equiv {f}_{D}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \theta}\sqrt{n}(\widehat{\theta}-\theta ),\\ {C}_{n}\equiv {f}_{D}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \rho}\sqrt{n}(\widehat{\rho}-\rho ).\end{array}$$

and note that
$\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{\widehat{F}}_{D}({G}^{-1}(\theta ,\rho )))-\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{F}_{D}({G}^{-1}(\theta ,\rho )))\to 0$ as *n → ∞* due to the equicontinuity of the process.

$$\begin{array}{l}{\mathrm{\Sigma}}_{1}\equiv \mathit{\text{var}}(\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho )))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{var}}({A}_{n})+\mathit{\text{var}}({B}_{n})+\mathit{\text{var}}({C}_{n})+\mathit{\text{cov}}({A}_{n},{B}_{n})+\mathit{\text{cov}}({A}_{n},{C}_{n})+\mathit{\text{cov}}({B}_{n},{C}_{n})\end{array}$$

(27)

The variance of *B _{n}* follows from that of
$\sqrt{n}(\widehat{\theta}-\theta )$ given in Result 1, and the variances of

Similarly, we have

$$\begin{array}{l}-\sqrt{n}(F\widehat{P}R(\widehat{\rho})-FPR(\rho ))\\ =\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\widehat{\theta},\widehat{\rho}))-{F}_{\overline{D}}({G}^{-1}(\theta ,\rho )))\\ =\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,\rho ))-{F}_{\overline{D}}({G}^{-1}(\theta ,\rho )))+{f}_{\overline{D}}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \theta}\sqrt{n}(\widehat{\theta}-\theta )\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{f}_{\overline{D}}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \rho}\sqrt{n}(\widehat{\rho}-\rho )+{o}_{p}(1)\\ \equiv {D}_{n}+{E}_{n}+{F}_{n}+{o}_{p}(1).\\ \end{array}$$

(28)

The variance of
$\sqrt{n}(F\widehat{P}R(\widehat{\rho})-FPR(\rho ))$, Σ_{2} (equation (13)), depends on the variances and covariances of the three terms

$$\begin{array}{l}{D}_{n}\equiv \sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,\rho ))-{F}_{\overline{D}}({G}^{-1}(\theta ,\rho ))),\\ {E}_{n}\equiv {f}_{\overline{D}}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \theta}\sqrt{n}(\widehat{\theta}-\theta ),\\ {F}_{n}\equiv {f}_{\overline{D}}({G}^{-1}(\theta ,\rho ))\frac{\partial {G}^{-1}(\theta ,\rho )}{\partial \rho}\sqrt{n}(\widehat{\rho}-\rho ).\end{array}$$

The variance of *E _{n}* can be obtained by using Result 1, and the variances of

The asymptotic variance o $\widehat{\overline{\sqrt{n}}}(T\overline{G-}TG)$, ${\sigma}_{10}^{2}$ is therefore

$${\sigma}_{10}^{2}={\mathrm{\Sigma}}_{1}+{\mathrm{\Sigma}}_{2}-2{\mathrm{\Sigma}}_{1,2},$$

where ∑_{1}_{,}_{2} is

$$\begin{array}{l}{\mathrm{\Sigma}}_{1,2}\equiv \mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(\widehat{\rho})-\mathit{\text{TPR}}(\rho )),\sqrt{n}(F\widehat{P}R(\widehat{\rho})-FPR(\rho )))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{cov}}({A}_{n}+{B}_{n}+{C}_{n},{D}_{n}+{E}_{n}+{F}_{n})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{cov}}({A}_{n},{E}_{n})+\mathit{\text{cov}}({A}_{n},{F}_{n})+\mathit{\text{cov}}({B}_{n},{D}_{n})+\mathit{\text{cov}}({B}_{n},{E}_{n})+\mathit{\text{cov}}({B}_{n},{F}_{n})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{cov}}({C}_{n},{D}_{n})+\mathit{\text{cov}}({C}_{n},{E}_{n})+\mathit{\text{cov}}({C}_{n},{F}_{n}).\end{array}$$

(29)

All of these covariance terms can be obtained using corresponding Results and Lemmas: *cov*(*A _{n}*

Let be the estimate of disease prevalence *ρ* from a cohort independent of the case-control sample, or the parent cohort within which the case-control sample is nested. Assume the size of the cohort is *λ* times the size of the case-control sample. Denote

$$\begin{array}{l}\widehat{F}*(t)=\rho {\widehat{F}}_{D}(t)+(1-\rho ){\widehat{F}}_{\overline{D}}(t)\\ \widehat{\theta}*=\left[\begin{array}{l}{\widehat{\theta}}_{0}^{*}\hfill \\ {\widehat{\theta}}_{1}^{*}\hfill \end{array}\right]=\left[\begin{array}{c}{\widehat{\theta}}_{0s}+\text{log}\left(\frac{{n}_{\overline{D}}}{{n}_{D}}\frac{\rho}{1-\rho}\right)\\ {\widehat{\theta}}_{1s}\end{array}\right].\end{array}$$

The following results are well established.

**Result 3** Let

$$\begin{array}{l}{A}_{0}(t)={\int}_{-\infty}^{t}\frac{\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y)}{(1+k\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y))}d{F}_{\overline{D}}(y)\\ {A}_{1}(t)={\int}_{-\infty}^{t}\frac{y\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y)}{(1+k\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y))}d{F}_{\overline{D}}(y)\\ {A}_{2}(t)={\int}_{-\infty}^{t}\frac{{y}^{2}\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y)}{(1+k\mathit{\text{exp}}({\theta}_{0}^{*}+{\theta}_{1}^{*}y))}d{F}_{\overline{D}}(y)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}A(t)=\left[\begin{array}{ll}{A}_{0}(t)\hfill & {A}_{1}{(t)}^{T}\hfill \\ {A}_{1}(t)\hfill & {A}_{2}(t)\hfill \end{array}\right],\end{array}$$

where *k n _{D}/n_{D̄}* and

$$\sqrt{n}\left[\begin{array}{l}{\widehat{\theta}}_{0}^{*}-{\theta}_{0}^{*}\hfill \\ {\widehat{\theta}}_{1}^{*}-{\theta}_{1}^{*}\hfill \end{array}\right]{\to}_{d}N(0,{\mathrm{\Sigma}}^{-1}),$$

where

$$\mathrm{\Sigma}=\frac{1+k}{k}\{{A}^{-1}-\left[\begin{array}{cc}1+k& 0\\ 0& 0\end{array}\right]\}.$$

A proof can be found in Prentice and Pyke (1979), Qin and Zhang (1997) and Zhang (2000).

The next set of results, Results 4–7, have been proved by Huang and Pepe (2008* _{a}*).

**Result 4** As *n → ∞*,
$\sqrt{n}(\widehat{F}*(t)-F*(t))$ converges to a normal random variable with mean 0 and variance

**Result 5**

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\sigma}_{F*}^{2}={(1-\rho )}^{2}(1+k){F}_{\overline{D}}(t)(1-{F}_{\overline{D}}(t))+{\rho}^{2}\frac{1+k}{k}{F}_{D}(t)(1-{F}_{D}(t)).\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta *),\sqrt{n}({\widehat{F}}_{D}(t)-{F}_{D}(t)))=\frac{n}{{n}_{D}}\{{A}^{-1}\left[\begin{array}{l}{A}_{0}(t)\hfill \\ {A}_{1}(t)\hfill \end{array}\right]-\left[\begin{array}{c}{F}_{D}(t)\\ 0\end{array}\right]\},\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta *),\sqrt{n}({\widehat{F}}_{\overline{D}}(t)-{F}_{\overline{D}}(t)))=\frac{n}{{n}_{\overline{D}}}\{-{A}^{-1}\left[\begin{array}{l}{A}_{0}(t)\hfill \\ {A}_{1}(t)\hfill \end{array}\right]+\left[\begin{array}{c}{F}_{\overline{D}}(t)\\ 0\end{array}\right]\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta *),\sqrt{n}(\widehat{F}*(t)-F(t)))=\frac{n}{{n}_{\overline{D}}}(1-\rho )\{-{A}^{-1}\left[\begin{array}{l}{A}_{0}(t)\hfill \\ {A}_{1}(t)\hfill \end{array}\right]+\left[\begin{array}{c}{F}_{\overline{D}}(t)\\ 0\end{array}\right]\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\frac{n}{{n}_{D}}\rho \{{A}^{-1}\left[\begin{array}{l}{A}_{0}(t)\hfill \\ {A}_{1}(t)\hfill \end{array}\right]-\left[\begin{array}{c}{F}_{D}(t)\\ 0\end{array}\right]\}.\end{array}$$

**Result 6**

$$\begin{array}{l}\mathit{\text{var}}(\sqrt{n}(\widehat{\rho}-\rho ))=\rho (1-\rho )/\lambda ,\\ \mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))=\left[\begin{array}{cc}\frac{1}{\lambda \rho (1-\rho )}& 0\\ 0& 0\end{array}\right]+\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}*-\theta )),\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{(}\widehat{\rho}-\rho ))=\left[\begin{array}{c}1/\lambda \\ 0\end{array}\right].\end{array}$$

**Result 7**

$$\begin{array}{l}\mathit{\text{var}}(\sqrt{n}(\widehat{F}(t)-F(t)))={({F}_{D}(t)-{F}_{\overline{D}}(t))}^{2}\rho (1-\rho )/\lambda +\mathit{\text{var}}(\sqrt{n}(\widehat{F}*(t)-F(t))),\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}(t)-F(t)))=\left[\begin{array}{c}\frac{{F}_{D}(t)-{F}_{\overline{D}}(t)}{\lambda}\\ 0\end{array}\right]+\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta ),\sqrt{n}(\widehat{F}*(t)-F(t))),\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}(t)-{F}_{D}(t)))=\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta *),\sqrt{n}({\widehat{F}}_{D}(t)-{F}_{D}(t))),\\ \mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{\overline{D}}(t)-{F}_{\overline{D}}(t)))=\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}*-\theta *),\sqrt{n}({\widehat{F}}_{\overline{D}}(t)-{F}_{\overline{D}}(t))).\end{array}$$

Most of the proofs in the following are exactly the same as those developed for a cohort study. Therefore we do not repeat the proofs that are the same. However, expressions for the components of the asymptotic variances that are different are provided. We will frequently refer to expressions in Results 4–7.

*Proof of Theorem item (i), (ii) and (iii)*

The proof is the same as the proof provided for cohort studies. Based on equation (16),

$$\begin{array}{l}{\sigma}_{1}{(p)}^{2}=\mathit{\text{var}}(\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p))))+{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial {R}^{-1}(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+2{(\frac{\partial {R}^{-1}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}(\widehat{F}({G}^{-1}(\theta ,p))-F({G}^{-1}(\theta ,p)))).\end{array}$$

Expressions for the three individual components can all be found in Results 6 and 7. Proofs for items (ii) and (iii) of the Theorem follow similar arguments.

*Proof of Theorem items (iv) and (v)*

According to equation (17),

$$\begin{array}{l}{\mathit{\text{cov}}}_{1}={(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial FPR(p)}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\theta ,p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\theta}-\theta ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{(\frac{\partial FPR(p)}{\partial \theta})}^{T}\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\theta}-\theta )).\end{array}$$

Results 6 and 7 provide expressions for the three individual terms.

However, the expressions for *cov*_{2} and *cov*_{3} are different from those under a cohort study design,

$$\begin{array}{l}{\mathit{\text{cov}}}_{2}\hspace{0.17em}=\mathit{\text{cov}}(\sqrt{n}(T\widehat{P}R(p)-\mathit{\text{TPR}}(p)),\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{D}({G}^{-1}(\theta ,p))-{F}_{D}({G}^{-1}(\theta ,p)))+\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}({F}_{D}({G}^{-1}(\widehat{\theta},p))-{F}_{D}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial \mathit{\text{TPR}}(p)}{\partial \theta})}^{T}\times \left[\begin{array}{c}1/\lambda \\ 0\end{array}\right]\end{array}$$

Similarly,

$$\begin{array}{l}{\mathit{\text{cov}}}_{3}=\mathit{\text{cov}}(\sqrt{n}(F\widehat{P}R(p)-FPR(p)),\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{cov}}(\sqrt{n}({\widehat{F}}_{\overline{D}}({G}^{-1}(\widehat{\theta},p))-{F}_{\overline{D}}({G}^{-1}(\theta ,p))),\sqrt{n}(\widehat{\rho}-\rho ))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial FPR(p)}{\partial \theta})}^{T}\times \left[\begin{array}{c}1/\lambda \\ 0\end{array}\right]\end{array}$$

Item (v) of the Theorem follows from a similar argument.

** Proof of Theorem (vi), (vii) and (viii)** These all follow similar arguments. We use (vii) to illustrate.

According to equation (21),

$$\begin{array}{l}{\sigma}_{7}{(t)}^{2}=\mathit{\text{var}}(\sqrt{n}(\widehat{R}({\nu}_{T}(t))-R({\nu}_{T}(t))))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{2}\mathit{\text{var}}(\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t))+{(\frac{\partial R({\nu}_{T}(t))}{\partial t})}^{T}\mathit{\text{var}}(\sqrt{n}(\widehat{\theta}-\theta ))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-2(\frac{\partial R({\nu}_{T}(t))}{\partial t})\mathit{\text{cov}}(\sqrt{n}(\widehat{\theta}-\theta ),\sqrt{n}({\widehat{F}}_{D}({F}_{D}^{-1}({\nu}_{T}(t)))-t))(\frac{\partial R({\nu}_{T}(t))}{\partial \theta}).\end{array}$$

The result follows by plugging in corresponding expressions from Result 2, 6 and 7. Proofs of items (vi) and (viii) follow similar arguments.

** Proof of Theorem item (ix)** Proof of Theorem (ix) is exactly the same as the proofs for a cohort study. Equations (22), (24) and (26) defined expressions for the components of the asymptotic variance of
$\sqrt{n}(P\widehat{E}V-PEV)$,
${\sigma}_{9}^{2}$. We only need to substitute

** Proof of Theorem item (x)** According to equations (27), (28) and (29), the three components of

$$\begin{array}{l}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathrm{\Sigma}}_{1}=\mathit{\text{var}}({A}_{n})+\mathit{\text{var}}({B}_{n})+\mathit{\text{var}}({C}_{n})+\mathit{\text{cov}}({A}_{n},{B}_{n})+\mathit{\text{cov}}({A}_{n},{C}_{n})+\mathit{\text{cov}}({B}_{n},{C}_{n});\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathrm{\Sigma}}_{2}=\mathit{\text{var}}({D}_{n})+\mathit{\text{var}}({E}_{n})+\mathit{\text{var}}({F}_{n})+\mathit{\text{cov}}({D}_{n},{E}_{n})+\mathit{\text{cov}}({D}_{n},{F}_{n})+\mathit{\text{cov}}({E}_{n},{F}_{n});\\ {\mathrm{\Sigma}}_{1,2}=\mathit{\text{cov}}({A}_{n},{E}_{n})+\mathit{\text{cov}}({A}_{n},{F}_{n})+\mathit{\text{cov}}({B}_{n},{D}_{n})+\mathit{\text{cov}}({B}_{n},{E}_{n})+\mathit{\text{cov}}({B}_{n},{F}_{n})\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\mathit{\text{cov}}({C}_{n},{D}_{n})+\mathit{\text{cov}}({C}_{n},{E}_{n})+\mathit{\text{cov}}({C}_{n},{F}_{n}).\end{array}$$

The following Results provide corresponding expressions for each of the individual components:

- Result 2:
*var*(*A*) and_{n}*var*(*D*)._{n} - Result 3:
*cov*(*B*_{n}*, E*)._{n} - Result 6:
*var*(*B*),_{n}*var*(*C*),_{n}*var*(*E*),_{n}*cov*(*B*_{n}*, C*),_{n}*var*(*F*),_{n}*cov*(*E*_{n}*, F*),_{n}*cov*(*B*_{n}*, F*),_{n}*cov*(*C*_{n}*, E*) and_{n}*cov*(*C*_{n}*, F*)._{n} - Result 7:
*cov*(*A*_{n}*, B*),_{n}*cov*(*D*_{n}*, E*),_{n}*cov*(*B*_{n}*,D*) and_{n}*cov*(*A*_{n}*, E*)._{n}

Furthermore, *cov*(*A _{n}*

^{}This work is supported in part by grants from the National Institutes of Health (R01 GM054438 and U01 CA086368).

- Baker SG, Kramer BS, Srivastava S. “Markers for Early Detection of Cancer: Statistical Guidelines for Nested Case-Control Studies,” BMC Medical Research Methodology. 2002;2:4. doi: 10.1186/1471-2288-2-4. [PMC free article] [PubMed] [Cross Ref]
- Bura E, Gastwirth JL. “The Binary Regression Quantile Plot: Assessing the Importance of Predictors in Binary Regression Visually,” Biometrical Journal. 2001;4:3, 5–21.
- Cook NR. “Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction,” Circulation. 2007;11:5, 928–935. [PubMed]
- Cook NR, Buring JE, Ridker PM. “The Effect of Including C-Reactive Protein in Cardiovascular Risk Prediction Models for Women,” Annals of Internal Medicine. 2006;14:5, 21–29. [PubMed]
- D’Agostino RB, Sr, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. “General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study,” Circulation. 2008;11:7, 743–753. [PubMed]
- Expert Panel on Detection EaToHBCiA “Executive Summary of the Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III),” The Journal of the American Medical Association. 2001;285(19):2486–2497. [PubMed]
- Gail MH, Pfeiffer RM. “On Criteria for Evaluating Models of Absolute Risk,” Biostatistics. 2005;6(2):227–239. doi: 10.1093/biostatistics/kxi005. [PubMed] [Cross Ref]
- Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1996.
- Gu W, Pepe MS. “Estimating the Capacity for Improvement in Risk Prediction with a Marker,” Biostatistics. 2009;10(1):172–186. doi: 10.1093/biostatistics/kxn025. [PMC free article] [PubMed] [Cross Ref]
- Gu W, Pepe MS. 2009a. “Measures to Summarize and Compare the Predictive Capacity of Markers,” UW Biostatistics Working Paper Series, Working Paper 342.
- Hamill PV, Drizd TA, Johnson CL, Reed RB, Roche AF. United States, Vital Health Statistics. Vol. 11. Washington, DC: 1977. “NCHS Growth Curves for Children Birth-8 Years,” pp. 1–74. [PubMed]
- Hosmer DW, Lemeshow S. 1989. Applied Logistic Regression (section 522). New York: Wiley
- Hu B, Palta M, Shao J. “Properties of
*R*^{2}Statistics for Logistic Regression,” Statistics in Medicine. 2006;25(8):1383–1395. doi: 10.1002/sim.2300. [PubMed] [Cross Ref] - Huang Y, Pepe MS. 2008a. “Semiparametric and Nonparametric Methods for Evaluating Risk Prediction Markers in Case-Control Studies,” UW Biostatistics Working Paper Series, Working Paper 333.
- Huang Y, Pepe MS. 2008b. “A Parametric ROC Model Based Approach for Evaluating the Predictiveness of Continuous Markers in Case-Control Studies,” Biometrics, doi: 10.1111/j.1541-0420.2005.00454.x [PMC free article] [PubMed]
- Huang Y, Pepe MS, Feng Z. “Evaluating the Predictiveness of a Continuous Marker,” Biometrics. 2007;63(4):1181–1188. [PMC free article] [PubMed]
- Hunink M, Glasziou P, Siegel J, Weeks J, Pliskin J, Elstein A, Weinstein M. Decision making in health and medicine. Cambridge University Press; 2006.
- Janes H, Pepe MS, Gu W. “Assessing the Value of Risk Predictions Using Risk Stratification Tables,” Annals of Internal Medicine. 2008;149(10):751–760. [PMC free article] [PubMed]
- Janssens ACJW, Aulchenko YS, Elefante S, Borsboom GJJM, Steyerberg EW, van Duijn CM. “Predictive Testing for Complex Diseases Using Multiple Genes: Fact or Fiction?,” Genetics in Medicine. 2006;8(7):395–400. doi: 10.1097/01.gim.0000229689.18263.f4. [PubMed] [Cross Ref]
- Knudson RJ, Lebowitz MD, Holberg CJ, Burrows B. “Changes in the Normal Maximal Expiratory Flow-Volume Curve with Growth and Aging,” The American Review of Respiratory Disease. 1983;12:7, 725–734. [PubMed]
- Pencina MJ, D’Agostino RB, Sr, D’Agostino RB, Jr, Vasan RS. “Evaluating the Added Predictive Ability of a New Marker: From Area Under the ROC Curve to Reclassification and Beyond,” Statistics in Medicine. 2008;2:7, 157–172. [PubMed]
- Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press; 2003.
- Pepe MS, Etzioni R, Feng Z, Potter JD, Thompson ML, Thornquist M, Winget M, Yasui Y. 2001. “Phases of Biomarker Development for Early Detection of Cancer,” Journal of the National Cancer Institute 93, 1054?061.10.1093/jnci/93.14.1054 [PubMed] [Cross Ref]
- Pepe MS, Feng Z, Gu W. “Comments on ’Evaluating the Added Predictive Ability of a New Marker: From Area Under the ROC Curve to Reclassification and Beyond’,” Statistics in Medicine. 2008b;2:7, 173–181. [PubMed]
- Pepe MS, Feng Z, Huang Y, Longton GM, Prentice R, Thompson IM, Zheng Y. “Integrating the Predictiveness of a Marker with its Performance as a Classifier,” American Journal of Epidemiology. 2008a;167(3):362–368. doi: 10.1093/aje/kwm305. [PMC free article] [PubMed] [Cross Ref]
- Pepe MS, Janes H. “Gauging the Performance of SNPs, Biomarkers and Clinical Factors for Predicting Risk of Breast Cancer,” Journal of the National Cancer Institute. 2008c;100(14):978–979. doi: 10.1093/jnci/djn215. [PMC free article] [PubMed] [Cross Ref]
- Pepe MS, Feng Z, Janes H, Bossuyt P, Potter J. “Pivotal Evaluation of the Accuracy of a Biomarker Used for Classification or Prediction: Standards for Study Design,” Journal of the National Cancer Institute. 2008d;100(20):1432–1438. doi: 10.1093/jnci/djn326. [PubMed] [Cross Ref]
- Pepe MS, Janes H, Gu W. “Letter by Pepe et al Regarding the Article, ’Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction’,” Circulation. 2007;116:e132. doi: 10.1161/CIRCULATIONAHA.107.709253. [PubMed] [Cross Ref]
- Prentice RL, Pyke R. “Logistic Disease Incidence Models and Case-Control Studies,” Biometrika. 1979;66(3):403–411. doi: 10.1093/biomet/66.3.403. [Cross Ref]
- Qin J. “Inferences for Case-Control and Semiparametric Two-Sample Density Ratio Models,” Biometrika. 1998;85(3):619–630. doi: 10.1093/biomet/85.3.619. [Cross Ref]
- Qin J, Lawless J. “Empirical Likelihood and General Estimating Equations,” The Annals of Statistics. 1994;66(3):403–411.
- Qin J, Zhang B. “A goodness of fit test for logistic regression models based on case-conrol data,” Biometrika. 1997;84(3):609–618. doi: 10.1093/biomet/84.3.609. [Cross Ref]
- Qin J, Zhang B. “Using logistic regression procedures for estimating receiver operating characteristic curves,” Biometrika. 2003;90(3):585–596. doi: 10.1093/biomet/90.3.585. [Cross Ref]
- Ransohoff DF. “Rules of Evidence for Cancer Molecular-Marker Discovery and Validation,” Nature Reviews Cancer. 2004;4(4):309–314. doi: 10.1038/nrc1322. [PubMed] [Cross Ref]
- Ridker PM, Hennekens CH, Buring JE, Rifai N. “C-Reactive Protein and Other Markers of Inflammation in the Prediction of Cardiovascular Disease in Women,” New England Journal of Medicine. 2000;34:2, 836–843. [PubMed]
- Ridker PM, Paynter NP, Rifai N, Gaziano JM, Cook NR. “C-Reactive Protein and Parental History Improve Global Cardiovascular Risk Prediction: The Risk Score for Men,” Circulation. 2008;11:8, 2243–2251. [PMC free article] [PubMed]
- Simon R. “Roadmap for Developing and Validating Therapeutically Relevant Genomic Classifiers,” Journal of Clinical Oncology. 2006;23(29):7332–7341. doi: 10.1200/JCO.2005.02.8712. [PubMed] [Cross Ref]
- Stern RH. “Evaluating New Cardiovascular Risk Factors for Risk Stratification,” The Journal of Clinical Hypertension. 2008;10(6):485–488. doi: 10.1111/j.1751-7176.2008.07814.x. [PubMed] [Cross Ref]
- Vickers AJ, Elkin EB. “Decision Curve Analysis: a Novel Method for Evaluating Prediction Models,” Medical Decision Making. 2006;26(6):565–574. doi: 10.1177/0272989X06295361. [PMC free article] [PubMed] [Cross Ref]
- Ware JH, Cai T. “Comments on ‘Evaluating the Added Predictive Ability of a New Marker: From Area Under the ROC Curve to Reclassification and Beyond’,” Statistics in Medicine. 2008;2:7, 185–187. [PubMed]
- Youden WJ. 1950. “Index for Rating Diagnostic Tests,” Cancer 3, 32?5.10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3 [PubMed] [Cross Ref]
- Zheng B, Agresti A. “Summarizing the Predictive Power of a Generalized Linear Model,” Statistics in Medicine. 2000;19(13):1771–1781. doi: 10.1002/1097-0258(20000715)19:13<1771::AID-SIM485>3.0.CO;2-P. [PubMed] [Cross Ref]

Articles from The International Journal of Biostatistics are provided here courtesy of **Berkeley Electronic Press**

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |