Home | About | Journals | Submit | Contact Us | Français |

**|**HHS Author Manuscripts**|**PMC2766283

Formats

Article sections

- SUMMARY
- 1. Introduction
- 2. Influenza vaccine study
- 3. Notation and data structure
- 4. Vaccine efficacy
- 5. Two structural assumptions
- 6. Identification of Px [Y(z) = 1]
- 7. Frequentist sensitivity analysis
- 8. Bayesian Inference
- 9. Data Analysis
- 10. Discussion
- Supplementary Material
- References

Authors

Related links

Biostatistics. Author manuscript; available in PMC 2009 October 23.

Published in final edited form as:

Published online 2006 March 23. doi: 10.1093/biostatistics/kxj031

PMCID: PMC2766283

NIHMSID: NIHMS143242

DANIEL O. SCHARFSTEIN, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21205, USA;

The publisher's final edited version of this article is available free at Biostatistics

See other articles in PMC that cite the published article.

Using validation sets for outcomes can greatly improve the estimation of vaccine efficacy (VE) in the field (Halloran and Longini, 2001; Halloran *and others*, 2003). Most statistical methods for using validation sets rely on the assumption that outcomes on those with no cultures are missing at random (MAR). However, often the validation sets will not be chosen at random. For example, confirmational cultures are often done on people with influenza-like illness as part of routine influenza surveillance. VE estimates based on such non-MAR validation sets could be biased. Here we propose frequentist and Bayesian approaches for estimating VE in the presence of validation bias. Our work builds on the ideas of Rotnitzky *and others* (1998, 2001), Scharfstein *and others* (1999, 2003), and Robins *and others* (2000). Our methods require expert opinion about the nature of the validation selection bias. In a re-analysis of an influenza vaccine study, we found, using the beliefs of a flu expert, that within any plausible range of selection bias the VE estimate based on the validation sets is much higher than the point estimate using just the non-specific case definition. Our approach is generally applicable to studies with missing binary outcomes with categorical covariates.

Many statistical methods have been developed to deal with estimating the causal effects of randomized treatments when outcomes are missing on some participants. Most methods rely on the non-identifiable assumption that the outcome of interest is missing at random (MAR) (Little and Rubin, 2002). If the outcome is not MAR, then effect estimates could be subject to selection bias. Rotnitzky *and others* (1998, 2001), Scharfstein *and others* (1999), and Robins *and others* (2000) developed a frequentist selection model that displays the sensitivity analysis over a plausible range of selection bias parameters. Scharfstein *and others* (2003) developed a Bayesian approach that allows the formal incorporation of prior beliefs about the degree of the selection bias to obtain the full posterior distribution, a single summary of the sensitivity analysis. Others, for example Little (1994) and Daniels and Hogan (2000), have developed pattern-mixture models for sensitivity analyses. Some other approaches include the work of Baker *and others* (2003), Molenberghs *and others* (2001), Verbeke *and others* (2001), and Vansteelandt *and others* (2006).

A particular case of missing data occurs if the outcome of interest is difficult or expensive to ascertain, so that a surrogate outcome may be used instead. The outcome of interest may be measured on some of the study participants in a subset called a validation sample, while the surrogate is measured on all participants. In this situation, statistical missing data methods are available to use the outcomes of interest in the validation sample to correct the bias based on the non-specific case definition alone (Pepe *and others*, 1994). Halloran and Longini (2001), Halloran *and others* (2003), and Chu and Halloran (2004) have demonstrated the potential use of these methods for estimating vaccine efficacy (VE) on the example of an influenza vaccine. In a randomized study with a planned random sample selected for the validation set, MAR would be a reasonable assumption. However, in many situations, the selected sample may be a convenience sample, so that MAR is unlikely to hold. Halloran *and others* (2003) presented a simple model to explore the sensitivity of the VE estimates to the magnitude of the departure from the MAR assumption. However, their approach was ad hoc and did not give confidence bounds on their estimators.

Here, we formulate a class of selection models, indexed by interpretable parameters, to evaluate the sensitivity to selection bias when using validation sets to estimate VE. Frequentist and Bayesian approaches to inference will be presented. In developing and applying our methodology to the re-analysis of the influenza vaccine study, we worked closely with a scientific expert. Our approach is generally applicable to missing binary outcomes with categorical covariates.

A field study of a trivalent, cold-adapted, influenza virus vaccine (CAIV-T) was conducted in Temple-Belton, Texas, and surrounding areas during the 2000–2001 influenza season. The field study was part of a larger community-based, non-randomized, open-label field study conducted from 1998–2001 (Piedra *and others*, 2001; Gaglani *and others*, 2003). In Temple-Belton, eligible healthy children and adolescents aged 18 months through 18 years were offered CAIV-T vaccine through the Scott & White (S & W) Clinics from 1998–2001. The analysis includes children who were S & W Health Plan members, and is concerned with the CAIV-T vaccinations administered in the influenza season 2000–2001. Children received a single dose of CAIV-T each year that they enrolled. The primary clinical outcome was a non-specific case definition called medically attended acute respiratory infection (MAARI), which included all ICD-9-CM diagnoses codes (Codes 381–383, 460–487) for upper and lower respiratory tract infections, otitis media and sinusitis. Any individual presenting with history of fever and any respiratory illness at S & W Clinics was eligible to have a throat swab (or nasal wash in young infants) for influenza virus culture. The decision to obtain specimens was made irrespective of whether a patient had received CAIV-T. The specific case definition is culture-confirmed influenza. Table 1 contains the data. The overall fraction of MAARI cases sampled was a little higher in the unvaccinated than in the vaccinated groups (*p* = 0.03).

Halloran *and others* (2003) analyzed the data by adapting the mean score method for validation sets (Pepe *and others*, 1994) with the goal of evaluating the protective VE of CAIV-T vaccination in healthy children during the influenza season 2000–01. Chu and Halloran (2004) developed a Bayesian method. The overall vaccine effectiveness estimate based on the non-specific case definition was 0.18 (95% CI: 0.11, 0.24). The overall efficacy estimates incorporating the surveillance cultures using the mean score method was 0.79 (95% CI: 0.51, 0.91) and the Bayesian method was 0.74 (95% HPD: 0.50, 0.88). In this situation, using the surveillance cultures as a validation set resulted in a four-fold increase in estimates, much closer to the efficacy estimate of 0.93 (95% CI: 0.88, 0.97) obtained in a double-blind, randomized controlled trial (Belshe *and others*, 1998).

In the influenza vaccine study in Texas, no surveillance cultures were positive in the age group 1.5–4 years. In Halloran *and others* (2003), a continuity correction of 0.5 was added to the number of cultured samples and to the number positive in that age group in the mean score analysis. For this age group, their estimate of VE using the mean score method was 0.91 (95% CI: −0.24,0.99). The Bayesian method of Chu and Halloran (2004) yielded an estimate of 1.00 (95% HPD: 0.52,1.00). So, the Bayesian method provided a much tighter measure of uncertainty than the mean score method with the continuity correction.

The results of Halloran *and others* (2003) and Chu and Halloran (2004) are valid only if the culture-confirmed influenza status is MAR. In consulting with influenza experts, we learned that this assumption can easily be violated in this study if physicians tend to select children whom they believe to have influenza for culturing. Our goal is to develop frequentist and Bayesian methods for sensitivity analyses for these and similar data. Further, we develop a fully Bayesian procedure that formally incorporates expert beliefs about the culturing mechanism.

In the vaccine field study, let *n* be the total number of participants, and *n*_{0} and *n*_{1} the number of non-vaccinated and vaccinated participants, respectively. Let *Z* denote the vaccination indicator, taking on the value 1 if a participant is vaccinated and 0 if not vaccinated. Let *A*(0) and *A*(1) denote the indicator of MAARI (1: yes, 0: no) for a participant if she had been, possibly contrary to fact, unvaccinated or vaccinated, respectively. The observed MAARI outcome *A* = *A*(*Z*) is observed for every participant. Let *Y*(0) and *Y*(1) denote influenza status (1: positive, 0: negative) for a participant if she had been, possibly contrary to fact, unvaccinated and vaccinated, respectively. Only one of these outcomes can be potentially observed. In this study, influenza status is biologically confirmed by a culture. In the validation substudy, a possibly non-random sample of the participants are biologically confirmed, so that influenza status, *Y* = *Y*(*Z*), is known for a subset of the participants. Let *R* be the validation indicator, where *R* = 1 if sampled for validation and *R* = 0, otherwise. Sampling for validation only occurs for those with *A* = 1. Let *X* denote age category (0: 1.5–4 years, 1: 5–9 years, 2: 10–18 years) measured at the time of study entry.

With this notation, the observed data for an individual are *O* = (*Z, X, A, R, Y*: *A* = *R* = 1). We assume that we observe *n* i.i.d. copies, **O** = {*O _{i}* :

Throughout, probabilities *P*, indexed by subgroup subscripts indicate restriction to the associated subpopulation. For example, for events *A* and *B*, *P _{z,x}* [

The scientific goal is to use the observed data to estimate the causal effect of vaccination on the outcome *Y* , within age levels as well as overall. Specifically, we want to estimate age-specific VE

$${\mathrm{VE}}_{S,x}=1-\frac{{P}_{x}[Y\left(1\right)=1]}{{P}_{x}[Y\left(0\right)=1]}$$

and overall VE

$${\mathrm{VE}}_{S}=1-\frac{{\Sigma}_{x=0}^{2}{P}_{x}[Y\left(1\right)=1]P[X=x]}{{\Sigma}_{x=0}^{2}{P}_{x}[Y\left(0\right)=1]P[X=x]}.$$

To identify VE_{S,x}, it is sufficient to identify *P _{x}* [

Before proceeding further, we will make two structural assumptions to facilitate identification of *P _{x}* [

*Z* is independent of {*A*(0), *A*(1), *Y*(0), *Y*(1)} given *X*.

This assumption states that vaccination status is independent of the potential outcomes {*A*(0), *A*(1), *Y*(0), *Y*(1)}, given age (*X*). That is, within levels of age, vaccination is randomized. Our expert felt that this assumption was reasonable, since there is no reason to believe that the decision to vaccinate was based on anything related to the potential outcomes.

*A*(*z*) = 0 implies *Y*(*z*) = 0.

We make this assumption because the study design called for passive case ascertainment. As a result, the data structure is such that no participants who do not appear in the clinic are cultured for confirmation of influenza infection. The above assumption states that if a participant, under vaccination status *z*, does not have MAARI, then she does not have medically attended influenza. The interest is in efficacy against medically attended, culture-confirmed influenza, not influenza infection.

With these assumptions, we can write

$${P}_{x}[Y\left(z\right)=1]={P}_{z,x}[Y\left(z\right)=1]$$

(6.1)

$$={P}_{z,x}[Y=1]$$

(6.2)

$$=\underset{r=0}{\overset{1}{\Sigma}}{P}_{z,x}[Y=1A=1,R=r]{P}_{z,x}[A=1,R=r].$$

(6.3)

Equation (6.1) follows from randomization within levels of *X*, (6.2) uses the fact that *Y* = *Y*(*Z*), and (6.3) follows from an application of the law of conditional probability and our second assumption above. Note that, for all *z, x, r, P _{z,x}* [

The most common assumption employed to identify these probabilities is that of MAR (Little and Rubin, 2002). MAR states that *R* is independent of *Y* given (*Z, A, X*). This implies that, for all *z, x*, *P _{z,x}* [

Scharfstein *and others* (1999, 2003) and Robins *and others* (2000) introduced a sensitivity analysis methodology in which a class of models (including MAR) are posited, each yielding identification of *P _{z,x}* [

$${P}_{z,x}[Y=1A=1,R=0]=\frac{{P}_{z,x}[Y=1A=1,R=1]\phantom{\rule{thinmathspace}{0ex}}\mathrm{exp}\left({\alpha}_{z,x}\right){c}_{z,x}}{,}$$

(7.1)

where

$${c}_{z,x}={P}_{z,x}[Y=0A=1,R=1]+{P}_{z,x}[Y=1A=1,R=1]\phantom{\rule{thinmathspace}{0ex}}\mathrm{exp}\left({\alpha}_{z,x}\right)$$

and *α _{z,x}* is a specified non-identifiable constant to be varied in the sensitivity analysis. Setting

$${P}_{x}[Y\left(z\right)=1]={P}_{z,x}[Y=1A=1,R=1]\left\{{P}_{z,x}[A=1,R=1]+\frac{\mathrm{exp}\left({\alpha}_{z,x}\right)}{{c}_{z,x}}{P}_{z,x}[A=1,R=0]\right\}.$$

(7.2)

It is possible to re-write Model (7.1) as the following selection model:

$$\mathrm{logit}{P}_{z,x}[R=0A=1,Y=y]={h}_{z,x}+{\alpha}_{z,x}y,$$

(7.3)

where

$${h}_{z,x}=\mathrm{log}\{\frac{1}{{c}_{z,x}}\frac{{P}_{z,x}[R=0A=1]{P}_{z,x}[R=1A=1]}{\}}.$$

(7.4)

For subjects with *Z* = *z*, *X* = *x*, and MAARI, *α _{z,x}* is interpreted as the log odds ratio of being unvalidated for diseased vs. undiseased subjects. So,

When eliciting plausible ranges for *α _{z,x}*, our expert found it easier to think about selection bias on a relative risk as opposed to an odds ratio scale. Specifically, he felt more comfortable expressing opinions about the relative risk of being validated given that a MAARI participant has influenza, compared with having another influenza-like illness. As a result, we re-formulated the above models in terms of the relative risk selection bias parameters

$${\beta}_{z,x}=\frac{{P}_{z,x}[R=1A=1,Y=1]{P}_{z,x}[R=1A=1,Y=0].}{}$$

(7.5)

We can re-parametrize Model (7.3) in terms of *β _{z,x}*. Letting

$$\begin{array}{cc}\hfill {\eta}_{z,x}& ={(1+\mathrm{exp}\left({h}_{z,x}\right))}^{-1}\hfill \\ \hfill {\beta}_{z,x}& =\frac{1+\mathrm{exp}\left({h}_{z,x}\right)}{1+\mathrm{exp}({h}_{z,x}+{\alpha}_{z,x})}\hfill \end{array}$$

or

$$\begin{array}{cc}\hfill {h}_{z,x}& =\mathrm{log}(1-{\eta}_{z,x})-\mathrm{log}\left({\eta}_{z,x}\right)\hfill \\ \hfill {\alpha}_{z,x}& =\mathrm{log}({\beta}_{z,x}^{-1}-{\eta}_{z,x})-\mathrm{log}(1-{\eta}_{z,x}).\hfill \end{array}$$

Because of this one-to-one relationship, specification of *β _{z,x}* will lead to identification of

$$\begin{array}{cc}\hfill {P}_{x}[Y\left(z\right)=1]& =\frac{{P}_{z,x}[Y=1A=1,R=1]{P}_{z,x}[A=1]{\beta}_{z,x}{P}_{z,x}[Y=0A=1,R=1]+{P}_{z,x}[Y=1A=1,R=1]}{}\hfill & =\frac{P[Z=z,X=x,A=1,R=1,Y=1]P[Z=z,X=x,A=1]/P[Z=z,X=x]}{{\beta}_{z,x}P[Z=z,X=x,A=1,R=1,Y=0]+P[Z=z,X=x,A=1,R=1,Y1]}.\hfill \hfill \end{array}$$

(7.6)

The frequentist non-parametric estimator of *P _{x}* [

$${\widehat{P}}_{x}[Y\left(z\right)=1]=\frac{\stackrel{~}{P}[Z=z,X=x,A=1,R=1,Y=1]\stackrel{~}{P}[Z=z,X=x,A=1]/\stackrel{~}{P}[Z=z,X=x]}{{\beta}_{z,x}\stackrel{~}{P}[Z=z,X=x,A=1,R=1,Y=0]+\stackrel{~}{P}[Z=z,X=x,A=1,R=1,Y=1]}.$$

(7.7)

The right-hand side of the above equation reduces to the results with the mean score method when *β _{z,x}* = 1, for all

We can then estimate VE_{S,x} by

$${\widehat{\mathrm{VE}}}_{S,x}=1-\frac{{\widehat{P}}_{x}[Y\left(1\right)=1]}{{\widehat{P}}_{x}[Y\left(0\right)=1]},$$

and VE_{S} by

$${\widehat{\mathrm{VE}}}_{S}=1-\frac{{\Sigma}_{x=0}^{2}{\widehat{P}}_{x}[Y\left(1\right)=1]\stackrel{~}{P}[X=x]}{{\Sigma}_{x=0}^{2}{\widehat{P}}_{x}[Y\left(0\right)=1]\stackrel{~}{P}[X=x]}.$$

In Section A of supplementary material available at *Biostatistics* online (http://www.biostatistics.oxfordjournals.org), we derive the large sample-based confidence intervals for VE_{S,x} and VE_{S}. The sensitivity analysis proceeds by varying the *β _{z,x}* over plausible ranges. When a stratum has a relatively small number of validation cultures and small number of positive cultures, as in our influenza example, these large sample confidence intervals may not perform well. When there are no positive cultures, one can use the continuity correction approach of Halloran

Scharfstein *and others* (2003) developed a Bayesian methodology that allows full posterior inference about the estimands of interest, by assuming informative prior distributions on the selection bias parameters on the odds ratio scale. Their work did not allow for covariates and assumed independent priors across treatment groups. Our goal is to extend their work to the relative risk parametrization of selection bias, discrete covariates, and dependence of the priors for the relative risk parameters across treatment groups.

To simplify notation, we let ${\beta}_{z}={({\beta}_{z,0},{\beta}_{z,1},{\beta}_{z,2})}^{\prime},\phantom{\rule{thinmathspace}{0ex}}\beta =({\beta}_{0}^{\prime},{\beta}_{1}^{\prime}),\phantom{\rule{thinmathspace}{0ex}}{\eta}_{z}={({\eta}_{z,0},{\eta}_{z,1},{\eta}_{z,2})}^{\prime},\phantom{\rule{thinmathspace}{0ex}}\eta ={({\eta}_{0}^{\prime},{\eta}_{1}^{\prime})}^{\prime},\phantom{\rule{thinmathspace}{0ex}}{p}_{z,x}={P}_{z,x}[Y=1A=1],{\mathit{p}}_{z}={({p}_{z,0},{p}_{z,1},{p}_{z,2})}^{\prime},\mathit{p}=({\mathit{p}}_{0}^{\prime},{\mathit{p}}_{1}^{\prime}),{\varphi}_{z,x,a}={P}_{z}[A=a,X=x],{\varphi}_{z}={({\varphi}_{z,0,0},{\varphi}_{z,0,1},{\varphi}_{z,1,0},{\varphi}_{z,1,1},{\varphi}_{z,2,0},{\varphi}_{z,2,1})}^{\prime}$, and $\varphi ={({\varphi}_{0}^{\prime},{\varphi}_{1}^{\prime})}^{\prime}$. With this notation,

$${P}_{x}[Y\left(z\right)=1]={P}_{z,x}[Y=1]={p}_{z,x}\frac{{\varphi}_{z,x,1}}{{\Sigma}_{a=0}^{1}{\varphi}_{z,x,a}},$$

and

$${\mathrm{VE}}_{S,x}=1-\frac{{p}_{1,x}}{{p}_{0,x}}\frac{{\varphi}_{1,x,1}}{{\varphi}_{0,x,1}}\frac{{\Sigma}_{a=0}^{1}{\varphi}_{0,x,a}}{{\Sigma}_{a=0}^{1}{\varphi}_{1,x,a}},$$

(8.1)

$${\mathrm{VE}}_{S}=1-\frac{{\Sigma}_{x=0}^{2}\left\{{p}_{1,x}{\varphi}_{1,x,1}\left({\Sigma}_{z=0}^{1}{\Sigma}_{a=0}^{1}{\varphi}_{z,x,a}\right)/\left({\Sigma}_{a=0}^{1}{\varphi}_{1,x,a}\right)\right\}}{{\Sigma}_{x=0}^{2}\left\{{p}_{0,x}{\varphi}_{0,x,1}\left({\Sigma}_{z=0}^{1}{\Sigma}_{a=0}^{1}{\varphi}_{z,x,a}\right)/\left({\Sigma}_{a=0}^{1}{\varphi}_{0,x,a}\right)\right\}}.$$

(8.2)

In the prior specification for ** β**, we provide two options: (1) Bayesian analogue of the frequentist sensitivity analysis and (2) fully Bayesian analysis. For option (1), we specify point-mass priors on

In our specification of the joint prior distribution on (** β**′,

**is independent of (**′,*β*′,*η*′)′.*p*_{1}is independent of_{0}.is independent of (*p*′,*β*′)′.*η*- The components of
are independent.*p* - The components of
may be correlated.*β* - Given
*β*,_{z,x}*η*is independent of {_{z,x}*β*_{1−z,x}′,*η*_{1−z,x}′ :*x*′ ≠*x*}. - •
*β*and_{z,x}*η*are only related through_{z,x}*β*'s restriction on the support of_{z,x}*η*._{z,x}

Thus, the prior distribution of (** β**′,

$$\pi (\beta ,\eta ,\mathit{p},\varphi )=\pi \left(\beta \right)\underset{z=0}{\overset{1}{\Pi}}\underset{x=0}{\overset{2}{\Pi}}\pi \left({p}_{z,x}\right)\pi ({\eta}_{z,x}{\beta}_{z,x})\pi \left({\varphi}_{z}\right).$$

We further assume that

*π*() is an informative prior on a compact subset of*β*^{6}. For the fully Bayesian analysis of the vaccine trial, we assume an informative multivariate normal prior on the logscale. The details of the informative priors are described in Section 9.1.*β**π*(*p*) is a uniform prior on [0, 1]._{z,x}*π*(*η*|_{z,x}*β*) is a uniform prior on $[0,\phantom{\rule{thinmathspace}{0ex}}\mathrm{min}({\beta}_{z,x}^{-1},\phantom{\rule{thinmathspace}{0ex}}1)]$. Specifically,_{z,x}where$$\pi ({\eta}_{z,x}{\beta}_{z,x})=\mathrm{max}\{{\beta}_{z,x},1\}I({\eta}_{z,x}\mathrm{min}\{1,1/{\beta}_{z,x}\}),$$*I*(*A*) denotes the indicator of the event*A*.*π*() is a non-informative prior on the set_{z}Assume that the combinations of ($$\left\{({\varphi}_{z,0,0},{\varphi}_{z,0,1},{\varphi}_{z,1,0},{\varphi}_{z,1,1},{\varphi}_{z,2,0},{\varphi}_{z,2,1}):0\le {\varphi}_{z,x,a}\le 1,\underset{x=0}{\overset{2}{\Sigma}}\underset{a=0}{\overset{1}{\Sigma}}{\varphi}_{z,x,a}=1\right\}.$$*x*,*a*) have*k*= 1,…,*K*categories with*n*subjects in each category. The prior on_{zk}is specified as a Dirichlet,_{z}*D*(1,…, 1). This represents a prior sample of*K*, the number of categories of (*x*,*a*). We could make this less informative by replacing the 1's with 1/*K*.

To sample from the posterior, we constructed a Gibbs sampling algorithm with data augmentation (Tanner and Wong, 1987) and slice sampling (Damien *and others*, 1999; Neal, 2003). Section B of the supplementary material available at *Biostatistics* online (http://www.biostatistics.oxfordjournals.org) provides a full description of the algorithm.

For Bayesian inference, we specify informative priors for the selection bias relative risk parameters, ** β**, by age group and vaccination status. We asked an influenza expert the following question: “If a physician were doing surveillance cultures during an influenza season, what is the probability that he would select the children who actually had true influenza over the children who just had non-specific respiratory symptoms to culture?” He responded that this was very hard to answer. One “would be more likely to be correct in the unvaccinated,” because unvaccinated children presenting with true influenza would have more typical, severe disease than the vaccinated children. One would be “less likely to be correct in young children under 5 years,” because children under 5 years experience many other severe respiratory diseases that could be mistaken for influenza, while older children are already immune to such diseases. He added that the degree of selection bias would also “depend on the rules for collection, for example, a certain number per week or with specific symptoms.”

Another influenza expert had similar views. He provided us with his best guess for each of the univariate relative risk selection bias parameters *β _{z,x}* defined in (7.5) and his belief about the interval that would likely contain 90% of the prior distribution for each

Best guess and 90% range for the informative prior distributions on the selection bias parameter **β** and log **β**

For our analysis, we plugged the elicitations into a multivariate Normal prior on the log ** β** scale as follows. We transformed the elicited best guesses for each

The expert believed that the correlation in selection bias among the strata would be high. He suggested a correlation as high as 0.90. The corresponding covariance matrices for *π*(** β**) were constructed from the marginal univariate Normal distributions and the correlations. We also performed the analysis assuming zero correlation.

All programs were written in R. The Markov Chain Monte Carlo algorithm was run for 500 000 iterations with 100 000 discarded as burn-in.

Figures Figures11 and and22 display results of the frequentist sensitivity analysis. These figures present results by age stratum. The selection bias parameters were varied over the 90% ranges elicited from the expert. It was not feasible to present parsimoniously the overall results because the selection bias parameters were too high-dimensional. This is a key drawback of the frequentist sensitivity analysis methodology.

Frequentist sensitivity analysis of *P*_{z,x} [*Y* = 1]. Shown are the estimated probabilities (and 95% confidence intervals) for influenza in the vaccinated and unvaccinated groups, for each age stratum, as a function of the relative risk selection bias parameter **...**

Frequentist sensitivity analysis of point estimates and lower 95% confidence bounds for the age-group-specific VE as a function of the relative risk selection bias parameters *β*_{1,x} (vaccinated) and *β*_{0,x} (unvaccinated) varied over the 90% **...**

Figure 1 shows the estimated probabilities (and 95% confidence intervals) for influenza in the vaccinated and unvaccinated groups, for each age stratum, as a function of *β _{z,x}*. Figure 2 shows the point estimates and lower 95% confidence bounds for the age-group-specific efficacy. The black diamonds indicate the results at the best guess of the expert. Within these ranges and within each age group, the VE estimates based on the validation sets are higher than the point estimates based on the non-specific definition, which were 0.2, 0.25, and 0.14 for the age groups 1.5–4, 5–9, and 10–18, respectively. The lower confidence bounds indicate the degree of variability.

The prior and posterior distributions for the relative risks of selection *β _{z,x}* 's by vaccination status and age group are the same, since there is no information in the data about the degree of selection bias (figure not shown). Figure 3 shows the Bayesian posterior distribution of the age-group-specific efficacy and overall efficacy using the informative prior distributions from Table 2, assuming a correlation of 0.9. The mode is 1.00 in the age group 1.5–4 years, since there are no positive cultures in the vaccinated group in that age group. The results assuming a zero correlation were nearly identical (not shown).

Posterior distributions of the overall VE and by age group using the informative prior distributions.

Table 3 compares the summaries of the Bayesian posterior distributions and of the frequentist estimates and 95% confidence intervals. The assumption of MAR results in an overestimate of the VE compared with the selection bias relative risk assumptions elicited from the expert. Consistent with Chu and Halloran (2004), the Bayesian posterior means are somewhat lower than the frequentist estimates. In general, the Bayesian credible intervals are tighter than the frequentist confidence intervals. This is especially true for the age group 1.5–4 years, where the validation sample is small and there are no positive cultures.

Results of Bayesian and frequentist analyses using surveilance cultures. For the Bayesian analyses, the posterior means (95% highest posterior density credible intervals) for vaccine efficacy are reported, for the frequentist analyses, the point estimates **...**

It is interesting to note that the Bayesian analysis using an informative prior for ** β** had a narrower credible interval than the analysis with

In this paper, we have developed both frequentist and Bayesian methods for analyzing missing binary outcomes that are thought to be informatively (i.e. related to the unobserved outcome) missing. Our relative risk parametrization of selection bias, incorporation for discrete covariates, and prior dependence of the selection bias across treatment groups extends earlier work (Scharfstein *and others*, 2003), which used the log odds ratio parametrization without covariates. We elicited informative age group and vaccination status prior distributions from an influenza expert. Bayesian inference with informative priors on the selection bias parameters provides a useful and parsimonious way of drawing inference about VE that incorporates expert uncertainty about the missing data mechanism. In addition, the Bayesian approach provides better small-sample inference.

The frequentist or Bayesian sensitivity analysis approach provides much greater detail than the single summary from the fully Bayesian analysis. As a consequence, when the dimension of the selection parameters is greater than 3 or 4, it is harder to visualize. While the advantage of the Bayesian sensitivity analysis is the finite sample performance, it is computationally very intensive. The frequentist sensitivity analysis is computationally more feasible and will perform well when the sample sizes are large.

We used our proposed methods to re-analyze an influenza vaccine field study. We did a formal sensitivity analysis to evaluate the effects of preferential selection of children with non-specific illness for obtaining surveillance cultures to confirm true influenza. Our analysis showed that under plausible ranges of selection bias, the VE estimates, though lower than when assuming MAR, are substantially higher than those based on the non-specific influenza-like illness definition alone. Our methods will be generally useful in future vaccine field studies, or other similar studies, in which confirmatory biological specimens are not MAR.

For our development in this paper, we made the assumption that any person who does not present with non-specific influenza-like illness also does not have medically attended influenza illness. Implicitly, we assumed a degenerate prior at zero for the *P _{z,x}* [

It is straightforward to extend our method to the situation that infection is confirmed, perhaps serologically, in a sample of people who did not have non-specific illness, *A*(*z*) = 0. In this case, we would not need Assumption 2. The scientific question then would be to estimate efficacy against infection, not medically attended disease, as in this paper. There would be additional selection bias parameters in that situation. However, if a study were well enough planned to sample asymptomatic participants, it would be hoped that the sampling would be planned to be random, so that selection bias would not play a role.

Our informative priors for the selection bias parameters were assumed to be multivariate Normal on the log ** β** scale. Prior distributions could also be constructed less parametrically using the information elicited from the experts. When using our methods to analyze future studies, the providers making the decision from whom to obtain biological specimens could be directly asked their prior beliefs about selection bias.

In ongoing research, we are extending the Bayesian methods to incorporate higher-dimensional covariates. We are also working on methods for longitudinal and time-to-event outcomes.

Daniel Scharfstein was partially supported by National Institutes of Health (NIH) grants R01-CA85295, P30-MH066247, and R01-A132475. Elizabeth Halloran was partially supported by NIH grant R01-AI32042. Michael Daniels was partially supported by NIH R01-CA85295 and R01-HL079457. Haitao Chu was supported in part by the NIH through the data coordinating centers for the Multicenter AIDS Cohort Study (UO1-AI-35043), the Women's Interagency HIV Study (UO1-AI-42590), and the Pediatric Chronic Kidney Disease cohort study (UO1-DK-066116). We thank Ira M. Longini, Jr., Pedro A. Piedra, W. Paul Glezen, and Arnold Monto for helpful comments during elicitation of the informative priors.

*Conflict of Interest:* None declared.

DANIEL O. SCHARFSTEIN, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21205, USA.

M. ELIZABETH HALLORAN, Program in Biostatistics and Biomathematics, Fred Hutchinson Cancer Research Center, Seattle, WA 98109, USA and Department of Biostatistics, University of Washington, Seattle, WA 98195, USA.

HAITAO CHU, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21205, USA.

MICHAEL J. DANIELS, Department of Statistics, University of Florida, Gainesville, FL 32611, USA.

- Baker SG, Ko C-W, Graudbard BI. A sensitivity analysis for nonrandomly missing categorical data arising from a national health disability survey. Biostatistics. 2003;4:41–56. [PubMed]
- Chu H, Halloran ME. Estimating vaccine efficacy using auxiliary outcome data and a small validation set. Statistics in Medicine. 2004;23:2697–712. [PubMed]
- Damien P, Wakefield J, Walker S. Gibbs sampling for Bayesian non-conjugate and hierarchical models by using auxiliary variables. Journal of the Royal Statistical Society, Series B. 1999;61:331–44.
- Daniels MJ, Hogan J. Reparametrizing the pattern mixture model for sensitivity analyses under informative dropout. Biometrics. 2000;56:1241–8. [PubMed]
- Gaglani MJ, Piedra PA, Herschler GB, Griffith ME, Kozinetz CA, Riggs MW, Fewlass C, Halloran ME, Longini IM, Glezen P. Direct effectiveness of the intranaal, live-attenuated trivalent, cold-adapted, influenza virus vaccine (CAIV-T) against the 2000–2001 influenza A (H1N1) and B epidemic in healthy children. Archives of Pediatric and Adolescent Medicine. 2003;158:65–73. [PubMed]
- Gustafson P, Greenland S. Curious phenomena in Bayesian adjustment for exposure misclassification. Statistics in Medicine. 2006;25:87–103. [PubMed]
- Halloran ME, Longini IM. Using validation sets for outcomes and exposure to infection in vaccine field studies. American Journal of Epidemiology. 2001;154:391–8. [PubMed]
- Halloran ME, Longini IM, Gaglani MJ, Piedra PA, Chu H, Herschler GB, Glezen WP. Estimating efficacy of trivalent, cold-adapted, influenza virus vaccine (CAIV-T) against influenza A (H1N1) and B using surveillance cultures. American Journal of Epidemiology. 2003;158:305–11. [PubMed]
- Little RJA. A class of pattern-mixture models for normal missing data. Biometrika. 1994;81:471–83.
- Little RJA, Rubin DB. Statistical Analysis with Missing Data. 2nd edition John Wiley and Sons; Hoboken, NJ: 2002.
- Molenberghs G, Kenward MG, Goetghbeur EJT. Sensitivity analysis for incomplete contingency tables. Applied Statistics. 2001;50:15–29.
- Neal RM. Slice sampling (with discussion) Annals of Statistics. 2003;31:705–67.
- Pepe MS, Reilly M, Fleming TR. Auxiliary outcome data and the mean score method. Journal of Statistical Planning and Inference. 1994;42:137–60.
- Piedra PA, Gaglani M, Herschler G, Riggs M, Griffith M, Kozinetz C, Glezen WP. Safety and effectiveness of trivalent, cold-adapted influenza vaccine (CAIV-T) in children. In: Osterhaus ADME, Cox N, Hampson AW, editors. Options for Control of Influenza IV. Excerpta Medica; Amsterdam: 2001. pp. 939–43.
- Robins JM, Rotnitzky A, Scharfstein DO. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In: Halloran ME, Berry DA, editors. Statistics in Epidemiology, Environment and Clinical Trials. Springer; New York: 2000. pp. 1–94.
- Rotnitzky A, Robins JM, Scharfstein D. Semiparametric regression for repeated outcomes with non-ignorable non-response. Journal of the American Statistical Association. 1998;93:1321–39.
- Rotnitzky A, Scharfstein D, Su T, Robins JM. Methods for conducting sensitivity analysis of trials with potentially non-ignorable competing causes of censoring. Biometrics. 2001;57:103–13. [PubMed]
- Scharfstein DO, Daniels MJ, Robins JM. Incorporating prior beliefs about selection bias into the analysis of randomized trials with missing outcomes. Biostatistics. 2003;4:495–512. [PMC free article] [PubMed]
- Scharfstein DO, Rotnitzky A, Robins JM. Adjusting for nonignorable drop-out using semiparametric nonresponse models (with discussion) Journal of the American Statistical Association. 1999;94:1096–146.
- Shardell M. The analysis of informatively coarsened discrete time-to-event data. Johns Hopkins University; Baltimore, MD: 2005. PhD. Thesis.
- Tanner MA, Wong WH. The calculation of the posterior distributions by data augmentation. Journal of the American Statistical Association. 1987;82:528–40.
- Vansteelandt S, Goetghebeur E, Kenward MG, Molenberghs G. Ignorance and uncertainty regions as inferential tools in a sensitivity analysis. Statistica Sinica. 2006 in press.
- Verbeke G, Molenberghs G, Thijs H, Lesaffre E, Kenward MG. Sensitivity analysis for nonrandom dropout: a local influence approach. Biometrics. 2001;57:7–14. [PubMed]

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |