Home | About | Journals | Submit | Contact Us | Français |

**|**HHS Author Manuscripts**|**PMC2920568

Formats

Article sections

- Abstract
- 1. INTRODUCTION
- 2. BACKGROUND
- 3. POWER, ROBUSTNESS AND OPTIMALITY
- 4. CHOOSING EXTERNAL WEIGHTS
- 5. ESTIMATED WEIGHTS
- 6. EXAMPLES
- 7. DISCUSSION
- References

Authors

Related links

Stat Sci. Author manuscript; available in PMC 2010 August 12.

Published in final edited form as:

Stat Sci. 2009 November; 24(4): 398–413.

doi: 10.1214/09-STS289PMCID: PMC2920568

NIHMSID: NIHMS217150

Kathryn Roeder, Professor of Statistics, Department of Statistics, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA;

Kathryn Roeder: ude.umc.tats@redeor; Larry Wasserman: ude.umc.tats@namressaw

See other articles in PMC that cite the published article.

Genetic investigations often involve the testing of vast numbers of related hypotheses simultaneously. To control the overall error rate, a substantial penalty is required, making it difficult to detect signals of moderate strength. To improve the power in this setting, a number of authors have considered using weighted *p*-values, with the motivation often based upon the scientific plausibility of the hypotheses. We review this literature, derive optimal weights and show that the power is remarkably robust to misspecification of these weights. We consider two methods for choosing weights in practice. The first, external weighting, is based on prior information. The second, estimated weighting, uses the data to choose weights.

Testing for association between genetic variation and a complex disease typically requires scanning hundreds of thousands of genetic polymorphisms. In a multiple testing situation, such as a genome-wide association study (GWAS), the null hypothesis is rejected for any test that achieves a *p*-value less than a predetermined threshold (usually on the order of 10^{−8}). Data from these investigations has renewed interest in the multiple testing problem. The introduction of the false discovery rate and a procedure to control it by Benjamini and Hochberg (1995) inspired hope that this would be an effective way to control error while increasing power (Storey and Tibshirani, 2003; Sabatti, Service and Freimer, 2003). To further bolster power, recent statistical methods have been proposed that up-weight and down-weight hypotheses, based on prior likelihood of association with the phenotype (Genovese, Roeder and Wasserman, 2006; Roeder et al., 2006; Roeder, Wasserman and Devlin, 2007; Wang, Li and Bucan, 2007). Such prior information is often available in practice.

Weighted procedures multiply the threshold by the weight *w*, for each test, raising the threshold when *w >* 1 and lowering it if *w <* 1. To control the overall rate of false positives, a budget must be imposed on the weighting scheme, so that the average weight is one. If the weights are informative, the procedure improves power substantially, but, if the weights are uninformative, the loss in power is usually small. Surprisingly, aside from this budget requirement, any set of nonnegative weights is valid (Genovese, Roeder and Wasserman, 2006). While desirable in some respects, this flexibility makes it difficult to select weights for a particular analysis.

The first such weighting scheme appears to be Holm (1979). Related ideas can be found in Benjamini and Hochberg (1997), Chen et al. (2000), Genovese, Roeder and Wasserman (2006), Kropf et al. (2004), Rosenthal and Rubin (1983), Schuster, Kropf and Roeder (2004), Westfall and Krishen (2001), Westfall, Kropf and Finos (2004), Blanchard and Roquain (2008) and Roquain and van de Wiel (2008), among others. Several of these approaches use data dependent weights and yet maintain familywise error control. There are, of course, other ways to improve power aside from weighting. Some notable recent approaches include Rubin, Dudoit and van der Laan (2006), Storey (2007), Donoho and Jin (2004), Signoravitch (2006), Westfall, Krishen and Young (1998), Westfall and Soper (2001), Efron (2007) and Sun and Cai (2007). Of these, our approach is closest to Rubin, Dudoit and van der Laan (2006).

In some cases, the optimal weights can be estimated from the data. An approach developed by Westfall, Kropf and Finos (2004) utilizes quadratic forms to construct such weights; however, this approach assumes the individual measurements are normally distributed. This approach is suited to applications such as microarray data for which the observations are approximately normally distributed. We are interested in applications such as tests for genetic association. In this setting the individual observations are discrete, but the test statistics are approximately normally distributed.

In general, *p*-value weighting raises several important questions. How much power can we gain if we guess well in the weight assignment? How much power can we lose if we guess poorly? In this paper we show that the optimal weights have a simple parametric form and we investigate various approaches for estimating these weights. We also show the power is very robust to misspecification of the weights. In particular, in Section 3 we show that (i) sparse weights (few large weights and minimum weight close to 1) lead to huge power gains for well specified weights, but minute power loss for poorly specified weights; and (ii) in the nonsparse case, under weak conditions, the worst case power for poorly specified weights is typically better than the power obtained using equal weights.

We consider two methods for choosing the weights: (i) external weights, where prior information (based on scientific knowledge or prior data) singles out specific hypotheses (Section 4) and (ii) estimated weights where the data are used to construct weights (Section 5). External weights are prone to bias, while estimated weights are prone to variability. The two robustness properties reduce concerns about bias and variance.

To motivate this work consider an example (Figure 1) of external weighting that arises in genetic epidemiology. To identify variants of genes that induce greater susceptability to disease, two types of studies (linkage and association) are often performed. Whole genome linkage analysis has been conducted for most major diseases. These data can be summarized by a linkage trace, a smooth stochastic process {*Z*(*s*): *s* [0*, L*]} where each *s* corresponds to a location on the genome. At points that correspond to a variant of a gene of interest, the mean of the process *μ*(*s*) = *E*(*Z*(*s*)) is a large positive value; however, due to extensive spatial correlation in the process, *μ*(*s*) is also nonzero in the vicinity of the variant. Tests for association between genetic polymorphisms and disease status for each of many genetic markers across the genome are also of interest. Like linkage analysis, the association statistics {*T _{j}*:

Consider a multiple testing situation in which *m* tests are being performed. Suppose *m*_{0} of the null hypotheses are true and *m*_{1} = *m* − *m*_{0} null hypotheses are false. We can categorize the *m* tests as in Table 1. In this notation *F* is the number of false positives. To control the familywise error rate, it is traditional to bound *P* (*F >* 0) at *α*. When the tests are independent, the simplest way to control this probability is to reject only those tests for which the *p*-value is less than *α/m*; this is called the Bonferroni procedure.

In 1995 Benjamini and Hochberg (BH) introduced a new approach to multiple hypothesis testing that controls the false discovery rate (FDR), defined as the expected fraction of false rejections among those hypotheses rejected. Let *P*_{(1)} *<* ··· *< P*_{(}_{m}_{)} be the ordered *p*-values from *m* hypothesis tests, with *P*_{(0)} 0. Then, the BH procedure rejects any null hypothesis for which *P* ≤ *T* with

$$T=max\{{P}_{(i)}:{P}_{(i)}\le \frac{\alpha i}{m}\}.$$

This quantity is of more scientific relevance than the overall type I error rate in GWAS. Also, the procedure is more powerful than the Bonferroni method. Adaptive variants of the procedure can increase power further at little additional computational expense; see Benjamini, Krieger and Yekutieli (2006) and Storey (2002).

BH controls the false discovery rate at level *αm*_{0}*/m*, where *m*_{0} is the number of true null hypotheses. With certain dependence assumptions on the *p*-values, this is true regardless of how many nulls are true and regardless of the distribution of the *p*-values under the alternatives (Benjamini and Yekutieli, 2001; Blanchard and Roquain, 2009; Sarkar, 2002). Under some distributional assumptions, Genovese and Wasserman (2002) show that, asymptotically, the BH method corresponds to rejecting all *p*-values less than a particular *p*-value threshold *u*^{*}. Specifically, *u*^{*} is the solution to the equation *H* (*u*) = *βu* and
$\beta =({\scriptstyle \frac{1}{\alpha}}-{A}_{0})/(1-{A}_{0})$, where *A*_{0} = *m*_{0}*/m* and *H* is the (common) distribution of the *p*-value under the alternative. The key result is that *α/m* ≤ *u*^{*} ≤ *α*, which shows that the BH method is intermediate between Bonferroni (corresponding to *α/m*) and uncorrected testing (corresponding to *α*). If *A*_{0} is close to 0, however, as it usually is in GWA, then *β* is a very large quantity and the power of the FDR is not much improved over the Bonferroni procedure.

The power of the BH method can be improved with adaptations. Blanchard and Roquain (2008) have given numerical comparisons of different adaptive procedures under dependence. Romano, Shaikh and Wolf (2008) have considered improving the adaptive procedure of Benjamini, Krieger and Yekutieli (2006) using the bootstrap. Sarkar and Heller (2008) have noted that the adaptive procedure of Benjamini et al. may not perform well compared to Storey’s (2002) procedure for certain parameter choices.

We are given hypotheses *H* = (*H*_{1}*,* …*, H _{m}*) and standardized test statistics

The *p*-values associated with the tests are *P* = (*P*_{1}*,* …*, P _{m}*), where

A *rejection set* is a subset of {1*,* …*, m*}. Say that *controls familywise error at level α* if ( ∩ ) ≤ *α*, where = {*j: H _{j}* = 0}. The

The weighted Bonferroni procedure (Rosenthal and Rubin, 1983; Genovese, Roeder and Wasserman, 2006) is as follows. Specify nonnegative weights *w* = (*w*_{1}, …*, w _{m}*) and reject hypothesis

$$j\in \mathcal{R}=\{j:\frac{{P}_{j}}{{w}_{j}}\le \frac{\alpha}{m}\}.$$

(1)

In the following lemma we show that as long as *m*^{−1}Σ* _{j} w_{j}* = 1, the rejection set controls familywise error at level

If = 1, then controls familywise error at level α.

Suppose that W_{j} = g(V_{j}, c), j = 1, …, m, for some random variables V_{1}, …, V_{m}, some constant c and some function g. Further, suppose that V_{j} has a known distribution H whenever j and that P_{j} is independent of V_{j} for all j . The rule that rejects when P_{j} ≤ αW_{j}/m controls familywise error at level α if c is chosen to satisfy (g(V_{j}, c)) ≤ 1.

Genovese, Roeder and Wasserman (2006) also showed that false discovery methods benefit by weighting. Recall that the false discovery proportion (FDP) is

$$\text{FDP}=\frac{\text{number}\phantom{\rule{0.16667em}{0ex}}\text{of}\phantom{\rule{0.16667em}{0ex}}\text{false}\phantom{\rule{0.16667em}{0ex}}\text{rejections}}{\text{number}\phantom{\rule{0.16667em}{0ex}}\text{of}\phantom{\rule{0.16667em}{0ex}}\text{rejections}}=\frac{\mid \mathcal{R}\cap {\mathcal{H}}_{0}\mid}{\mid \mathcal{R}\mid},$$

where the ratio is defined to be 0 if the denominator is 0. The false discovery rate (FDR) is FDR = (FDP). Benjamini and Hochberg (1995) proved FDR ≤ *α* if = {*j: P*_{(}_{j}_{)} ≤ *T*} where *T* = max{*j: P*_{(}_{j}_{)} ≤ *j α/m*}. Genovese, Roeder and Wasserman (2006) showed that FDR ≤ *α* if the *P _{j}* ’s are replaced by

The optimal weights, derived below, can be re-expressed as optimal cutoffs for testing. Specifically, rejecting when *P _{j}/w_{j}* ≤

The power of a single, one-sided alternative in the unweighted case (*w _{j}* = 1) is

$$\pi ({\xi}_{j},1)=\mathbb{P}({T}_{j}>{z}_{\alpha /m})=\overline{\mathrm{\Phi}}({z}_{\alpha /m}-{\xi}_{j}).$$

The power^{1} in the weighted case is

$$\begin{array}{l}\pi ({\xi}_{j},{w}_{j})=\mathbb{P}\left({P}_{j}<\frac{\alpha {w}_{j}}{m}\right)\\ =\mathbb{P}\left({T}_{j}>{\overline{\mathrm{\Phi}}}^{-1}\left(\frac{\alpha {w}_{j}}{m}\right)\right)\\ =\overline{\mathrm{\Phi}}({z}_{\alpha {w}_{j}/m}-{\xi}_{j}).\end{array}$$

(2)

Weighting increases the power when *w _{j} >* 1 and decreases the power when

Given *θ* = (*ξ*_{1}*,* …*, ξ _{m}*) and

$$\frac{1}{{m}_{1}}\sum _{j=1}^{m}\pi ({\xi}_{j},{w}_{j})I({\xi}_{j}>0),$$

where
${m}_{1}={\sum}_{j=1}^{m}I({\xi}_{j}>0)$. More generally, if *ξ* is drawn from a distribution *Q* and *w* = *w*(*ξ*) is a weight function, we define the average power ∫*π*( *ξ, w*(*ξ*))*I* (*ξ >* 0) *dQ*(*ξ*)*/*∫*I* (*ξ >* 0) *dQ*(*ξ*). If we take *Q* to be the empirical distribution of (*ξ*_{1}*,* …*, ξ _{m}*), then this reduces to the previous expression. In this formulation we require

In the following theorem we see that the set of optimal weight functions form a one parameter family indexed by a constant *c*.

Given θ = (ξ_{1}, …, ξ_{m}), the optimal weight vector w = (w_{1}, …, w_{m}) that maximizes the average power subject to w_{j} ≥ 0 and = 1 is w = (ρ_{c}(ξ_{1}), …, ρ_{c}(ξ_{m})), where

$${\rho}_{c}(\xi )=\left(\frac{m}{\alpha}\right)\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{c}{\xi}\right)I(\xi >0),$$

(3)

and c c(θ) is defined by the condition

$$\frac{1}{m}\sum _{j=1}^{m}{\rho}_{c}({\xi}_{j})=1.$$

(4)

The proof, essentially a special case of Spjøtvoll (1972), is in the Appendix. Figure 2 displays the function *ρ _{c}*(

The power at an alternative with mean ξ under optimal weights is (c/ξ − ξ/2). The average power under optimal weights, which we call the oracle power, is

$$\frac{1}{{m}_{1}}\sum _{j=1}^{m}\overline{\mathrm{\Phi}}\left(\frac{c}{{\xi}_{j}}-\frac{{\xi}_{j}}{2}\right)I({\xi}_{j}>0),$$

where m_{1} =Σ_{j} I (ξ_{j} > 0).

The oracle power is not attainable since the optimal weights depend on *θ* = (*ξ*_{1}*,* …*, ξ _{m}*). In practice, the weights will either be chosen by prior information or by estimating the

Sparse weights (minimum weight close to 1) are highly robust. If most weights are less than 1 and the minimum weight is close to 1, then correct specification (large weights on alternatives) leads to large power gains but incorrect specification (large weights on nulls) leads to little power loss.

Worst case analysis. Weighted hypothesis testing, even with poorly chosen weights, typically does as well or better than Bonferroni except when the the alternative means are large, in which both have high power.

Let us now make these statements precise. Also, see Genovese, Roeder and Wasserman (2006) and Roeder et al. (2006) for other results on the effect of weight misspecification.

Consider first the case where the weights take two distinct values and the alternatives have a common mean *ξ*. Let *ε* denote the fraction of hypotheses given the larger of the two values of the weights *B*. Then, the weight vector *w* is proportional to

$$(\underset{k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{B,\dots ,B}},\underset{m-k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{1,\dots ,1}}),$$

where *k* = *εm* and *B >* 1*,* and, hence, the normalized weights are

$$w=(\underset{k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{{w}_{1},\dots ,{w}_{1}}},\underset{m-k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{{w}_{0},\dots ,{w}_{0}}}),$$

where

$${w}_{1}=\frac{B}{\epsilon B+(1-\epsilon )},\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}{w}_{0}=\frac{1}{\epsilon B+(1-\epsilon )}.$$

We say that the weights are sparse if *ε* is small. Provided *B* is considerably less than 1*/ε*, most weights are near 1 in the sparse case.

Rather than investigate the average power, we focus on a single alternative with mean *ξ*. The power gain by up-weighting this hypothesis is the power under weight *w*_{1} minus the unweighted power *π*( *ξ, w*_{1}) − *π*( *ξ,* 1). Similarly, the power loss for down-weighting is *ρ*(*ξ,* 1) − *π*( *ξ, w*_{0}). The gain minus the loss, which we call the robustness function, is

$$\begin{array}{l}R(B,\epsilon )\equiv (\pi (\xi ,{w}_{1})-\pi (\xi ,1))+(\pi (\xi ,1)-\pi (\xi ,{w}_{0}))\\ =\overline{\mathrm{\Phi}}({z}_{\alpha {w}_{1}/m}-\xi )+\overline{\mathrm{\Phi}}({z}_{\alpha {w}_{0}/m}-\xi )-2\overline{\mathrm{\Phi}}({z}_{\alpha /m}-\xi ).\end{array}$$

The gain outweighs the loss if and only if *R*(*B, ε*) *>* 0 (Figure 3).

Robustness function for m = 1000. In this example, ξ = z_{α/m} which has power 1/2 without weighting. The gain of correct weighting far outweighs the loss for incorrect weighting as long as the fraction of large weights ε is small. **...**

In the sparse weighting scenario *k* is small and *w*_{0} ≈ 1 by assumption, consequently, an analysis of *R*(*B, ε*) sheds light on the effect of weighting on power, without the added complications involved in a full analysis of average power.

Fix B > 1. Then, lim_{ε→0} R(B, ε) > 0. Moreover, there exists ε^{*}(B) > 0 such that R(B, ε) > 0 for all ε < ε^{*}(B).

We can generalize this beyond the two-valued case as follows. Let *w* be any weight vector such that = 1. Now define the (worst case) robustness function

$$R(\xi )\equiv \underset{\{j:{w}_{j}>1,{H}_{j}=1\}}{min}\{\pi (\xi ,{w}_{j})-\pi (\xi ,1)\}-\underset{\{j:{w}_{j}<1,{H}_{j}=1\}}{max}\{\pi (\xi ,1)-\pi (\xi ,{w}_{j})\}.$$

We will see that *R*(*ξ*) *>* 0 under weak conditions and that the maximal robustness is obtained for *ξ* near the Bonferroni cutoff *z _{α/m}*.

A necessary and sufficient condition for R(ξ) > 0 is

$${R}_{b,B}(\xi )\equiv \mathrm{\Phi}({z}_{\alpha B/m}-\xi )+\mathrm{\Phi}({z}_{\alpha b/m}-\xi )-2\mathrm{\Phi}({z}_{\alpha /m}-\xi )\le 0,$$

(5)

where *B* = min{*w _{j}*:

$${R}_{b,B}(\xi )=-\mathrm{\Delta}(\xi )+O(1-b),$$

where

$$\mathrm{\Delta}(\xi )=(\mathrm{\Phi}({z}_{\alpha /m}-\xi )-\mathrm{\Phi}({z}_{\alpha B/m}-\xi ))>0$$

and, as b → 1, μ({ξ: R(ξ) < 0}) → 0 and inf_{ξ} R(ξ) → 0.

Based on the theorem, we see that there is overwhelming robustness as long as the minimum weight is near 1. Even in the extreme case *b* = 0, there is still a safe zone, an interval of values of *ξ* over which *R*(*ξ*) *>* 0.

Suppose that B ≥ 2. Then there exists ξ_{*} > 0 such that R_{B,b}(ξ) > 0 for all 0 ≤ ξ ≤ ξ_{*} and all b. An upper bound on ξ_{*} is z_{α/m} − 1/(z_{α/m} − z_{Bα/m}).

Even if the weights are not sparse, the power of the weighted test tends to be acceptable.

The result holds even though the weights themselves can be very sensitive to changes in *θ*. Consider the following example. Suppose that *θ* = (*ξ*_{1}*,* …*, ξ _{m}*) where each

$$\begin{array}{l}A={\overline{\mathrm{\Phi}}}^{-1}\left(\frac{\alpha}{(m(\gamma K+a))}\right),\\ B={\overline{\mathrm{\Phi}}}^{-1}\left(\frac{K\alpha}{(m(\gamma K+a))}\right).\end{array}$$

This arrangement yields weights *w*_{0} and *w*_{1} on *u* and *ξ* such that *w*_{0}*/w*_{1} = *K*. For example, if *m* = 1000, *α* = 0.05, *a* = 0.1, *γ* = 0.1, *K* = 1000 and *c* = 0.1, then *u* = 0.03 and *ξ* = 9.8. The optimal weight on *ξ* under *Q* is 10 but under it is 0.00999 and so is reduced by a factor of 1001. More generally, we have the following result which shows that the weights are, in a certain sense, a discontinuous function of *θ*.

Fix α and m. For any δ > 0 and ε > 0 there exists Q = (1 − a)δ_{0} + aδ_{ξ} and = (1 − a − γ)δ_{0} + γδ_{u} + aδ_{ξ} such that d(Q, ) < δ, and (ξ)/ρ(ξ) < ε, where a = α/4, d(Q, ) = sup_{ξ} |Q(−∞, ξ], (−∞, ξ]| is the Kolmogorov–Smnirnov distance, ρ is the optimal weight function for Q and is the optimal weight function for .

Fortunately, this feature of the weight function does not pose a serious hurdle in practice because it is possible to have high power even with poor weights. In Figure 4 the plots on the left show the power as a function of the alternative mean *ξ*. The dark solid line shows the lowest possible power assuming the weights were estimated as poorly as possible (under conditions specified below). The lighter solid line is the power of the unweighted (Bonferroni) method. The dotted line shows the power under theoretically optimal weights. The worst case weighted power is typically close to or larger than the Bonferroni power except for large *ξ* when they are both large.

Power as a function of the alternative mean ξ. In these plots, a = 0.01, m = 1000 and α = 0.05. There are (1 − a)m nulls and ma alternatives with mean ξ. The left plots show what happens when the weights are incorrectly **...**

To begin formal analysis, assume that each mean is either equal to 0 or *ξ* for some fixed *ξ >* 0. Thus, the empirical distribution is *Q* = (1 − *a*) *δ*_{0} + *aδ _{ξ}*, where

As the weights are a function of *c*, we first need to find *c* as a function of *u*. The normalization condition (4) reduces to

$$\gamma \overline{\mathrm{\Phi}}\left(\frac{u}{2}+\frac{c}{u}\right)+a\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{c}{\xi}\right)=\frac{\alpha}{m},$$

(6)

which implicitly defines the function *c*(*u*). First we consider what happens when *u* is restricted to be less than *ξ*.

Assume that α/m ≤ γ + a ≤ 1. Let Q = (1 − a)δ_{0} + aδ_{ξ} and = (1 − a − γ)δ_{0} + γδ_{u} + aδ_{ξ} with 0 ≤ u ≤ ξ. Let C(ξ) = sup_{0≤u≤ ξ}c(u) and define ξ_{0} = z_{α/(m(γ+a))}:

- For ξ ≤ ξ
_{0}, C(ξ) = ξξ_{0}− ξ^{2}/2. For ξ> ξ_{0}, C(ξ) is the solution to$$\gamma \overline{\mathrm{\Phi}}(\sqrt{2c})+a\overline{\mathrm{\Phi}}\left(\frac{c}{\xi}+\frac{\xi}{2}\right)=\frac{\alpha}{m}.$$In this case, $C(\xi )={z}_{\alpha /(m\gamma )}^{2}/2+O(a)$. - Let ${\xi}_{\ast}={z}_{\alpha /m}+\sqrt{{z}_{\alpha /m}^{2}-{z}_{q}^{2}}$, where q = α(1 − α)/(mγ). For ξ< ξ
_{*},$$\underset{0<u<\xi}{inf}\pi (u)\ge {\pi}_{\text{Bonf}}.$$(7)For ξ ≥ ξ_{*}we have$$\underset{0<u<\xi}{inf}\pi (u)\ge \overline{\mathrm{\Phi}}\left(\frac{{z}_{\alpha /(m\gamma )}^{2}-{\xi}_{\ast}^{2}}{2{\xi}_{\ast}}\right)-O(a)$$(8)$$\approx 1-\overline{\mathrm{\Phi}}\left(\sqrt{2log\frac{1-a}{\gamma}}\right)-O(a)$$(9)$$\ge 1-\frac{\gamma}{1-a}-O(a).$$(10)The factor $\overline{\mathrm{\Phi}}(\sqrt{2log(1-a)/\gamma})\approx \gamma /(1-a)$ is the worst case power deficit due to misspecification. Now we drop the assumption that*u*≤*ξ*.

Let Q = (1 − a)δ_{0} + aδ_{ξ} and let Q_{u} (1 − a − γ)δ_{0} + γδ_{u} + aδ_{ξ}. Let π_{u} denote the power at ξ using the weights computed under Q_{u}.

- The least favorable u is ${u}_{\ast}\equiv arg{min}_{u\ge 0}{\pi}_{u}=\sqrt{2{c}_{\ast}}={z}_{\alpha /(m\gamma )}+O(a)$, where c
_{*}solves$$\gamma \overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}})+a\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{{c}_{\ast}}{\xi}\right)=\frac{\alpha}{m}$$and ${c}_{\ast}={z}_{\alpha /(m\gamma )}^{2}/2+O(a)$. - The minimal power is$$\underset{u}{inf}{\pi}_{u}=\overline{\mathrm{\Phi}}\left(\frac{{c}_{\ast}}{\xi}-\frac{\xi}{2}\right)=\overline{\mathrm{\Phi}}\left(\frac{{z}_{\alpha /(m\gamma )}^{2}-{\xi}^{2}}{2\xi}\right)+O(a).$$
- A sufficient condition for inf
_{u}π_{u}to be larger than the power of the Bonferroni method is $\xi \ge {z}_{\alpha /m}+\sqrt{{z}_{\alpha /m}^{2}-{z}_{\alpha /(m\gamma )}^{2}}+O(a)$.

One approach to choosing external weights (or test statistic cutoffs) is to use empirical Bayes methods to model prior information while being careful to preserve error control as in Westfall and Soper (2001), for example. Here we consider a simple method that takes advantage of the robustness properties we have discussed. We will focus here on the two-valued case. Thus,

$$w=(\underset{k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{{w}_{1},\dots ,{w}_{1}}},\underset{m-k\phantom{\rule{0.16667em}{0ex}}\text{terms}}{\underbrace{{w}_{0},\dots ,{w}_{0}}}),$$

where *k* = *εm*, *w*_{1} = *B/*(*εB* + (1 − *ε*)) and *w*_{0} = 1*/*(*εB* + (1 − *ε*)). In practice, we would typically have a fixed fraction of hypotheses *ε* that we want to give more weight to. The question is how to choose *B*. We will focus on choosing *B* to produce weights with good properties at interesting values of *ξ*. Now large values of *ξ* already have high power. Very small values of *ξ* have extremely low power and benefit little by weighting. This leads us to focus on constructing weights that are useful for a *marginal effect*, defined as the alternative *ξ*_{0} that has power 1/2 when given weight 1. Thus, the marginal effect is *ξ*_{0} = *z _{α/m}*. In the rest of this section then we assume that all nonzero

Fix 0 *< ε <* 1 and vary *B*. As we increase *B*, we will eventually reach a point *B*_{0}(*ε*) where *R*(*B, ε*) *<* 0, which we call the turnaround point. Formally, *B*_{0}(*ε*) = sup{*B: R*(*B, ε*) *>* 0}. The top panel in Figure 5 shows *B*_{0}(*ε*) versus *ε*which shows that for small *ε* we can choose *B* large without loss of power. The bottom panel shows *R*(*B, ε*) for *ε* = 0.1. Ideally, for a given *ε,* one chooses *B* near *B*_{*}(*ε*), the value of *B* that maximizes *R*(*B, ε*).

Top plot: turnaround point B_{0}(ε) versus ε. Bottom plot shows the robustness function R(B, 0.1) versus B. The turnaround point B_{0}(ε) is shown with a vertical dotted line.

Fix 0 < ε < 1. As a function of B, R(B, ε) is unimodal and satisfies R(1, ε) = 1, R′(1, ε) > 0 and R(∞, ε) < 0. Hence, B_{0}(ε) exists and is unique. Also, R(B, ε) has a unique maximum at some point B^{*}(ε) and R(B^{*}(ε), ε) > 0.

When *ε* is very small, *B* can be large, provided *w*_{0} ≈ 1. For example, suppose we want to increase the chance of rejecting one particular hypothesis so that *ε*= 1*/m*. Then,

$${w}_{1}=\frac{mB}{B+m-1}\approx B,\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}{w}_{0}=\frac{1}{B+m-1}\approx 1$$

and

$$\begin{array}{l}\underset{m\to \infty}{lim}\underset{B\to \infty}{lim}\pi ({\xi}_{j},{w}_{1})=1,\\ \text{while}\phantom{\rule{0.16667em}{0ex}}\underset{m\to \infty}{lim}\underset{B\to \infty}{lim}\pi ({\xi}_{j},{w}_{0})=\frac{1}{2}.\end{array}$$

The next results show that binary weighting schemes are optimal in a certain sense. Suppose we want to have at least a fraction *ε* with high power 1 − *β* and otherwise we want to maximize the minimum power.

Consider the following optimization problem: Given 0 < ε < 1 and 0 < β < 1/2, find a vector w = (w_{1}, …, w_{m}) that maximizes min_{j} π(ξ_{m}, w_{j} ) subject = 1, and #{j: π(w_{j}, ξ_{m}) ≥ 1 − β}/m ≥ ε. The solution is given by c = (z_{α/m} + z_{1−β} ), B = cm(1 − ε)/(α − εcm), w_{1} = B/(εB + (1 − ε)), w_{0} = 1/(εB + (1 − ε)) and k = εm.

If our goal is to maximize the number of alternatives with high power while maintaining a minimum power loss, the solution is given as follows.

Consider the following optimization problem: Given 0 < β < 1/2, find a vector w = (w_{1}, …, w_{m}) that maximizes #{j: π(w_{j}, ξ_{m}) ≥ 1 − β} subject to = 1, and min_{j} π(w_{j}, ξ_{m}) ≥ δ. The solution is

$$\begin{array}{l}{w}_{1}=\frac{m}{\alpha}\overline{\mathrm{\Phi}}({z}_{\alpha /m}+{z}_{1-\beta}),\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}{w}_{0}=\frac{m}{\alpha}\overline{\mathrm{\Phi}}({z}_{\alpha /m}+{z}_{\delta}),\\ \epsilon =\frac{1-{w}_{0}}{{w}_{1}-{w}_{0}}\end{array}$$

and k = mε.

A special case that falls under this theorem permits the minimum power to be 0. In this case *w*_{0} = 0 and *ε*= 1*/w*_{1}.

In practice, *ξ _{j}* is not known, so it must be estimated to utilize the weight function. A natural choice is to build on the two stage experimental design (Satagopan and Elston, 2003; Wang et al., 2006) and split the data into subsets, using one subset to estimate

To gain a strong advantage with data-based weights, prior information is needed. One option is to order the tests (Rubin, Dudoit and van der Laan, 2006), but with a large number of tests this can be challenging. The type of prior information readily available to investigators is often nonspecific. For instance, SNPs might naturally be grouped, based on features that make various candidates more promising for this disease under investigation. For a brain-disorder phenotype we might cross-classify SNPs by categorical variables such as functionality, brain expression and so forth. The SNPs in one group may seem most promising, a priori, while those in another seem least promising. Intermediate groups may be somewhat ambiguous. It is easy to imagine additional variables that further partition the SNPs into various classes that help to separate the more promising SNPs from the others. While this type of information lends itself to grouping SNPs, it does not lead directly to weights for the groups. Indeed, it might not even be possible to choose a natural ordering of the groups. What is needed is a way to use the data to determine the weights, once the groups are formed.

Until recently, methods for weighted multiple-testing required that prior weights be developed independently of the data under investigation (Genovese, Roeder and Wasserman, 2006; Roeder, Wasserman and Devlin, 2007). Here we provide a data-based estimate of weights based on results of grouped analysis. One way to implement this approach is to follow these steps:

- Partition the tests into subsets , …
*,*, with the*k*th group containing*r*elements, ensuring that_{k}*r*is at least 20–30._{k} - Calculate the sample mean and variance ${S}_{k}^{2}$ for the test statistics in each group.
- Label the
*i*th test in group*k*,*T*. At best, only a fraction of the elements in each group will have a signal, hence, we assume that for_{ik}*i*= 1*,*…*, r*the distribution of the test statistics is approximated by a mixture model_{k}$${T}_{ik}\sim (1-{\pi}_{k})N(0,1)+{\pi}_{k}N({\xi}_{k},1)$$or$${T}_{ik}\sim (1-{\pi}_{k}){\chi}_{1}^{2}(0)+{\pi}_{k}{\chi}_{1}^{2}({\xi}_{k}^{2}),$$where*ξ*is the signal size for those tests with a signal in the_{k}*k*th group. This is an approximation because the signal is likely to vary across tests. The mixture of normals is only appropriate when the tests are one-sided. For two-sided alternatives, the*χ*^{2}is the natural approach. This test squares the noncentrality parameter, effectively removing any ambiguity about the direction of the associations. - Estimate (
*π*) using the method of moments estimator (for details see the Appendix). Because_{k}, ξ_{k}*ξ*has no meaning when_{k}*π*= 0, the_{k}is set to 0 when_{k}is close to zero. For the normal model the estimators are_{k}$${\widehat{\pi}}_{k}={Y}_{k}^{2}/({Y}_{k}^{2}+{S}_{k}^{2}-1),\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}{\widehat{\xi}}_{k}={Y}_{k}/{\pi}_{k},$$(11)provided1_{k}>*/r*; otherwise_{k}= 0._{k}For the*χ*^{2}model they are$${\widehat{\xi}}^{2}=\frac{({S}_{k}^{2}+{Y}_{k}^{2}+3)}{{Y}_{k}-1},\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}\phantom{\rule{0.38889em}{0ex}}{\widehat{\pi}}_{k}=\frac{{Y}_{k}-1}{{\widehat{\xi}}_{k}^{2}},$$(12)provided*Y*> 1 and 1_{k}*/r*(_{k}<_{k}<*r*− 1)_{k}*/r*; otherwise_{k}= 0._{k} - For each of the
*k*groups, construct weights*w*(). It is apparent in Figure 1 that if |_{k}|_{k}*< δ*, for*δ*near 0, then*w*() ≈ 0 and it is unlikely that any tests in the_{k}*k*th group will be significant, regardless of the*p*-value. The stochastic quantity*δ*depends upon the relative values of (_{1}*,*…*,*), and the number of elements in each group. For this reason we have found that smoothing the weights generally improves the power of the procedure. We suggest using a linear combination such as_{K}$${\widehat{w}}_{k}=(1-\gamma )w({\widehat{\xi}}_{k})+\gamma {K}^{-1}\sum _{k}w({\widehat{\xi}}_{k}),$$with*γ*= 0.01 or 0.05. The larger the choice of*γ*, the more evenly distributed the weights across groups. Alternatively, one could smooth the weights by using a Stein shrinkage estimator or bagging procedure to obtain a more robust estimator of (*ξ*_{1}*,*…*, ξ*) (Hastie, Tibshirani and Friedman, 2001). Regardless of how the weights are smoothed, one should renorm them to ensure the weights sum to_{K}*m*. Each test in group*k*receives the weight*ŵ*. Another effect of the smoothing is to ensure that each group gets a weight greater than 0._{k}

This weighting scheme relies on data-based estimators of the optimal weights, but with a partition of the data sufficiently crude to preserve the control of family-wise error rate. The approach is an example of the “sieve principle” (Bickel et al., 1993). The sieve principle works because the number of parameters estimated is far less than the number of observations. Thus, many observations are used to estimate each parameter. Consequently, parameters are estimated with substantially less variability than if they were estimated using only the test statistics from the particular gene under investigation. Because the weights are determined by the size of the tests in the entire cluster, the probability of upweighting simply because a single test is large, due to chance, is small.

In a study of nicotine dependence, Saccone et al. (2007) used binary weights in a candidate gene study. Their study involved 3713 genetic variants (single nucleotide polymorphisms or SNPs) encompassing 348 genes. The genes were divided into two types: 52 nicoinic and dopaminergic receptor genes; and 296 other candidate genes. Each SNP associated with a gene in the first group was allocated ten times the weight of a gene in the other category. Using a generous false discovery rate (*α* = 0.4), they identified 39 SNPs; 78% of these were nicotine receptors, in contrast to the fraction of nicotine receptors overall (15%).

For family-based study designs, tests of association are based on transmission data. In these studies, data are available from which one can compute the potential power to detect a signal at each SNP tested; see Ionita-Laza et al. (2007) for a detailed explanation of this unique feature of family-based data. Because the data used to calculate the power are independent of the test statistics for association, these data are available for construction of the weights. Motivated by this possibility, Ionita-Laza et al. (2007) developed a weighting scheme. Using independent data, they ranked the SNPs from most to least promising, in terms of power. They then constructed an exponential weighting scheme, based on simulations of genetic models. The scheme results in a small number of SNPs receiving a top weight, successively more SNPs receiving correspondingly lower weights, and finally a large number receiving the lowest weight. In their simulations they found that the power of the test can often be doubled using this procedure. Using the FHS data, they apply the technique to a genome-wide association study with 116,204 SNPs and 923 participants. The phenotype of interest is height. Using their weighting scheme, they obtained one significant result with weights and none without weights.

Finding variation in the genetic code that increases the risk for complex diseases, such as Type II diabetes and schizophrenia, is critically important to the advancement of genetic epidemiology. In the Introduction we describe a means by which weights could be extracted from linkage data. Here we illustrate the idea with both data and simulations.

In the analysis of 955 cases and 1498 controls enrolled in a genome-wide association study, McQueen and colleagues (2008) used weights derived from published linkage results. They combined results from 11 linkage studies on bipolar disorder to obtain *Z* scores corresponding to the locations of each association test. From the linkage results they computed weighted *p*-values using the cummulative normal weight function (Roeder et al., 2006). Although none of their results were genome-wide significant, they obtained promising results in four regions. Three of these are obtained due to strong *p*-values in combination with a linkage peak. One signal did not correspond to a linkage peak, but continued to be in the top tier of *p*-values, after weights were applied.

To illustrate how binary weights could be derived from such linkage data, we present a realistic synthetic example. Using the methods described in Roeder et al. (2006), we create a linkage trace that captures many of the features found in actual linkage traces. In this simulation we generate a full genome (23 chromosomes) and place 20 disease variants at random, one per chromosome. The signals from these variants were designed to yield weak signals with broad peaks. Next, we simulated 100,000 normally distributed association test statistics mapped to the same genome. Again, 20 of these tests were generated under the alternative hypothesis of association. These signals were also weak.

To illustrate the synthetic data, six typical chromosomes are displayed in Figure 1. Each displayed chromosome has one true signal, with the association test statistic at that location indicated by an upspike; none of the association tests generated under the null hypothesis are plotted. Without weights, only 2 of the 20 signals could be detected using a Bonferonni correction. Using binary weights, as described above, with *ε* = 0.05 and *B* = 10, we discover 5 of the 20 signals. In the left column of the figure all three signals were discovered, while in the right column none were discovered (indicated by presence of a down-spike). Comparing the top row, we see that both signals were up-weighted in the correct location, but the association signal was not strong enough in the top right chromosome to achieve significance. Alternatively, in the bottom left panel the association statistic was substantial enough to reject the null hypothesis without the benefit of up-weighting.

To examine the robustness of the procedure to choice of weights, we tried 4 choices of *ε* (0.01, 0.05, 0.1, 0.2) with 1 ≤ *B* ≤ 50. We made no false discoveries with any of these choices. The power is displayed in Figure 6. To assist in the choice of parameters, we have found it helpful to examine the number of discoveries for each choice. In this example, the number of discoveries varied between 2 (unweighted, i.e., *B* = 1) to 6 (*ε* = 0.2, *B* ≥ 10). Five discoveries were made for a broad range of choices. In principle, choosing (*ε, B*) to maximize the number of discoveries can inflate the error rate. In our simulations we have found that searching within the family of weights defined by 1 or 2 parameters, such as this binary weight system based upon a linkage trace, tends to provide very close to nominal protection against false discoveries.

Several authors have explored the effect of weights on the power of multiple testing procedures [e.g., Westfall, Kropf and Finos (2004)]. These investigations show that the power of multiple testing procedures can be increased by using weighted *p*-values. Here we derive the optimal weights for a commonly used family of tests and show that the power is remarkably robust to misspecification of these weights.

The same ideas used here can be applied to other testing methods to improve power. In particular, weights can be added to the FDR method, Holm’s step-down test, and the Donoho and Jin (2004) method. Weighting ideas can also be used for confidence intervals. Another open question is the connection with Bayesian methods which have already been developed to some extent in Efron et al. (2001).

GWAS for some phenotypes such as Type 1 diabetes have yielded exciting results (Todd et al., 2007), while results for other complex diseases have been much less successful. Presumably, many studies do not have sufficient power to detect the genetic variants associated with the phenotypes, even though thousands of cases and controls have been genotyped. To bolster power, we recommend up-weighting and down-weighting hypotheses, based on prior likelihood of association with the phenotype. For instance, Wang, Li and Bucan (2007) describe pathway-based approaches for the analysis of GWAS.

Multiple testing arises in GWAS analyses in other contexts as well. Frequently, multiple tests, assuming different genetic models, are applied to each genetic marker. Multiple markers in a neighborhood can be analyzed simultaneously to increase the signal, using haplotypes, multivariate models and fine-mapping techniques. Data are often collected in multiple stages of the experiment, and at each stage promising markers are tested for association. In summary, many questions concerning multiple testing remain open in the context of GWAS.

The authors thank Jamie Robins for helping us to clarify several issues. Research supported in part by National Institute of Mental Health Grant MH057881 and NSF Grant AST 0434343.

The familywise error is

$$\begin{array}{l}\mathbb{P}((\mathcal{R}\cap {\mathcal{H}}_{0})>0)=\mathbb{P}\left({P}_{j}\le \frac{\alpha {w}_{j}}{m}\phantom{\rule{0.16667em}{0ex}}\text{for}\phantom{\rule{0.16667em}{0ex}}\text{some}\phantom{\rule{0.16667em}{0ex}}j\in {\mathcal{H}}_{0}\right)\\ \le \sum _{j\in {\mathcal{H}}_{0}}\mathbb{P}\left({P}_{j}\le \frac{\alpha {w}_{j}}{m}\right)=\frac{\alpha}{m}\sum _{j\in {\mathcal{H}}_{0}}{w}_{j}\\ \le \alpha \overline{w}=\alpha .\end{array}$$

The familywise error is

$$\begin{array}{l}\mathbb{P}((\mathcal{R}\cap {\mathcal{H}}_{0})>0)=\mathbb{P}\left({P}_{j}\le \frac{\alpha {W}_{j}}{m}\phantom{\rule{0.16667em}{0ex}}\text{for}\phantom{\rule{0.16667em}{0ex}}\text{some}\phantom{\rule{0.16667em}{0ex}}j\in {\mathcal{H}}_{0}\right)\\ \le \sum _{j\in {\mathcal{H}}_{0}}\mathbb{P}\left({P}_{j}\le \frac{\alpha {W}_{j}}{m}\right)\\ =\sum _{j\in {\mathcal{H}}_{0}}{\mathbb{E}}_{H}\left(\mathbb{P}\left({P}_{j}\le \frac{\alpha {w}_{j}}{m}|{W}_{j}={w}_{j}\right)\right)\\ =\sum _{j\in {\mathcal{H}}_{0}}{\mathbb{E}}_{H}(\alpha {W}_{j}/m)=\frac{\alpha}{m}\sum _{j\in {\mathcal{H}}_{0}}{\mathbb{E}}_{H}({W}_{j})\\ \le \frac{{m}_{0}\alpha}{m}\le \alpha .\end{array}$$

Let *C* denote the set of hypotheses with *ξ _{j} >* 0. Power is optimized if

$$\frac{1}{{m}_{1}}\sum _{j\in C}\overline{\mathrm{\Phi}}\left({\overline{\mathrm{\Phi}}}^{-1}\left(\frac{\alpha {w}_{j}}{m}\right)-{\xi}_{j}\right),$$

with constraint

$$\sum {w}_{j}=m.$$

Choose *w* to maximize

$$\pi =\frac{1}{{m}_{1}}\sum _{j\in C}\overline{\mathrm{\Phi}}\left({\overline{\mathrm{\Phi}}}^{-1}\left(\frac{\alpha {w}_{j}}{m}\right)-{\xi}_{j}\right)-\lambda (m-\sum {w}_{i})$$

by setting the derivative to zero

$$\begin{array}{l}\frac{\partial}{\partial {w}_{i}}\pi =-\lambda +\frac{\phi ({\overline{\mathrm{\Phi}}}^{-1}(\alpha {w}_{j}/m)-{\xi}_{j})}{\phi ({\overline{\mathrm{\Phi}}}^{-1}(\alpha {w}_{j}/m))}\frac{\alpha}{m}=0,\\ \frac{m\lambda}{\alpha}=\frac{\phi ({\overline{\mathrm{\Phi}}}^{-1}(\alpha {w}_{j}/m)-{\xi}_{j})}{\phi ({\overline{\mathrm{\Phi}}}^{-1}(\alpha {w}_{j}/m))}.\end{array}$$

The *w* that solves these equations is given in (3). Finally, solve for *c* such that Σ* _{i} w_{i}* =

The first statement follows easily by noting that the worst case corresponds to choosing weight *B* in the first term in *R*(*ξ*) and choosing weight *b* in the second term in *R*(*ξ*). The rest follows by Taylor expanding *R _{b,B}* (

With *b* = 0, *R _{b,B}* (

$$\overline{\mathrm{\Phi}}({z}_{B\alpha /m}-\xi )-2\overline{\mathrm{\Phi}}({z}_{\alpha /m}-\xi )\ge 0.$$

(13)

With *B* ≥ 2, (13) holds at *ξ* = 0. The left-hand side is increasing in *ξ* for *ξ* near 0, but (13) does not hold at *ξ* = *z _{α/m}*. So (13) must hold in the interval [0

Choose *K >* 1 such that 1*/*(*K* + 1) *<* 1*/a* − *ε*. Choose 1 *> γ >* (2*α* − *a*)*/K*. Choose a small *c >* 0. Let
$\xi =A+\sqrt{{A}^{2}-2c}$ and
$u=B-\sqrt{{B}^{2}-2c}$, where

$$\begin{array}{l}A={\overline{\mathrm{\Phi}}}^{-1}\left(\frac{\alpha}{(m(\gamma K+a))}\right),\\ B={\overline{\mathrm{\Phi}}}^{-1}\left(\frac{K\alpha}{(m(\gamma K+a))}\right).\end{array}$$

Then *ρ*(*ξ*) = 1*/a* and ( *ξ*) = 1*/*(*K* + 1). Now *d*(*Q, *) = *γ*. Taking *K* sufficiently large and *γ* sufficiently close to (2*α* − *a*)*/K* makes *γ< δ*.

It is convenient to prove Theorem 3.8 before proving Theorem 3.7.

Let *c*_{*} solve

$$\gamma \overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}})+a\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{{c}_{\ast}}{\xi}\right)=\frac{\alpha}{m}.$$

(14)

We claim first that, for any *c > c*_{*}, there is no *u* such that the weights average to 1. Fix *c > c*_{*}. The weights average to 1 if and only if

$$\gamma \overline{\mathrm{\Phi}}\left(\frac{c}{u}+\frac{u}{2}\right)+a\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{c}{\xi}\right)=\frac{\alpha}{m}.$$

(15)

Since *c > c*_{*} and since the second term is decreasing in *c*, we must have

$$\overline{\mathrm{\Phi}}\left(\frac{c}{u}+\frac{u}{2}\right)>\overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}}).$$

The function *r*(*u*) = (*c/u* + *u/*2) is maximized at
$u=\sqrt{2c}$. So
$r(\sqrt{2c})>r(u)$. But
$r(\sqrt{2c})=\overline{\mathrm{\Phi}}(\sqrt{2c})$. Hence,
$\overline{\mathrm{\Phi}}(\sqrt{2c})\ge r(u)\ge \overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}})$. This implies *c < c*_{*}, which is a contradiction. This establishes that sup* _{u} c*(

$$\gamma \overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}})+a\overline{\mathrm{\Phi}}\left(\frac{\xi}{2}+\frac{{c}_{\ast}}{\xi}\right)=\gamma \overline{\mathrm{\Phi}}(\sqrt{2{c}_{\ast}})+O(a).$$

Now set this expression equal to *α/m* and solve.

Define *c*_{*} as in (14). If
${u}_{\ast}=\sqrt{2{c}_{\ast}}\le \xi $, then the the proof proceeds as in the previous proof. So we first need to establish for which values of *ξ* is this true. Let
$r(c)=\gamma \overline{\mathrm{\Phi}}(\sqrt{2c})+a\overline{\mathrm{\Phi}}(\xi /2+c/\xi )$. We want to find out when the solution of *r*(*c*) = *α/m* is such that
$\sqrt{2c}\le \xi $, or, equivalently, *c* ≤ *ξ*^{2}*/*2. Now *r* is decreasing in *c*. Since *γ* + *a* ≥ *α/m*, *r*(−∞) ≥ *α/m*. Hence, there is a solution with *c* ≤ *ξ*^{2}*/*2 if and only if *r*(*ξ*^{2}*/*2) ≤ *α/m*. But *r*(*ξ*^{2}*/*2) = (*γ* + *a*) (*ξ*)*,* so we conclude that there is such a solution if and only if (*γ* + *a*) (*ξ*) ≤ *α/m*, that is, *ξ* ≥ *z _{α/}*

Now suppose that *ξ < ξ*_{0}. We need to find *u* ≤ *ξ* to make *c* as large as possible in the equation *v*(*u, c*) *γ* (*u/*2 + *c/u*) + *a* (*ξ/*2 + *c/ξ*) = *α/m*. Let *u*_{*} = *ξ* and *c*_{*} = *ξz _{α/}*

For the second claim, note that the power of the weighted test beats the power of Bonferroni if and only if the weight *w* = (*m/α*) (*ξ/*2 + *C*(*ξ*)*/*2) ≥ 1, which is equivalent to

$$C(\xi )\le \xi {z}_{\alpha /m}-{\xi}^{2}/2.$$

(16)

When *ξ* ≤ *ξ*_{0}, *C*(*ξ*) = *ξξ*_{0} − *ξ*^{2}*/*2. By assumption, *γ* + *a* ≤ 1 so that *z _{α/}*

Finally, we turn to (8). In this case,
$C(\xi )={z}_{\alpha /(m\gamma )}^{2}/2+O(a)$. The worst case power is
$\overline{\mathrm{\Phi}}(C(\xi )/\xi -\xi /2)=\overline{\mathrm{\Phi}}({z}_{\alpha /(m\gamma )}^{2}/(2\xi )-\xi /2)+O(a)$. The latter is increasing in *ξ* and so is at least
$\overline{\mathrm{\Phi}}({z}_{\alpha /(m\gamma )}^{2}/(2{\xi}_{\ast})-{\xi}_{\ast}/2)+O(a)=\overline{\mathrm{\Phi}}(({z}_{\alpha /(m\gamma )}^{2}/(2{\xi}_{\ast})-{\xi}_{2}^{\ast})/(2{\xi}_{\ast}))+O(a)$, as claimed. The next two equations follow from standard tail approximations for Gaussians. Specifically, a Gaussian quantile *z _{β/m}* can be written as
${z}_{\beta /m}=\sqrt{2log({mL}_{m}/\beta )}$, where

Setting *π*(*w, ξ _{m}*) =(

^{1}For a two-sided alternative the power is

$$\pi ({\xi}_{j},{w}_{j})=\overline{\mathrm{\Phi}}({z}_{\alpha {w}_{j}/2m}-{\xi}_{j})+\overline{\mathrm{\Phi}}({z}_{\alpha {w}_{j}/2m}+{\xi}_{j}).$$

Kathryn Roeder, Professor of Statistics, Department of Statistics, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA.

Larry Wasserman, Professor of Statistics, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA.

- Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Statist Soc Ser B. 1995;57:289–300. MR1325392.
- Benjamini Y, Hochberg Y. Multiple hypotheses testing with weights. Scand J Statist. 1997;24:407–418. MR1481424.
- Benjamini Y, Krieger AM, Yekutieli D. Adaptive linear step-up procedures that control the false discovery rate. Biometrika. 2006;93:491–507. MR2261438.
- Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann Statist. 2001;29:1165–1188. MR1869245.
- Bickel P, Klaassen C, Ritov Y, Wellner J. Technical report. Johns Hopkins Series in the Mathematical Statistics; Baltimore, Maryand: 1993. Efficient and adaptive estimation for semiparametric models. MR1245941.
- Blanchard G, Roquain E. Two simple sufficient conditions for FDR control. Electron J Stat. 2008;2:963–992. MR2448601.
- Blanchard G, Roquain E. Adaptive FDR control under independence and dependence. J Mach Learn Res. 2009 To appear.
- Chen JJ, Lin KK, Huque M, Arani RB. Weighted
*p*-value adjustments for animal carcinogenicity trend test. Biometrics. 2000;56:586–592. [PubMed] - Donoho D, Jin J. Higher criticism for detecting sparse heterogeneous mixtures. Ann Statist. 2004;32:962–994. MR2065195.
- Efron B. Simultaneous inference: When should hypothesis testing problems be combined? Ann Appl Statist. 2007;2:197–223. MR2415600.
- Efron B, Tibshirani R, Storey JD, Tusher V. Empirical Bayes analysis of a microarray experiment. J Amer Statist Assoc. 2001;96:1151–1160. MR1946571.
- Genovese C, Wasserman L. Operating characteristics and extensions of the false discovery rate procedure. J R Stat Soc Ser B Stat Methodol. 2002;64:499–517. MR1924303.
- Genovese CR, Roeder K, Wasserman L. False discovery control with
*p*-value weighting. Biometrika. 2006;93:509–524. MR2261439. - Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer; New York: 2001. MR1851606.
- Holm S. A simple sequentially rejective multiple test procedure. Scand J Statist. 1979;6:65–70. MR0538597.
- Ionita-Laza I, McQueen M, Laird N, Lange C. Genomewide weighted hypothesis testing in family-based association studies, with an application to a 100K scan. Am J Hum Genet. 2007;81:607–614. [PubMed]
- Kropf S, Läuter J, Eszlinger M, Krohn K, Paschke R. Nonparametric multiple test procedures with data-driven order of hypotheses and with weighted hypotheses. J Statist Plann Inference. 2004;125:31–47. MR2086887.
- McQueen M, et al. Personal communication. 2008.
- Roeder K, Bacanu S-A, Wasserman L, Devlin B. Using linkage genome scans to improve power of association in genome scans. Am J Hum Genet. 2006;78:243–252. [PubMed]
- Roeder K, Wasserman L, Devlin B. Improving power in genome-wide association studies: Weights tip the scale. Genet Epidemiol. 2007;31:741–747. [PubMed]
- Romano JP, Shaikh AM, Wolf M. Control of the false discovery rate under dependence using the bootstrap and subsampling. TEST. 2008;17:417–442. MR2470085.
- Roquain E, van de Wiel M. Multi-weighting for FDR control. 2008. Available at arXiv:0807.4081.
- Rosenthal R, Rubin D. Ensemble-adjusted
*p*-values. Psychol Bull. 1983;94:540–541. - Rubin D, Dudoit S, van der Laan M. A method to increase the power of multiple testing procedures through sample splitting. Stat Appl Genet Mol Biol. 2006;5 Art. 19 (electronic). MR2240850. [PubMed]
- Sabatti C, Service S, Freimer N. False discovery rate in linkage and association genome screens for complex disorders. Genetics. 2003;164:829–833. [PubMed]
- Saccone S, Hinrichs AL, Saccone N, Chase G, Konvicka K, Madden P, Breslau N, Johnson E, Hatsukami D, Pomerleau O, Swan G, Goate A, Rutter J, Bertelsen S, Fox L, Fugman D, Martin N, Montgomery G, Wang J, Ballinger D, Rice J, Bierut L. Cholinergic nicotinic receptor genes implicated in a nicotine dependence association study targeting 348 candidate genes with 3713 SNPs. Hum Mol Genet. 2007;16:36–49. [PMC free article] [PubMed]
- Sarkar S, Heller R. Comments on: Control of the false discovery rate under dependence using the bootstrap and subsampling. TEST. 2008;17:450–455. MR2470085.
- Sarkar SK. Some results on false discovery rate in stepwise multiple testing procedures. Ann Statist. 2002;30:239–257. MR1892663.
- Satagopan J, Elston R. Optimal two-stage genotyping in population-based association studies. Genet Epidemiol. 2003;25:149–157. [PubMed]
- Schuster E, Kropf S, Roeder I. Micro array based gene expression analysis using parametric multivariate tests per gene—a generalized application of multiple procedures with data-driven order of hypotheses. Biom J. 2004;46:687–698. MR2108612.
- Signoravitch J. Technical report. Harvard Biostatistics; 2006. Optimal multiple testing under the general linear model.
- Skol A, Scott L, Abecasis G, Boehnke M. Joint analysis is more efficient than replication-based analysis for two-stage genome-wide association studies. Nat Genet. 2006;38:390–394. [PubMed]
- Spjøtvoll E. On the optimality of some multiple comparison procedures. Ann Math Statist. 1972;43:398–411. MR0301871.
- Storey JD. A direct approach to false discovery rates. J R Stat Soc Ser B Stat Methodol. 2002;64:479–498. MR1924302.
- Storey JD. The optimal discovery procedure: A new approach to simultaneous significance testing. J R Stat Soc Ser B Stat Methodol. 2007;69:347–368. MR2323757.
- Storey J, Tibshirani R. Statistical significance for genome-wide studies. Proc Natl Acad Sci USA. 2003;100:9440–9445. MR1994856. [PubMed]
- Sun W, Cai TT. Oracle and adaptive compound decision rules for false discovery rate control. J Amer Statist Assoc. 2007;102:901–912. MR2411657.
- Todd J, Walker N, Cooper J, Smyth D, Downes K, Plagnol V, Bailey R, Nejentsev S, Field S, Payne F, Lowe C, Szeszko J, Hafler J, Zeitels L, Yang J, Vella A, Nutland S, Stevens H, Schuilenburg H, Coleman, Maisuria M, Meadows W, Smink LJ, Healy B, Burren O, Lam A, Ovington N, Allen J, Adlem E, Leung H, Wallace C, Howson J, Guja C, Ionescu-Tirgovi C, Simmonds M, Heward J, Gough S, Dunger D, Wicker L, Clayton D. Genetics of Type 1 Diabetes in Finland. Wellcome Trust Case Control Consortium. Robust associations of four new chromosome regions from genome-wide analyses of type 1 diabetes. Nat Genet. 2007;39:857–864. [PMC free article] [PubMed]
- Wang H, Thomas D, Pe’er I, Stram D. Optimal two-stage genotyping designs for genome-wide association scans. Genet Epidemiol. 2006;30:356–368. [PubMed]
- Wang K, Li M, Bucan M. Pathway-based approaches for analysis of genomewide association studies. Am J Hum Genet. 2007;81:1278–1283. [PubMed]
- Westfall P, Krishen A, Young S. Using prior information to allocate significance levels for multiple endpoints. Stat Med. 1998;17:2107–2119. [PubMed]
- Westfall PH, Krishen A. Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. J Statist Plann Inference. 2001;99:25–40. MR1858708.
- Westfall PH, Kropf S, Finos L. IMS Lecture Notes Monogr Ser. Vol. 47. IMS; Beachwood, OH: 2004. Weighted FWE-controlling methods in high-dimensional situations. Recent Developments in Multiple Comparison Procedures; pp. 143–154. MR2118598.
- Westfall PH, Soper KA. Using priors to improve multiple animal carcinogenicity tests. J Amer Statist Assoc. 2001;96:827–834. MR1963409.

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |