Carefully chosen weights can, in general, improve power of combination methods. A motivation for the weighting may follow from the fact that different studies might be differently powered, and that should be reflected by the weighting. Consider the combined P-value of the weighted Z-test: where Z_i = Φ⁻¹(1 −p_i); p_i is a P -value for the i-th study of k studies in total, w_i are weights, and Φ, Φ⁻¹ denote the standard normal cumulative distribution function and its inverse. Lipták suggested that the weights in this method “should be chosen proportional to the ‘expected’ difference between the null hypothesis and the real situation and inversely proportional to the standard deviation of the statistic used in the i-th experiment” and further suggested that when nothing else is available but the sample sizes of the studies (n_i), then the square root of n_i can be used as a weight (Lipták, 1958). Won et al. verified Lipták’s claim more formally by showing that his test has optimal power when weights are set to the expected difference (i.e. the effect size) over the known or the estimated standard error (Won et al., 2009). This method of weighting requires knowledge of anticipated effect sizes for all combined studies, which is rarely available. Weightings by the estimated standard error or by the square root of sample size are more feasible in practice.

When different samples are taken from similar populations, a model that assumes a common effect size and direction among samples is appropriate. The ideal approach in this case is to pool raw data from all samples and to conduct a single statistical test. Whitlock considered such a test with its P-value and evaluated how well a combined P-value approximates this “true” P-value (Whitlock, 2005). He evaluated Fisher’s method for combining P-values (Fisher, 1932) as well as the unweighted and weighted Z-tests, using one-sided P-values. Indeed, Whitlock found via simulation experiments that a weighted version of the combination Z-test outperformed both Fisher’s and Stouffer’s methods. Nevertheless, weighted versions of Fisher’s method exist and it had remained unclear whether the power of a weighted version of Fisher’s method may be as powerful as that of the weighted Z-test. This issue was recently taken on by Chen who found that Lancaster’s generalization of Fisher’s test was more powerful than the weighted Z-test (Chen, 2011). In Chen’s application, P-values were transformed to chi-square variables by an inverse chi-square transformation with the degrees of freedom equal to the sample size of the study, i.e. Lancaster’s statistic is with the distribution .

Both Whitlock and Chen used non-optimal weights for the weighted Z-test, setting them to the sample sizes of the studies. The original Whitlock’s conclusions are valid, but the weights need to be adjusted according to suggestions by Lipták and Won et al. In Whitlock’s setup, samples that corresponded to different studies were drawn from the same population. In this setup, the T-test based on pooled raw data can be viewed as an “ideal” test. In this case, optimal weights for the weighted Z method are given by the square root of the sample sizes, . These weights are optimal in the sense that the combined P -value approximates the P value of the test based on raw data. This can be seen from writing out a Z statistic based on pooled raw data in terms of statistics for the individual studies. The pooled data statistic is , where is the sample average for the total sample of size n_T and Ŝ_T is the sample standard deviation. Suppose that we split the sample into two parts of sizes n_X, n_Y and calculate sample means (,) and standard deviations (Ŝ_X, Ŝ_Y) separately for these two samples. We can write the pooled statistic in terms of the two means as

while the weighted statistic that combines information from the two samples is

The pieces and can be recovered from P-values for the two samples by the inverse normal transformation. This statistic is the weighted Z-test for combining P-values. We can see that Z_w approximates Z_total when the weights w_X, w_Y are set to . The same argument holds for more than two samples. Regarding Lancaster’s method, Chen noted cautiously that setting degrees of freedom to the sample size of the i-th study “may not be optimal”. It is an optimal weighting however for his simulation setup, where samples are obtained from the same population. This follows from the fact that the chi-square distribution for the i-th statistic approaches a normal distribution with the variance 2n_i: when the degrees of freedom are set to n_i, the variance of the i-th statistic in Lancaster’s method is proportional to the variance of the corresponding term for the optimally weighted Z-test. Thus, power advantage of Lancaster’s method over the weighted Z method observed by Chen was at least to some degree due to the usage of non-optimal weights for the Z method. As I will verify by simulation experiments, power of the optimally weighted Z method at conventional 1% and 5% levels is very similar to that of Lancaster’s method.

In this study, one-sided P-values were assumed. Such P-values are appropriate for meta-analytic combination of P-values from several studies. Two-sided P-values are generally inappropriate, because they are oblivious to the effect direction. Two-sided P-values from two studies in which the effect direction is flipped can both be small nevertheless, resulting in an inappropriately small combined P-value. On the other hand, combined result of corresponding one-sided P-values will properly reflect cancellation of the pooled effect that would have been observed if raw data from the two studies were combined.

Despite the fact that the mechanics of the meta-analytic process involves manipulation of one-sided P-values, it is often the case that the final result needs to be a two-sided P-value. For example, when allele frequencies are compared between two groups of individuals classified based on the presence or absence of a trait, the null hypothesis is usually that the frequency is the same, and the alternative hypothesis does not specify a particular effect direction. The weighted Z-test provides an important advantage in dealing with this situation, due to symmetry of the normal transformation. There are two possible one-sided combined P-values for each assumed effect direction, but with the weighted Z method, the combined P-value for the first assumed direction is the same distance from ¹/₂ as the combined P-value for the second assumed direction. Therefore, one can arbitrarily assume either one of the two directions when computing one-sided P-values, and obtain a combined one-sided P-value, p_one-sided. The two-sided combined P -value is the same regardless of the assumed direction:

What if available individual P-values are all two-sided? Often, studies report P-values that correspond to statistics such as |T| and |Z|, or its squared value, i.e. the one degree of freedom chi-square. These individual P-values can be converted to one-sided before combining as follows:

Once again, the assumed effect direction can be chosen arbitrarily. For example, in testing for association of an allele with a trait at a biallelic locus A/a, we can arbitrarily choose one of the alleles, e.g. allele A. Then the “effect direction” for i-th study is positive if there is positive correlation of that allele with the presence of the trait in that study. Once these one-sided P-values are combined, the result can be converted back to two-sided by Equation (3).

Another advantage of the weighted Z test is that it can be easily extended to account for the case of correlated statistics between studies. For the test to be valid under independence, we need an assumption that the set of {Z_i} jointly follows a multivariate normal distribution under the null hypothesis. If cor(Z_i, Z_j) = r_ij, the modification amounts to replacing the denominator in Equation (1) with . The multivariate normal assumption is often justified asymptotically, and in certain situations the correlations {r_ij} are known. For example, when each Z_i is a result of comparing group i of sample size n_i to a common “control” group of sample size n₀ by the two-sample T -test, then (Dunnett, 1955). In principle, a variation of the weighted Fisher’s method can be extended to this situation, if we can assume that chi-square statistics formed from individual P-values can be represented by squares of underlying multivariate normal variables with correlations r_ij. However, required computations are more involved. First, the two degree of freedom chi-square transformation in Equation (2) would have to be replaced with the one degree of freedom transformation. Then one would need to compute eigenvalues of , where w is the vector of weights and R R is the matrix of squared correlations. Finally, to compute the combined P-value, one can use the fact that the weighted sum of these correlated chi-squares can be represented by the sum of independent weighted chi-squares with weights given by the above eigenvalues (Box, 1954). Thus, one can use the observed weighted sum of correlated chi-squares with weights substituted by the eigenvalues as an input to a routine for computing the cumulative distribution of the sum of independent weighted chi-squares.

Although there is no single method for combining P-values that is most powerful in all situations, a meta analytic setup considered by Whitlock and extended here to include study heterogeneity is quite general, because many forms of one-sided statistics approach a normal distribution asymptotically. Therefore, the – or –weighted Z-test for combining one-sided P-values can be recommended in most situations.