The simplest approach to gene set testing relies on 2 × 2 tables of gene set membership (in gene set or not) vs. significance (significant or not). Gene set testing is often performed using a χ² or Fisher's Exact test [5], which rely critically upon the assumption that individual genes, and their associated test statistics, are independent, hereafter referred to as the independence assumption [6]. Many software tools use independence assumption methods (reviewed in [7]) and these tools are employed by the majority of publications using gene set testing (Figure ​(Figure1a).1a). However, independence of genes is a false assumption for expression microarrays: indeed, correlation forms the basis for informative techniques such as hierarchical clustering [8,9]. Several authors have noted, via simulation or analysis of a few experimental data sets, that correlation adversely affects the selection of significant gene sets [6,10-13]. Investigators who have recognized this problem employed resampling-based approaches that maintain the correlation structure of the expression data and produce an empirical estimate of gene set significance [14,15]. Resampling approaches have also been integrated into many software tools to address this problem [2,10,16-19]. Nevertheless, the use of independence assumption-based methods continues to increase (Figure ​(Figure1b1b).

As the independence assumption is clearly violated for gene expression microarrays [11,20,21], how can we explain the persistent use of independence methods? Independence assumption methods may appeal to the belief that the occurrence of multiple members of a gene set on a list of significant ones "cannot be due to chance." Indeed, in a post-hoc interpretation, the fact that the genes within a set are correlated is often, incorrectly, viewed as reinforcing the biological plausibility [5,22], despite evidence presented here and elsewhere that the presence of correlation increases false-positive rates. It is conceivable to argue that correlation within a gene set itself provides additional information, if such correlation varied substantially across different experiments. However, as we show here, correlation within a gene set is largely a persistent property that is preserved across a wide variety of sample sources and experimental conditions.

One simple way to describe the effect of correlation is through a standardized enrichment test statistic. Let z be the signed square root of the independence assumption χ² statistic (see Methods). Under the null hypothesis (no enrichment), z should have approximately mean 0 and variance 1. However, the true variance can be much greater, with an increase proportional to (m - 1)ρ (see Methods), where m is the number of genes in the set, and ρ is the average correlation among genes in the set (excluding self-correlations). Independence assumption methods assume ρ = 0, implying that the variance equals 1. However, if the average internal correlation of a gene set is positive, then the variance of z can be greatly inflated, even if this correlation is modest, depending on the gene set size m.

To empirically demonstrate that variance inflation results in the false significance of gene sets, we randomly permuted the sample labels associated with each of the 202 Gene Expression Omnibus data sets 10,000 times, and then performed a gene set analysis on each permutation using a common independence assumption method. For each permutation a list of "significant" genes with p ≤ 0.05 (using a t-test or F-test, as appropriate) was created in order to have a meaningfully-sized gene list, and the standard χ² statistic was used to assess the significance of each gene set. A gene set was called significant if it had a Bonferroni-corrected (for the number of sets) χ² p-value ≤ 0.05. Note that this is a highly conservative threshold, intended to guarantee that no more than 5% of the permutations result in one or more gene sets (falsely) called significant. The gene set null hypothesis is that each gene set has the same pattern and proportion of differentially expressed genes as any other set [24]. Permutation of the sample labels induces the null hypothesis for each gene, and thus no gene sets should exhibit enrichment for differential expression. Hence, any gene set declared significant is a false positive.

To determine the overall experiment-wise increase in false positives, we counted the number of permutations in which at least one gene set was declared significant, using the Bonferroni-corrected 0.05 threshold. If the independence assumptions were true, no more than 5% of the permutations would give rise to significant gene sets. However, the observed proportion is much higher (Figure ​(Figure3;3; Additional file 1, Figure S1). False positive rates often exceed 50%, and in some cases are above 80%, and are even higher when the False Discovery Rate (FDR) is used as a multiple comparison correction. Clearly such high false positive rates have serious implications for the conclusions drawn by any study which uses independence assumption methods to provide biological interpretation of microarray data.

In addition, as predicted, we found that variance inflation is directly related to the false-positive rate for a gene set. For each GO Biological Process (Figure ​(Figure4)4) or KEGG pathway (Additional file 1, Figure S2), we show that the empirical variances of the standardized enrichment statistic (which often greatly exceed 1) are highly correlated with the per-category false positive rates. Specifically, for each gene set, we calculated the variance of the square root of the χ² statistic across all permutations, and plotted it against the number of permutations in which that gene set was called "significant" using a Bonferroni correction.

Fortunately, the shortcomings of independence assumption methods can be addressed through the use of resampling-based methods. These methods use the resampling data to construct an empirical null distribution for the gene set test statistics, taking into account the correlation structure between genes and providing a more accurate assessment of statistical significance. Several existing tools use resampling based approaches [2,16-19], either permuting sample labels relative to gene expression, or performing bootstrap resampling [24]. We empirically investigated the performance of the permutation approach, which we will refer to as "resampling" to avoid confusion with the permutations described above. The sample labels for each data set were resampled without replacement 10,000 times, and the resulting enrichment statistic distribution was used to obtain an empirical p-value for each gene set. For example, if the observed enrichment statistic for a gene set was greater than or equal to 99% of the resampled enrichment statistics, a p-value of 0.01 was assigned to that gene set. As the resampling units are entire arrays, correlation between genes is maintained under resampling. Therefore the effects of correlation within gene sets apply to both the resampled and observed data, properly controlling the false positive rate. To demonstrate that the resampling approach correctly controls the false positive rate, we applied the resampling procedure to all permutations of all 202 datasets, using the same χ² statistic as above. Using the empirical distribution of test statistics under resampling to generate the gene set p-value, proper control of the false positive rate was obtained (Figure ​(Figure3;3; Additional file 1, Figure S1, right-most boxplots).

While the analyses above clearly show that gene correlation increases the false positive rate for independence assumption methods, it may be tempting to argue that correlation does not affect false positive rates in real datasets, which presumably include true enrichment. To investigate this, we computed the proportion of observed datasets in which each gene set was declared significant by the independence assumption method, and compared these values to the previously generated proportions observed under random permutation. The results for GO Biological Process categories (Figure ​(Figure5)5) and KEGG pathways (Additional file 1, Figure S3) demonstrate a strong relationship. In other words, categories declared statistically significant by independence assumption methods in actual datasets are often declared significant when treatment assignments are made purely at random. This strongly calls into question the biological conclusions drawn from such analyses, as we have demonstrated that such correlation is a predominant source of misinterpretation, rather than a biological confirmation. Indeed, some gene sets have such high false positive rates (Figures ​(Figures33 and ​and5)5) under independence assumption that one or more sets may be found significant in multiple studies, further enhancing apparent biological confirmation.

Gene set testing, in which enrichment of significant differentially expressed genes is sought among gene sets, is by now a standard method to provide biological interpretation for gene expression data. Groups of genes with a common biological function, cellular localization, regulation, or chromosomal location may hold additional clues regarding underlying biology, or potentially improve prediction or classification. While many easy-to-use tools have been developed to facilitate pathway- or gene set-based analysis, inter-gene correlation within gene sets violates the independence assumption that underlies many gene set analysis methods. Indeed, our meta-analysis demonstrates that the correlation patterns persist across a wide variety of mouse and human experiments. Thus, we argue that correlation patterns should largely be viewed in terms of their effects on false positives, as we find no evidence that correlation within the gene set confers additional plausibility to a gene set finding.

Though it may be viewed as a technical matter, violation of the independence assumption is not a mere statistical detail. It is a tangible phenomenon that increases the chance of falsely declaring a gene set significant. The false positive rates established by our study are very high - sometimes an order of magnitude beyond the intended false positive rate. Array resampling methods correctly handle inter-set correlation and are not subject to these high false positive rates. Barry et al. [24] describe investigations of power and provides context on the meaning of type I error control for resampling-based methods. The null distribution induced by permutation here is a special case of the expanded definition in [24], because the null hypothesis holds for each gene. Bootstrap resampling methods were shown in [24] to have more power than array permutation methods in simulated and real data sets, and expansions beyond the existing resampling approaches are worthy of investigation. There are a wide number of available packages that implement these methods, such as GSEA [2], Catmap [16], SAFE [17], ErmineJ [10], and SAM-GS [18].

Several methods have been proposed which use multivariate modeling of array data to perform gene set testing [19,30,31]. Although comparisons using such approaches were not a main focus, we briefly tested whether these methods correctly control the false positive rate, by permuting the sample labels for one data set (GDS266) 1,000 times and counting the number of permutations in which at least one GO BP category was called significant using a 5% FDR. The globaltest procedure [19] uses a score-based test statistic to test for correlation of an entire set of genes with clinical outcome, and can be used for gene set testing. The parametric version of globaltest incorporates the observed correlation structure, and is compared to a χ² distribution. However, we found that 94% of permutations called at least one GO BP category significant, with a median of 10 categories called significant in each permutation, suggesting difficulties in proper control of the false positive rate. It is worth noting that use of empirical covariance estimates is extremely challenging, especially when gene sets are larger than the sample size. GlobalANCOVA [30] fits a linear model between each gene and the experimental covariates, and uses permutation testing to obtain empirical p-values. We found that GlobalANCOVA found no significant GO BP categories under permutation, suggesting that GlobalANCOVA properly controls the false positive rate, and is perhaps conservative. Several related methods have been proposed [31,32] that test for differential expression or predictive ability of sets of genes. Many of these approaches attempt to incorporate gene-gene correlation structure parametrically, and are thus computationally appealing. However, until more comprehensive investigations have been performed for the multivariate procedures, it seems prudent to examine the false positive rate in 1,000 permutations of their data before accepting the results at face value. Other investigators have explored more complex procedures involving stochastic dependence [33], while deeper explorations of random-set methods [34] may be valuable, but still essentially rely on independence assumptions. Several other methods are reviewed and compared in [35-37]. In summary, despite the availability of software and a wide array of methods that address inter-gene correlation, the literature on microarray analysis is rife with studies that use independence assumption methods.

As the use of independence assumption methods in gene array studies is widespread, we conclude that published false gene set findings may be common. The precise impact of the overly optimistic statistical support for many gene set findings is difficult to assess, because we do not know the underlying truth; however, a basic requirement of a valid statistical test is that it controls false positives under the null hypothesis, and we have clearly demonstrated here that methods that rely on independence assumptions are therefore invalid in this sense.