The arrival of massive amounts of data from genome-wide association studies (GWAS) has turned up the heat on a similar debate in the field of genetic epidemiology. On the one hand, the admittedly simple approach of averaging over genetic and environmental backgrounds and testing each marker in a GWAS for association with the studied trait marginally has proven quite successful,^2,3 despite early concerns that by ignoring the underlying complexity this naïve approach would fail.^4,5 On the other hand, the loci discovered to date do not come close to explaining the observed heritability for most studied traits.⁶ “Pathway analyses” acknowledge complexity by considering multiple loci simultaneously and relating them to known functional annotations. In principle, pathway analyses could lead to new discoveries missed by the simple marginal analyses.^7–11 Moreover, successful identification of associated pathways can clarify disease pathogenesis; indeed, for some phenotypes, multiple loci identified through GWAS have been linked to common pathways.^12–15

However, in practice, multilocus pathway analyses present a number of challenges, some of which should be very familiar to epidemiologists.^16,17 The paper by Breitling et al¹⁸ in this issue discusses the issue of overfitting when there is a large number of potential predictor variables. Other issues may be less familiar—the consequences of the peculiarities of genomic data and how it is annotated. Here we discuss 3 loosely defined approaches to “pathway analysis” and touch on potential pitfalls for each.

As Breitling et al point out, extreme care must be taken with data-mining techniques that search over a large set of possible models, when applied to data sets with a large number of markers. (The “axon guidance pathway” contained between 1,195 and 1,460 markers, depending on the data set analyzed.) Failing to appropriately correct for the model selection procedure can lead to drastic, downwardly biased P-values and overestimates of the precision of predicted trait values based on the fit model in a new data set. Breitling et al use a permutation procedure to correct for this overfitting, and report a P-value for association that is 35 orders of magnitude larger than that presented by Lesnick et al in their abstract.²⁴ We note that in the presence of population stratification or differential genotyping errors, a simple permutation procedure that randomizes phenotypes against genotypes may still yield downwardly biased P-values. Permutations should be done within strata assumed to be homogeneous with respect to ancestry or genotyping errors, if possible.

An alternative approach to assess statistical significance is to compare the observed test statistic for the pathway to the distribution of the test statistic across multiple synthetic “pathways”: sets of genes randomly drawn from the genome (and assumed not to be associated with the studied trait). Although this procedure might account for population stratification bias or genotyping errors (assuming these affect the tested pathway and the rest of the genome similarly), it comes with its own set of potential biases. In particular, this approach could be confounded by differences in characteristics between the tested and synthetic pathways, such as differences in gene size, linkage disequilibrium, etc. Ideally, the synthetic pathways should be matched to the tested pathway on these characteristics. Lesnick et al implemented this approach by randomly selecting individual SNPs from the genome and found that the observed test statistic for the axon guidance pathway was far larger than for any of the 4000 random marker sets simulated. However, by drawing markers randomly from their genome-wide data set, the authors ensured that the linkage disequilibrium patterns in the axon guidance pathway and the synthetic pathways differed greatly.

Data-mining techniques typically require no missing data on predictors or outcome variables. Lesnick et al implemented a form of complete-case analysis, restricting the models tested at each stage of their stepwise procedure to those with less than a certain amount of missing data. Breitling et al had difficulty replicating Lesnick et al’s procedure, and noted that small differences in tuning parameters could lead to quite different results, in both significance of the overall test for association and the markers selected in the final model. This is a disconcerting property for any analysis, and suggests that its conclusions may not be reliable. As data sharing becomes more widespread in the genetic epidemiology community,^25,26 other investigators will be able to replicate published analyses and assess their sensitivity to different modeling assumptions, as Breitling et al have done. This should lead to more robust scientific conclusions in the long run. We are happy to note that the GWAS data used in Lesnick et al are now available to qualified researchers through dbGAP (Study Accession: phs000048.v1.p1).

First, in the context of genome-wide association studies, the relatively small number of common marker alleles truly associated with any given trait (perhaps several hundred at the most, of hundreds of thousands tested), and the small effect for the majority of these markers, require large sample sizes to differentiate the test-statistic distribution for associated markers from that for null markers.

Second, there are many different ways to group genes according to function, and these are of varying relevance and quality. Some metabolic pathways and cellular processes are well studied, leading to a bias in genome annotation. A gene known to be involved in apoptosis, say, is likely to be involved in other (currently unknown) processes and pathways as well. Annotations may also be mistaken. The popular Gene Ontolology (GO) classifications have an associated evidence code: less than 1% of GO annotations have been confirmed experimentally.³⁸ When scanning over many gene groups for evidence of association, interpretation of results should ideally factor in prior beliefs regarding the spectrum of causal variants (limited in number and effect by observed heritability). Other important considerations are the informativeness of different groupings (some are too broad or too narrow to be of much use), and the plausibility that different functional groupings might contain multiple loci associated with the trait under study.³⁹