Our evaluation was designed in 2004 and data collection and analyses were conducted in 2004–2007. For the main analysis, we drew a five percent simple random sample (SRS) of articles that were returned by the query described above, published from 2001 to 2003, and curated in HuGE Pub Lit before May 30th, 2004 (n = 8,115) to yield a dataset of 406 articles. To provide an updated description of reporting practices and to assess improvements in reporting, we randomly selected (SRS) 40 articles that were published during 2006 from articles that were returned by our database query, added to PubMed in 2006, and curated in HuGE Pub Lit before May 18, 2007 (n = 5,353). After each article was read, 91 from 2001–2003 and 12 from 2006 were excluded from the analysis for the following reasons: not written in English (2001–2003: n = 28, 2006: n = 6), population screening studies (2001–2003: n = 23, 2006: n = 0), clinical trials or pharmacogenomic studies (2001–2003: n = 16, 2006: n = 3), not full-length articles (i.e., letter or abstract) (2001–2003: n = 11, 2006: n = 0), failed to fulfill the inclusion criteria for HuGE Pub Lit [7] on closer scrutiny (2001–2003: n = 6, 2006: n = 1), family studies (2001–2003: n = 3, 2006: n = 2), studies of genetic tests (2001–2003: n = 2, 2006: n = 0), or meta-analyses (2001–2003: n = 2, 2006: n = 0).

Data were abstracted from each original publication in duplicate by two independent data extractors. All discrepancies between the independent extractors were discussed and a consensus was reached.

For the 2001–2003 articles, a standardized abstraction form was developed and piloted for 10 articles; the form was revised according to the results of this pilot study to ensure that the definitions for the collected items were clear and unambiguous. Items on this final form were designed to collect information on the reporting of study design, genotyping method, population stratification, analytical methods (including the analysis of multiple genetic variants and gene-environment interactions), and study inferences. In addition, the final form accommodated different observational study designs, multiple groups of study participants, and the consideration of more than one postulated genetic risk factor as well as additional environmental factors. Articles were coded as potentially misclassifying the disease or environmental exposure status of study participants when the article did not explicitly state that these factors were directly measured for all participants in the study population. When multiple groups of study participants were reported in a study, we recorded the sample size of the largest group for cohort and cross-sectional studies and the largest case and control groups for case-control studies. Items were collected separately for case and control groups for case-control studies and for all study participants regardless of disease status for cohort and cross-sectional studies. For the purpose of this analysis, data collected for case and control groups were combined so that statistics could be calculated for all study participants. Information (e.g., mean or median age and sex distribution) was considered as given for all study participants only if it was provided for all case and control groups. Additionally, for case-control studies, we recorded whether cases and controls were described as drawn from the same population according to one or more of the following definitions: 1) geographic region, 2) clinical population, 3) general population (i.e., ethnic group), or whether information on the choice of suitable controls was missing or incomplete.

Fourteen items were assessed in the HuGE articles published in 2006. These included the number of study participants, genes, polymorphisms, and environmental factors assessed in gene-environment interactions. In addition, we selected ten items that were applicable to all study designs and that had been reported in fewer than 50% of the articles published in 2001–2003.

Representative of the literature in this field, most of the studies in our samples were small: only about 10 percent of the studies reported sample sizes that exceeded 1,000. Several meta-analyses have found significant differences between the results of small and large genetic association studies [6,19]. Growing evidence suggests that individual genetic variants impart only a modest effect on the risk of developing complex, multifactorial diseases [20-22]. Thus, enrolling many thousands or even tens of thousands of individuals may be required to achieve the necessary power to identify and validate true genetic associations [13,20,23,24].

Our ability to assess the potential for selection biases was severely limited in many of the studies we examined. Although most studies provided some qualitative descriptions of the study participants (such as origin and enrollment criteria), reporting was sporadic for even simple descriptors, such as age and sex. Potentially important details, such as the number of exclusions or the number of samples that could not be genotyped, were often omitted.

Misclassification can severely limit study power and bias the results [14,23,25-27]. We determined about a tenth of studies may have misclassified their phenotypic outcomes and three-fourths of studies may have misclassified their environmental factors. A small proportion of studies reported measures such as genotyping replicate samples and blinding the research staff [27,28] to help ensure that the genetic data were not misclassified. Although the practice of detecting genotyping errors through tests of Hardy-Weinberg Equilibrium is still being debated [27,29-31], approximately half of studies reported HWE test results.

Population stratification may occur when study participants are selected from subpopulations with a different prevalence of the phenotypes and genotypes [21,30,32,33]. Although the extent to which population stratification contributes to spurious findings remains debatable [34-36], most of the articles in our sample provided descriptions for the ethnic origin of the study participants, and almost all case-control studies indicated that the cases and controls were drawn from the same population. A few studies reported the use of unlinked markers to provide evidence that population stratification was not an issue in the analysis. Genome-wide association studies provide considerable genetic data to examine and correct for population stratification (e.g. by principal component analysis) [37-39].

As a result of the selective reporting of significant results from multiple analyses and publication bias, the extent of type I error in the published literature may be great [11,14,21,23,26,28,30,40,41]. Although most studies reported results for only a few polymorphisms and environmental factors, it is difficult to determine the number actually tested; only a minority of articles reported using corrections for multiple comparisons, even for the reported associations. Reporting justifications for specific genetic comparisons could suggest that studies were founded on an a priori hypothesis and were not the result of selective reporting. However, less than one-half of the studies that assessed dominant or recessive genetic models provided justifications for the use of these specific comparisons and not others. Among articles that described gene-gene or gene-environment interactions, a substantial proportion reported statistically significant results. However, many of these may be spurious, given the limited power of most studies to identify true associations, let alone interaction effects [20,32]. The high frequency of "positive" results in our study sample could reflect a combination of multiple testing, selective reporting, and publication bias [14].

The use of common reporting standards could increase the transparency of research methodology, thus helping to identify selective reporting and sources of bias and confounding while allowing for a more complete synthesis of data across consortia or in meta-analyses [1,10,42]. The results of our study were presented at a HuGENet sponsored workshop, Strengthening the Reporting of Genetic Associations (STREGA) [43]. The workshop concluded with an agreement to develop an extension of the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) statement [44] to address some of the specific challenges (e.g., generation of genetic data, population stratification, haplotype inference, HWE, and multiple testing) posed by reporting results of genetic epidemiology studies. STREGA is currently finalized for publication, as of writing this article.