The interaction between Neanderthals and early modern humans has been a long-standing question in human evolutionary studies. Specimens with morphological traits typical of Neanderthals have been found across Eurasia, from southwest Spain in Europe to southern Siberia in Asia. The first appearance of these traits is as early as 400 thousand years ago (kya), and they persist until about 30 kya (Krause et al. 2007
). Some paleoanthropologists have argued for interbreeding between Neanderthals and early modern humans, using finds such as the child found in Lagar Velho, Portugal, that shows a mixture of Neanderthal and early modern human skeletal characteristics (Duarte et al. 1999
). Critics, however, have been skeptical that these finds really suggest potential admixture (Tattersall and Schwartz 1999
). Genetic evidence using Neanderthal mitochondrial DNA has consistently shown Neanderthals falling outside the range of modern human variation (Krings et al. 1997
; Serre et al. 2004
; Green et al. 2008
). This result was taken by many to mean no interbreeding occurred between Neanderthals and early modern humans, although Nordborg (1998)
showed that low levels of admixture could not be excluded by the mitochondrial DNA data.
Green et al. (2010)
sequenced the first draft of the Neanderthal genome and presented evidence from genomic data that present-day non-African human populations share more genetic variants with Neanderthals than did modern African human populations represented by Yorubans. Part of their evidence was based on a four-taxon statistic, called the D
statistic (Green et al. 2010
; Reich et al. 2010
; Durand et al. 2011
). The D
statistic quantifies the excess sharing of derived sites between the Neanderthal and any two modern human populations. A nonzero value of D
indicates that one of the modern human populations is more similar to the Neanderthal than is the other. Green et al. found that D
statistics indicated greater similarity between Neanderthals and non-African populations than between Neanderthals and African populations.
Green et al. (2010)
proposed a model in which 1–4% of non-African genomes result from admixture from Neanderthals into the ancestors of non-African populations after the separation of Africans from non-Africans. These results imply that Neanderthals and early modern humans did interbreed. However, recent admixture is not the only hypothesis consistent with the observations. Substructure in early hominin populations in Africa could produce the same patterns (Slatkin and Pollack 2008
; Durand et al. 2011
The ancient substructure in Africa model posits that there were two or more subpopulations of hominins in Africa with limited gene flow. Then, ancestors of Neanderthals emigrated from the same subpopulation from which the ancestors of present-day non-Africans later emigrated. As a consequence, non-Africans would be slightly more genetically similar to Neanderthals than would Africans because of their more recent common ancestry. After the ancestors of Neanderthals emigrated, the gene flow between the ancestors of present-day Africans and non-Africans would be sufficiently high until the out-of-Africa event, thus making the Africans and non-Africans more genetically similar to one another than either is to Neanderthals. In this model, no later interbreeding between Neanderthals and early modern humans occurred. Durand et al. (2011)
showed that both models could account for the greater similarity of non-Africans than Africans to Neanderthals.
To distinguish between these two models, we develop here a new approach that relies on the site frequency spectrum (sfs
). Durand et al. (2011)
suggested that the ancient structure model results in more variation in gene tree depth than the recent admixture model. Greater variance in tree depth would alter the frequency spectrum but not the D
statistic. Here, we show that the sfs
appropriately conditioned can distinguish between recent admixture and ancient structure because it is particularly sensitive to episodes of recent admixture. We construct the sfs
for non-Africans, conditioning on sites that have the derived allele in the Neanderthal draft genome and the ancestral allele in one randomly sampled African chromosome. This doubly conditioned frequency spectrum (dcfs
) is enriched for sites in non-African sequences that are Neanderthal specific. Similarly to the D
statistic, the sites explored are shared derived sites between Neanderthals and non-African humans and are likely to be informative about a recent admixture event.
We derive the analytical expression of the dcfs in non-Africans for a null model with no gene flow and compare a series of simulated dcfs for demographic models of both recent admixture and ancient structure. In the simulations, we allow for a variety of demographic histories, including bottlenecks in population size, ongoing gene flow between present-day human populations, population growth in early humans, varying admixture rates, and different rates of ancient gene flow. The shape of the dcfs are observed for each parameter set and compared with the observed non-African dcfs.
The observed dcfs
are computed using four modern human populations from the Complete Genomics Diversity Panel (CGDP) and the draft sequence of the Neanderthal genome (Green et al. 2010
). Following Green et al. (2010)
, we chose the Yoruba population (YRI), the Utah residents with European ancestry (CEU), the Japanese (JPT), and the Han Chinese (CHB) from the CGDP to represent the African, the European, and the Asian populations. The Yoruba population is not representative of all the African populations as current African populations are very diverse (Campbell and Tishkoff 2008
; Tishkoff et al. 2009
). However, the CGDP has several individuals of Yoruba ancestry and like most Africans, the Yoruba population probably had no interactions with Neanderthals. Using these populations, we assess whether the dcfs
better supports a demographic history of recent admixture or ancient structure.