Sex hormones such as estrogen and testosterone are essential for sexually dimorphic behaviors in vertebrates. However, the hormone-activated molecular mechanisms that control the development and function of the underlying neural circuits remain poorly defined. We have identified numerous sexually dimorphic gene expression patterns in the adult mouse hypothalamus and amygdala. We find that adult sex hormones regulate these expression patterns in a sex-specific, regionally-restricted manner, suggesting that these genes regulate sex typical behaviors. Indeed, we find that mice with targeted disruptions of each of four of these genes (Brs3, Cckar, Irs4, Sytl4) exhibit extremely specific deficits in sex specific behaviors, with single genes controlling the pattern or extent of male sexual behavior, male aggression, maternal behavior, or female sexual behavior. Taken together, our findings demonstrate that various components of sexually dimorphic behaviors are governed by separable genetic programs.