Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Science. Author manuscript; available in PMC 2014 May 3.
Published in final edited form as:
PMCID: PMC3644373

Structural Basis for Molecular Recognition at Serotonin Receptors


Serotonin or 5-hydroxytryptamine (5-HT) regulates a wide spectrum of human physiology through the 5-HT receptor family. We report the crystal structures of the human 5-HT1B G protein-coupled receptor bound to the agonist anti-migraine medications ergotamine and dihydroergotamine. The structures reveal similar binding modes for these ligands, which occupy the orthosteric pocket and an extended binding pocket close to the extracellular loops. The orthosteric pocket is formed by residues conserved in the 5-HT receptor family, clarifying the family-wide agonist activity of 5-HT. Compared to the accompanying structure of the 5-HT2B receptor, the 5-HT1B receptor displays a 3 angstrom outward shift at the extracellular end of helix V, resulting in a more open extended pocket that explains subtype selectivity. Together with docking and mutagenesis studies, these structures provide a comprehensive structural basis for understanding receptor-ligand interactions and designing subtype-selective serotonergic drugs.

The neuromodulator serotonin (5-hydroxytryptamine; 5-HT) is essential for diverse functions at nearly every organ system in the human body (14). The activity of 5-HT is mediated through activation of members of a large family of 5-HT receptor proteins, which can be grouped into seven subfamilies (5-HT1-7) on the basis of sequence homology and signaling mechanisms (5). Except for the 5-HT3 receptor, which is a ligand-gated ion channel, the other twelve members are heterotrimeric guanine nucleotide binding protein (G protein) coupled receptors (GPCRs). The serotonergic system is a target of many widely prescribed drugs including atypical anti-psychotics, anti-migraine medications, anxiolytics and anti-depressants (1), and the recently approved anti-obesity medication lorcaserin (6, 7). However, clinical use of several serotonergic drugs caused unexpected side effects arising from off-target interactions with 5-HT receptor subtypes and related receptors for biogenic amine (1, 4, 8, 9).

The 5-HT1B receptor couples to G protein alpha subunits Gi or Go and is widely expressed in the brain and the cardiovascular system. In the CNS, the 5-HT1B receptor functions as an inhibitory presynaptic receptor to modulate the release of 5-HT and many other neurotransmitters (1, 2). The 5-HT1B receptor is a primary molecular target for the anti-migraine drugs ergotamine (ERG) and dihydroergotamine (DHE), which are efficacious 5-HT1B receptor agonists (10). Off-target activation of the related 5-HT2B receptor is responsible for the valvulopathic activity of many approved drugs and is the main reason for their withdrawal (912). We report two crystal structures of the human 5-HT1B receptor bound to the full agonists, ERG and DHE (tables S1 and S2). Comparison with the accompanying structure of the human 5-HT2B receptor bound to ERG (13) reveals critical structural determinants for ligand recognition and subtype selectivity, and provides a structural rationale for designing safer and better serotonergic drugs.

Crystallization studies of the 5-HT1B receptor were done with engineered constructs, 5-HT1B-1 and 5-HT1B-2 (14), which crystallized with ERG and DHE at resolutions of 2.7 Å and 2.8 Å, respectively. Due to the high similarity between these two structures (figure S2), for brevity we focus on the structure of the 5-HT1B-1/ERG complex for analysis and discussion of key structural features for ligand recognition and selectivity in 5-HT1B versus 5-HT2B receptors.

The main fold of the 5-HT1B receptor consists of a canonical seven-transmembrane (7TM) α-helical bundle (Fig. 1A). The extracellular loop 2 (ECL2) that partially covers the ligand binding pocket is stabilized by a C1223.25-C199ECL2 disulfide bond, highly conserved in GPCRs. Part of the N terminus folds on top of the binding pocket where Y40 forms hydrogen-bond interactions with ligand binding residue D3527.36 (figure S5) (15, 16). This feature suggests that the N terminus could have a role in ligand recognition in the 5-HT1B receptor by interacting with residues within the binding pocket.

Fig. 1
Overall architecture of the 5-HT1B receptor bound to ERG and comparison of the ligand binding pocket shapes of the 5-HT1B receptor and the 5-HT2B receptor. (A) The 5-HT1B receptor is shown as a light blue colored ribbon cartoon, with N terminus, ICL1 ...

The 5-HT1B/ERG complex structure revealed a large ligand binding cavity defined by residues from helices III, V, VI, VII and ECL2, comprising an orthosteric pocket embedded deep in the 7TM core and an extended binding pocket close to the extracellular entrance (Fig. 1). ERG adopts a binding mode with the ergoline ring system occupying the orthosteric binding pocket and the cyclic tripeptide moiety bound to the upper extended binding pocket (Fig. 2C). In the orthosteric pocket, the ergoline scaffold is anchored through the salt bridge interaction between its positively charged nitrogen and the carboxylate of D1293.32 which is fully conserved in 5HT and other monoamine receptors. The side chain of D1293.32 is further stabilized by a hydrogen bond to the hydroxyl of Y3597.43. Side chains of C1333.36, I1303.33, W3276.48, F3306.51 and F3316.52 form a narrow hydrophobic cleft, which packs tightly against the nearly planar ergoline ring system. In addition, the indole N-H hydrogen forms a hydrogen bond with T1343.37 (Fig. 2A). Comparison with the ERG bound 5-HT2B receptor structure revealed that the orthosteric binding pockets in two receptors are very similar, with the key interactions conserved (Fig. 2, A, D and E). The only difference is observed in the region where residues from helix V contact ERG: due to the lack of a side chain at G2215.42, the side chain of F2175.38 in the 5-HT2B receptor reaches into the ligand binding pocket and packs on top of the ERG indole ring; by comparison, the corresponding interaction in the 5-HT1B receptor occurs between the side chain of S2125.42 and ERG, while Y2085.38 does not interact with the ligand due to the outward shift of helix V (Fig. 2, A and D). Significant differences are observed in the extended binding pockets: the extended binding pocket of the 5-HT1B receptor is broader than that of the 5-HT2B receptor (Fig. 1, B, C and D) due to the 3.0 Å outward shift of the top of helix V. Moreover, in contrast to the relatively bulky M2185.39 of the 5-HT2B receptor, the corresponding residue of the 5-HT1B receptor is a smaller threonine, which results in a further expansion of this pocket (Fig. 2B). Despite these differences in contact residues between the two subtypes, the cyclic tripeptide moiety of ERG maintains a similar overall orientation relative to the ergoline moiety, likely stabilized by an intramolecular hydrogen bond (Fig. 2, D and E). The conformations of the phenyl group of the ERG cyclic tripeptide do differ: it contacts with L3476.58 and V3486.59 in the 5-HT2B receptor, whereas in the 5-HT1B receptor, it rotates to occupy a cavity close to T2095.39 at helix V. Correspondingly, M3376.58 turns into the pocket to contact with the phenyl ring of the ligand in the 5-HT1B receptor. As shown in the accompanying paper, these differences in ERG interactions correlate with different functional states: at the 5-HT1B receptor ERG causes full activation; whereas ERG induces intermediate G protein activation and β-arrestin biased signaling at the 5-HT2B receptor (13).

Fig. 2
Comparison of ligand-receptor interactions in 5-HT1B/ERG and 5-HT2B/ERG structures. (A), (B) and (C) Superposition of the ligand binding pockets of 5-HT1B receptor (light blue) and 5-HT2B receptor (white). The carbons of ligand ERG in the 5-HT1B and the ...

Mutations of several residues in the orthosteric binding site of the 5-HT1B receptor, including the conserved D1293.32, abolished the binding of the radioligand lysergic acid diethylamide (LSD) (table S4). The prototypical hallucinogen LSD has the same ergoline moiety as ERG, and these interactions within the orthosteric pocket appear to be the main driving force for binding of ergolines. In contrast, none of the tested mutations of the extended binding pocket residues drastically reduced the binding affinity of ERG, suggesting that the cyclic tripeptide moiety of ERG is accommodated without contributing substantially to the binding affinity. This extended binding site partially overlaps with the sites inferred in interaction with allosteric modulators in the M2 muscarinic acetylcholine receptor (17, 18) suggesting its potential role in mediating allosteric modulations at 5-HT receptors (13).

Alignment of all human 5-HT GPCRs sequences shows that the residues in the orthosteric binding pockets are much more conserved than those in the extended binding pockets (figure S7). This likely reflects an evolutionary pressure to maintain the structure of the orthosteric binding pocket for recognition of the endogenous ligand 5-HT (19). Molecular docking of 5-HT into the orthosteric binding pocket of the 5-HT1B and 5-HT2B receptors revealed important residues involved in the recognition of 5-HT (Fig. 3A), many of which have been previously implicated by site-directed mutagenesis and molecular modeling studies (20, 21). 5-HT shares a common chemical scaffold with ergolines (figure S4) and recognizes the 5-HT receptors in a similar manner as ERG and DHE. D3.32 forms a salt bridge with the positively charged amino group of 5-HT, while T3.37, which is highly conserved in 5-HT receptors and most other aminergic GPCRs, forms a hydrogen bond with the indole N-H hydrogen. This hydrogen bond appears to be important for the recognition of 5-HT by 5-HT1B and 5-HT2B receptors because mutating T3.37 to alanine reduces the affinity of 5-HT at both receptors by more than 10 fold (table S4). The indole ring points towards residues on helix V: S2125.42 and A2165.46 at the 5-HT1B receptor; G2215.42 and A2255.46 at the 5-HT2B receptor. This ligand-receptor interface is less polar in the 5-HT receptor family compared with those of other biogenic amine receptors (table S5), thereby perfectly matching the property of the 5-HT indole head group, which is less polar than those of other aminergic receptor native agonists, such as epinephrine, dopamine and histamine.

Fig. 3
Docking of the promiscuous (5-HT and LSD) and selective (sumatriptan and norfenfluoramine) ligands into the binding pockets of the 5-HT1B and 5-HT2B receptor structures. Docking of 5-HT (A), LSD (B), sumatriptan (C) and norfenfluramine (D) into the orthosteric ...

LSD promiscuously binds to 5-HT receptors: it has agonist activity at most 5-HT receptors (table S6), whereas it is a potent antagonist at 5-HT7A receptor (13). Docking of LSD into the 5-HT1B and 5-HT2B receptor structures suggests that binding of the LSD ergoline moiety in the orthosteric binding pocket is essentially identical to that of the ERG ergoline moiety (Fig. 3B). Many single point mutations within the orthosteric binding pocket reduce or abolish the binding of LSD at both 5-HT1B and 5-HT2B receptors (table S4). The conservation of the orthosteric binding pocket provides an atomic-level explanation for the observed promiscuous interactions between drugs like LSD and 5-HT receptors.

Triptans are 5-HT analogs that are among the most frequently prescribed anti-migraine medications which act primarily through 5-HT1B and 5-HT1D receptors (1, 2, 22). A common structural feature of triptans is the large substitution group at the 5′ position of the indole ring. Functional assays indicate that triptans act as agonists of the 5-HT1B receptor, but not the 5-HT2B receptor (table S7). To address the structural determinants of this selectivity, we performed docking simulations at both 5-HT1B and 5-HT2B receptors (figure S8). Whereas triptans were well accommodated in the 5-HT1B receptor binding pocket, the narrower extended binding pocket in the helix V region of the 5-HT2B receptor forced the ligands to adopt unfavorable positions (Fig. 3C and figure S8C). Among the tested triptans, donitriptan and eletriptan showed relatively higher potency in Gq –mediated signaling at the 5-HT2B receptor (table S7). These two triptans have longer and more flexible linkers between 5′ substituents and the indole ring, enabling them to fit better in the narrow binding pocket of the 5-HT2B receptor. The M2185.39A mutation of the 5-HT2B receptor, which increases the space in the binding pocket, significantly enhanced the potency of donitriptan and eletriptan (figure S9), although their potency was still lower than that at the 5-HT1B receptor. Thus, the broader opening near the extracellular end of helix V in the 5-HT1B receptor appears to be important for selectively accommodating the 5′ substituents of triptan ligands, whereas the narrower pocket in the 5-HT2B receptor shows reduced binding to these compounds.

Norfenfluramine is the active metabolite of fenfluramine, which is one of the two components of the infamous ‘Fen-Phen’ anti-obesity cocktail. The anti-obesity effect of norfenfluramine occurs mainly through activation of the 5-HT2C receptor (1), but due to its high potency and efficacy as a 5-HT2B receptor agonist, it can cause life-threatening side effects including pulmonary hypertension and heart valve disease (9, 2325). To elucidate the structural basis for norfenfluramine’s subtype selectivity, we simulated its binding to the 5-HT1B and 5-HT2B receptor structures (Fig. 3D). Norfenfluramine tightly fit the orthosteric binding pocket of the 5-HT2B receptor with F2175.38 and M2185.39 forming a hydrophobic cap that interacted with the trifluoromethyl group. The F2175.38A mutation (table S8) and the M2185.39V mutation (26), both reduce the potency of norfenfluramine compared to that for the wild type 5-HT2B receptor. In the docking model of the 5-HT1B receptor, these close contacts are missing, thus the potency of norfenfluramine is reduced compared to that at the 5-HT2B receptor (table S8).

Species-specific differences in the ligand binding properties of 5-HT1B and other 5-HT receptors from rodents and humans have impeded the extrapolation of findings from animal model to humans (27, 28). The rodent 5-HT1B receptors, for instance, have a much higher binding affinity than the human 5-HT1B receptor for certain adrenergic compounds caused by a difference at position 7.39 (N351 for rat and mouse, T355 for human) (27, 29). Modeling of the T355N mutation into the binding pocket of human 5-HT1B receptor revealed that N3557.39, together with D1293.32 and Y3597.43, form a polar interaction network that anchors the propanolamine moiety of adrenergic antagonists, propranolol and cyanopindolol, thereby mimicking the recognition modes observed in both the β1 and β2 adrenergic receptors (Fig. 4). Hydrophobic interactions of I1303.33, W3276.48, F3306.51 and F3316.52 with the aromatic rings of propranolol and cyanopindolol are largely conserved between 5-HT1B and β-adrenergic receptors (30, 31), allowing high affinity binding of these antagonists in the human 5-HT1B T3557.39N and rodent 5-HT1B receptors. These findings thereby provide a structural explanation for the pharmacological differences between these rodent and human GPCRs.

Fig. 4
Structural basis for differences in the pharmacological properties between human and rodent 5-HT1B receptors. The high affinity β-adrenergic antagonists propranolol (A) and cyanopindolol (B), both shown in yellow colored carbons, are docked in ...

In conclusion, comparative analysis of the 5-HT1B and 5-HT2B receptor structures and functions, together with specific mutagenesis studies, provided a comprehensive framework for understanding ligand promiscuity and selectivity, thereby aiding development of safer and more effective medications that target the GPCR superfamily.

Supplementary Material

Supplementary Online Materials


This work was supported by the NIGMS PSI:Biology grant U54 GM094618 for biological studies and structure production (target GPCR-118) (V.K., V.C. and R.C.S.), NIH Common Fund in Structural Biology grant P50 GM073197 for technology development (V.C. and R.C.S.); the Jay and Betty Van Andel Foundation, R01 DK071662, Ministry of Science and Technology (China) grants 2012ZX09301001-005 and 2012CB910403, and Amway (China) (H.E.X.); R01 MH61887, U19 MH82441, R01 DA27170 and the NIMH Psychoactive Drug Screening Program (X.-P.H., E.V., J.D.M. and B.L.R.) and the Michael Hooker Chair of Pharmacology (B.L.R.). D.W. is supported by a Boehringer Ingelheim Fonds PhD Fellowship. We thank J. Velasquez for help on molecular biology; T. Trinh, K. Allin and M. Chu for help on baculovirus expression; I. Wilson for careful review and scientific feedback on the manuscript; K. Kadyshevskaya for assistance with figure preparation; A. Walker for assistance with manuscript preparation; J. Smith, R. Fischetti, and N. Sanishvili for assistance in development and use of the minibeam and beamtime at GM/CA-CAT beamline 23-ID at the Advanced Photon Source, which is supported by National Cancer Institute grant Y1-CO-1020 and National Institute of General Medical Sciences grant Y1-GM-1104; and Z. Wawrzak and J. S. Brunzelle for assistance in data collection at the beamlines of sector 21 (Life Sciences Collaborative Access Team), which is funded in part by the Michigan Economic Development Corp. and the Michigan Technology Tri-Corridor Grant 085P1000817. Use of the Advanced Photon Source was supported by the Office of Science of the United States Department of Energy. The coordinates and the structure factors have been deposited in the Protein Data Bank under the accession codes: 5-HT1B/ERG (4IAR), 5-HT1B/DHE (4IAQ).

References and Notes

1. Berger M, Gray JA, Roth BL. The expanded biology of serotonin. Annu Rev Med. 2009;60:355. [PubMed]
2. Hoyer D, et al. International Union of Pharmacology classification of receptors for 5-hydroxytryptamine (Serotonin) Pharmacol Rev. 1994;46:157. [PubMed]
3. Roth BL, Hanizavareh SM, Blum AE. Serotonin receptors represent highly favorable molecular targets for cognitive enhancement in schizophrenia and other disorders. Psychopharmacology (Berl) 2004;174:17. [PubMed]
4. Roth BL, Sheffler DJ, Kroeze WK. Magic shotguns versus magic bullets: selectively non-selective drugs for mood disorders and schizophrenia. Nat Rev Drug Discov. 2004;3:353. [PubMed]
5. Kroeze WK, Kristiansen K, Roth BL. Molecular biology of serotonin receptors structure and function at the molecular level. Curr Top Med Chem. 2002;2:507. [PubMed]
6. Goldenberg MM. Pharmaceutical approval update. PT. 2012;37:499. [PMC free article] [PubMed]
7. Smith BM, et al. Discovery and structure-activity relationship of (1R)-8-chloro-2,3,4,5-tetrahydro-1-methyl-1H-3-benzazepine (Lorcaserin), a selective serotonin 5-HT2C receptor agonist for the treatment of obesity. J Med Chem. 2008;51:305. [PubMed]
8. Besnard J, et al. Automated design of ligands to polypharmacological profiles. Nature. 2012;492:215. [PMC free article] [PubMed]
9. Roth BL. Drugs and valvular heart disease. N Engl J Med. 2007;356:6. [PubMed]
10. Villalon CM, Centurion D, Valdivia LF, de Vries P, Saxena PR. Migraine: pathophysiology, pharmacology, treatment and future trends. Curr Vasc Pharmacol. 2003;1:71. [PubMed]
11. Affected drugs include the anti-migraine drugs: ERG, DHE and methysergide (usage limited); the appetite suppressant drugs: fenfluramine, dex-fenfluramine and aminorex (all withdrawn); and some drugs used for treating Parkinson’s Disease: pergolide and cabergoline (both withdrawn).
12. Rothman RB, Baumann MH. Serotonin releasing agents. Neurochemical, therapeutic and adverse effects. Pharmacol Biochem Behav. 2002;71:825. [PubMed]
13. Wacker D, et al. Crystal Structure of the 5-HT2B Receptor: Biological Selectivity of Serotonin Receptors. Science (companion paper) 2012
14. Materials and methods and supplementary figures and tables are available as supplementary material on Science Online.
15. Superscripts refer to the Ballesteros-Weinstein numbering, in which the most conserved among class A GPCRs residues in each transmembrane helix are designated x.50, where x is the helix number.
16. Ballesteros JA, Weinstein H. Integrated methods for the construction of three-dimensional models and computational probing of structure-function relations in G protein-coupled receptors. Methods Neurosci. 1995;25:366.
17. Residues, Y177ECL2, W4227.35, T4237.36 in muscarinic acetylcholine receptor M2 are reported to be involved in mediating the action of allosteric modulators. Corresponding residues in 5-HT1B receptor, V200ECL2, F3517.35, D3527.36 are part of the extended binding site.
18. Haga K, et al. Structure of the human M2 muscarinic acetylcholine receptor bound to an antagonist. Nature. 2012;482:547. [PMC free article] [PubMed]
19. Katritch V, Cherezov V, Stevens RC. Diversity and modularity of G protein-coupled receptor structures. Trends Pharmacol Sci. 2012;33:17. [PMC free article] [PubMed]
20. Braden MR, Nichols DE. Assessment of the roles of serines 5.43(239) and 5.46(242) for binding and potency of agonist ligands at the human serotonin 5-HT2A receptor. Mol Pharmacol. 2007;72:1200. [PubMed]
21. Shapiro DA, Kristiansen K, Kroeze WK, Roth BL. Differential modes of agonist binding to 5-hydroxytryptamine(2A) serotonin receptors revealed by mutation and molecular modeling of conserved residues in transmembrane region 5. Mol Pharmacol. 2000;58:877. [PubMed]
22. Mehrotra S, et al. Current and prospective pharmacological targets in relation to antimigraine action. Naunyn Schmiedebergs Arch Pharmacol. 2008;378:371. [PubMed]
23. Connolly HM, et al. Valvular heart disease associated with fenfluramine-phentermine. N Engl J Med. 1997;337:581. [PubMed]
24. Rothman RB. Anorexogen-related cardiac valvulopathy. Ann Intern Med. 2002;136:779. [PubMed]
25. Rothman RB, et al. Evidence for possible involvement of 5-HT(2B) receptors in the cardiac valvulopathy associated with fenfluramine and other serotonergic medications. Circulation. 2000;102:2836. [PubMed]
26. Setola V, Dukat M, Glennon RA, Roth BL. Molecular determinants for the interaction of the valvulopathic anorexigen norfenfluramine with the 5-HT2B receptor. Mol Pharmacol. 2005;68:20. [PubMed]
27. Adham N, et al. A single point mutation increases the affinity of serotonin 5-HT1D alpha, 5-HT1D beta, 5-HT1E and 5-HT1F receptors for beta-adrenergic antagonists. Neuropharmacology. 1994;33:387. [PubMed]
28. Nelson DL, Lucaites VL, Audia JE, Nissen JS, Wainscott DB. Species differences in the pharmacology of the 5-hydroxytryptamine2 receptor: structurally specific differentiation by ergolines and tryptamines. J Pharmacol Exp Ther. 1993;265:1272. [PubMed]
29. Oksenberg D, et al. A single amino-acid difference confers major pharmacological variation between human and rodent 5-HT1B receptors. Nature. 1992;360:161. [PubMed]
30. Cherezov V, et al. High-resolution crystal structure of an engineered human beta2-adrenergic G protein-coupled receptor. Science. 2007;318:1258. [PMC free article] [PubMed]
31. Warne T, et al. Structure of a beta1-adrenergic G-protein-coupled receptor. Nature. 2008;454:486. [PMC free article] [PubMed]