Search tips
Search criteria 


Logo of jrsocmedLink to Publisher's site
J R Soc Med. 2005 January; 98(1): 18–20.
PMCID: PMC1079232

How the brain decides what we see

John Smythies, MD FRCP

Until recently it was thought that vision shows us what is actually 'out there' at any moment. It was supposed that all the images printed on the retina are coded and transmitted to the cortex; thus every detail of our visual experience would mirror faithfully the ongoing events in the stimulus field. The first clues that things are not that simple came from clinical investigations by Ramachandran and others.1 These showed that visual scotomata are not experienced as blank areas in the visual field but that the brain fills them in with (hallucinatory) sensations that the brain computes would most probably have been in this locus if there had been no scotoma. Clearly the brain mechanisms responsible for this 'filling in' were not evolved merely to fill in scotomata. They must have some wider function. Next, experiments in visual psychology have revealed that the brain will often construct sensations that override the visual input if the brain judges that the latter is too improbable. A notable example is reported by Kovács et al.2 These workers took two photographs, one of a monkey's face and the second of a leafy tropical jungle. They converted these into two pastiches each composed of portions of each photo so that in the location where one photo showed part of the monkey's face the other showed leafy jungle. Then each pastiche was shown separately to each retina so that retinal rivalry occurred. Under these circumstances, the subject did not see what was actually there—i.e., the two pastiches alternating—but rather a complete monkey face alternating with a complete leafy jungle. Clearly the brain had suppressed the improbable mixed pastiche in favour of what it was familiar with (and thus computed what was more probable).

It was then discovered that, during a saccade (rapid movement of the eye), information coming from the eye is suppressed and what we see is largely virtual reality created by the brain from memory.3 Kleiste et al.4 expressed it thus:

'When you look into a mirror and move your eyes left to right, you will see that you cannot observe your own eye movements. This demonstrates the phenomenon of saccadic suppression: during saccadic eye movements, visual sensitivity is much reduced.'

In this paper they presented evidence, from psychophysical and fMRI studies, that this saccadic suppression selectively targets the magnocellular visual stream. They concluded: 'The range of effects in our data and in single cell data, however, argues against a single thalamic mechanism that suppresses all cortical input. Instead, we speculate that saccadic suppression relies on multiple mechanisms operating in different cortical areas.' In other words, 'filling in' has a temporal as well as a spatial dimension, based on a widespread network that includes the superior colliculus and parts of the thalamus. Moreover, eye movements are closely linked also to attention shifts. For example, Deubel and Schneider5 report a fundamental difference between the preparation for saccades and the preparation for manual reaching—i.e., the difference between responses to the instructions 'in 5 seconds flip your eyes to the target' (which is delayed saccade) and 'in 5 seconds put out your hand to touch the target'. For delayed saccades, attention is pinned to the saccade target until the onset of the response. This is not the case for manual reaching.

Thus what we see does not depend entirely on what is out there but also to a considerable extent on what the brain computes to be most probably out there. So we can ask how extensive is this filling-in process, what is its normal function and how is it done? Some light on these questions may come from digital television compression technology. This seeks to send the maximum possible signal with the minimum computational power compatible with an adequate picture. Sending every detail of every scene over the TV system is efficient but carries a high computational cost. This is called the I programme. So TV engineers have supplemented this with a second programme, P, that derives the most probable scenes following scene 'A' based on the system's memory and its previous experience of such scene sequences. This is supplemented by a third ('P frame') programme that records only differences between scenes. The P programme is inefficient but cheap in computational cost. So the art is in finding the optimum mixture of I and P.

The brain seems to use a similar system. If nothing much is happening the brain may increase its use of the memoryderived P system at the expense of its retina-derived I system (P>I). Then, if a new and potentially significant stimulus (S) is received, it may switch over more to its I system (I>P) so that the new S can be more effectively analysed and dealt with. Also the change can be partial and localized. For example, the P system may continue to supply the background, leaving the I system to focus on the new S, now at the centre of attention.

The question then arises, what is the physiological mechanism that controls the P:I ratio? Yu and Dayan6 have suggested that acetylcholine (ACh) promotes the I programme (or exact inference model) and inhibits the P programme (or naïve inference model). When a new and potentially significant S is received, ACh neurons in the nucleus basalis in the basal forebrain are activated. These are the sole suppliers of ACh to the cortex. Yu and Dayan suggest that ACh potentiates thalamocortical conductance (essential for the I programme) by a nicotinic post-synaptic stimulation of cortical cells in layer IV of the cortex. At the same time it inhibits cortico-cortical conductance (essential for the P programme) by a muscarinic activation of inhibitory presynaptic receptors in layers I/II of the cortex.

This mechanism may explain why the ACh projection to the cortex uses mainly targeted synaptic transmission,7-9 but with some diffuse volume transmission as well.10-12 This is unlike the projections used by the other 'neuromodulators' dopamine, norepinephrine and serotonin that use exclusively volume transmission in the cortex. (In volume transmission a dense meshwork of fine axons bear many boutons-en-passage along their lengths that do not make synapses with the receiving neurons but simply spray their transmitter load into extracellular space where it diffuses to the nearest neurons.) The dual action described by Yu and Dayan means that ACh must be able to stimulate nicotinic but not muscarinic receptors in layer IV of the cortex, and vice versa in layer I/II. A diffuse volume transmission system could not accomplish this.

Yu and Dayan6 suggest that the role of ACh in the cortex is to report uncertainty in top-down (P) information—with the effect of boosting bottom-up sensory (I) information at the expense of top-down expectations. However, I would like to suggest that ACh may have an executive (motor) function in addition to any signalling role. This executive function would be to switch the brain's programmes from P>I to I>P, either globally or locally.

ACh also plays a role in saccades that are the motor expression of increased interest in the locus to which the saccade is directed. During the actual saccade, as I have indicated, the P system is potentiated and the I system is reduced. So do cortical levels of ACh fall during a saccade? A saccade is initiated by a cholinergic signal from the brainstem pedunculopontine nucleus that activates nicotinic receptors in the intermediate (motor) layer of the superior colliculus. Kobayashi et al.13 reported that a subset of pedunculopontine neurons (13/70) varied their activity levels with task performance, discharging at a higher rate in successful trials than in error trials. A combination of responses related to saccade execution, reward delivery, and task performance was observed in pedunculopontine neurons. Thus a saccade should both enhance I (at the saccade location immediately before the saccade) and enhance P (during the saccade).

In the muscarinic realm the sparse data do not as yet allow any such analysis.14,15 We await information on the central issue of whether reduced cortical cholinergic activity during a saccade is responsible for, or is associated with, reduced activity in the thalamocortical projection and/or increased activity in the cortico-cortical projection. So how could the observed P>I alteration during a saccade be achieved? The projection of the pedunculopontine nucleus to the nucleus basalis is massive and uses glutamate as its transmitter. The nucleus basalis contains large numbers of inhibitory GABA-ergic neurons as well as its large cholinergic neurons. It is not known whether the excitatory pedunculopontine projection to the nucleus basalis targets the ACh neurons or the GABA-ergic interneurons (Mesulam M, personal communication). If the latter is the case, this could provide the necessary mechanism. The pedunculopontine nucleus would initiate the saccade by its projection to the superior colliculus and subsequently inhibit the I system and promote the P system in the cortex by exciting inhibitory GABA-ergic cells in the nucleus basalis to reduce ACh release in the cortex. Clearly further experiments are needed to test this hypothesis.

An added complication is provided by the fact that the superior colliculus has a second cholinergic input—from the parabigeminal nucleus—that would appear to be functionally different from its input from the pedunculopontine nucleus. This second input goes to the (sensory) stratum griseum superficiale where it activates excitatory presynaptic nicotinic receptors on glutamatergic terminals (of axons from retinal neurons) that synapse on GABA-ergic interneurons.16,17 Binns and Salt16 conclude '... the ultimate effect of nicotinic receptor activation is to depress visual responses'. Thus here the nicotinic ACh system would appear to have the paradoxical effect of reducing the upward flow of collicular activity. The significance of this second cholinergic input to the superior colliculus is not at present clear except that this nucleus may be an integral part of a midbrain circuit that generates target location information.18

There is another intriguing possibility relating to the role of saccades, this time during rapid-eye-movement (REM) sleep. The saccades that are so prominent a feature of REM sleep are commonly thought to be due to the changing focus of attention on dream images during the course of the dream. However, REM sleep is an expression of pure freewheeling P activity. We have seen that, during waking saccades, P is promoted and I inhibited. Perhaps one function of saccades during REM sleep is to do the same? As we saw above, the initiation of saccades has the opposite effect. Dominance of one or other of these results might depend crucially on the timing or duration of these effects.

This hypothesis has implications for the philosophy of perception. At present most commentators assume that what we see is driven entirely by the sensory input. In other words we see what is actually out there. This position is inconsistent with the experimentally demonstrated fact that much of what we see in normal vision is virtual reality driven by the brain's P programmes.19


I am grateful to Maximilien d'Oreye for instruction in television compression technology and to Terry Sejnowski and Angela Wu for helpful comments on this paper.


1. Ramachandran VS, Blakeslee S. Phantoms in the Brain. New York, Morrow, 1998
2. Kovács I, Papathomas TV, Yang M, Fehér A. When the brain changes its mind: Intraocular grouping during retinal rivalry. Proc Natl Acad Sci USA 1996;93: 508-11 [PubMed]
3. Yarrow K, Haggard P, Heal R, Brown P, Rothwell JC. Illusory perceptions of space and time preserve cross-saccadic perceptual continuity. Nature 2001;414: 302-4 [PubMed]
4. Kleiser R, Seitz RJ, Krekelberg B. Neural correlates of saccadic suppression in humans. Curr Biol 2004;14: R195-7 [PubMed]
5. Deubel H, Schneider WX. Delayed saccades, but not delayed manual aiming movements, require visual attention shifts. Ann N Y Acad Sci 2003;1004: 289-96 [PubMed]
6. Yu AJ, Dayan P. Acetylcholine in cortical inference. Neural Networks 2002;15: 719-30 [PubMed]
7. Gaykema RP, van Weeghel R, Hersh LB, Luiten PG. Prefrontal cortical projections to the cholinergic neurons in the basal forebrain. J Comp Neurol 1991;303: 563-83 [PubMed]
8. Smiley JF, Subramanian M, Mesulam MM. Monoaminergic-cholinergic interactions in the primate basal forebrain. Neuroscience 1999;93: 817-29 [PubMed]
9. Turrini P, Casu MA, Wong TP, De Koninck Y, Ribeiro-da-Silva A, Cuello AC. Cholinergic nerve terminals establish classical synapses in the rat cerebral cortex: synaptic pattern and age-related atrophy. Neuroscience 2001;105: 277-85 [PubMed]
10. Umbriaco D, Garcia S, Beaulieu C, Descarries L. Relational features of acetylcholine, noradrenaline, serotonin and GABA axon terminals in the stratum radiatum of adult rat hippocampus (CA1). Hippocampus 1995;5: 605-20 [PubMed]
11. Mechawar N, Watkins KC, Descarries L. Ultrastructural features of the acetylcholine innervation in the developing parietal cortex of rat. J Comp Neurol 2002;443: 250-8 [PubMed]
12. De Lima AD, Singer W. Cholinergic innervation of the cat striate cortex: a choline acetyltransferase immunocytochemical analysis. J Comp Neurol 1986;250: 324-38 [PubMed]
13. Kobayashi Y, Inoue Y, Yamamoto M, Isa T, Aizawa HJ. Contribution of pedunculopontine tegmental nucleus neurons to performance of visually guided saccade tasks in monkeys. J Neurophysiol 2002;88: 715-31 [PubMed]
14. Oliva GA. Drug-induced variations in the probability of occurrence of multiple corrective saccades. Percept Mot Skills 2001;92: 687-90 [PubMed]
15. Vanni-Mercier G, Debilly G. A key role for the caudoventral pontine tegmentum in the simultaneous generation of eye saccades in bursts and associated ponto-geniculo-occipital waves during paradoxical sleep in the cat. Neuroscience 1998;86: 571-85 [PubMed]
16. Binns KE, Salt TE. The functional influence of nicotinic cholinergic receptors on the visual responses of neurones in the superficial superior colliculus. Vis Neurosci 2000;17: 283-9 [PubMed]
17. Lee PH, Schmidt M, Hall WC. Excitatory and inhibitory circuitry in the superficial gray layer of the superior colliculus. J Neurosci 2001;21: 8145-53 [PubMed]
18. Cui H, Malpeli JG. Activity in the parabigeminal nucleus during eye movements directed at moving and stationary targets. J Neurophysiol 2003;89: 3128-42 [PubMed]
19. Smythies JR, Ramachandran VS. An empirical refutation of the Direct Realist theory of perception. Inquiry 1998;40: 437-8

Articles from Journal of the Royal Society of Medicine are provided here courtesy of Royal Society of Medicine Press