Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Neurosci. Author manuscript; available in PMC 2010 October 7.
Published in final edited form as:
PMCID: PMC2858413

Initiating the development of multisensory integration by manipulating sensory experience


The multisensory integration capabilities of superior colliculus (SC) neurons emerge gradually during early postnatal life as a consequence of experience with cross-modal stimuli. Without such experience neurons become responsive to multiple sensory modalities but are unable to integrate their inputs. The present study demonstrates that neurons retain sensitivity to cross-modal experience well past the normal developmental period for acquiring multisensory integration capabilities. Experience surprisingly late in life was found to rapidly initiate the development of multisensory integration, even more rapidly than expected based on its normal developmental time course. Furthermore, the requisite experience was acquired by the anesthetized brain and in the absence of any of the stimulus-response contingencies generally associated with learning. The key experiential factor was repeated exposure to the relevant stimuli, and this required that the multiple receptive fields of a multisensory neuron encompassed the cross-modal exposure site. Simple exposure to the individual components of a cross-modal stimulus was ineffective in this regard. Furthermore, once a neuron acquired multisensory integration capabilities at the exposure site, it generalized this experience to other locations, albeit with lowered effectiveness. These observations suggest that the prolonged period during which multisensory integration normally appears is due to developmental factors in neural circuitry in addition to those required for incorporating the statistics of cross-modal events; that neurons learn a multisensory principle based on the specifics of experience and can then apply it to other stimulus conditions; and that the incorporation of this multisensory information does not depend on an alert brain.

Keywords: Dark Rearing, Colliculus, Vision, Auditory, Maturation, Plasticity


The capability to integrate information from multiple senses is a characteristic feature of superior colliculus (SC) neurons. It is through this process that the spatiotemporally concordant cross-modal stimuli that are derived from the same event produce markedly enhanced multisensory responses and an increased likelihood of SC-mediated orientation responses to it. This phenomenon is referred to as multisensory enhancement, which is a specific case of the more general phenomenon of multisensory integration (Stein and Stanford, 2008).

This ability to integrate information from different senses is not restricted to any given species or brain structure. Rather, it is found in all species that have been examined and in neurons at many locations within their nervous systems (Stein and Meredith, 1993; Zahar et al., 2009). The ubiquity of multisensory integration speaks to its broad adaptive significance for a variety of functions in a host of organisms that face very different ecological challenges. Nevertheless, and despite the substantial survival challenges faced by the newborn, extant data suggest that this capability appears comparatively late in postnatal development. Thus, for example, SC neurons in the newborn are incapable of multisensory integration because they have not yet become responsive to multiple sensory inputs (Stein et al., 1973), or because they cannot yet synthesize those cross-modal inputs (Wallace and Stein, 1997, , 2001).

It is only after considerable experience during early life with the statistics of cross-modal events that the brain is able to craft the spatial, temporal, and other principles that are characteristic of multisensory integration (Wallace and Stein, 2007). When, for example, visual-auditory experience is precluded by rearing cats in the dark, their SC neurons do not express their signature ability to integrate visual-nonvisual inputs (Wallace et al., 2004). Normally this capacity appears within the first few months of postnatal life, a time during which rapid changes in neural architecture are taking place in many interrelated brain structures (Katz, 1999; Hensch, 2004). It is often referred to as a “sensitive” period, because the absence of relevant experience during this time is difficult or impossible to compensate for later in life, and many normal functions are delayed or precluded. For example, if not corrected soon enough, early blindness or deafness can have long-standing detrimental effects on unisensory and/or multisensory functions (Hubel, 1978; Anooshian and Bryan, 1979; Carlson et al., 1987; Welsh et al., 1996; Putzar et al., 2007).

The present studies were initiated to determine whether the cross-modal sensory experience required for instantiating multisensory integration in SC neurons could be acquired at maturity, far later in life than is normally the case. The results reveal that the adult brain is surprisingly efficient in incorporating these experiences and in developing this capacity, and that these processes do not require an alert brain, an associated reward, or any of the stimulus-response contingencies normally associated with learning.

Materials and Methods

All protocols used were in accordance with the Guide for the Care and Use of Laboratory Animals (National Institutes of Health Publication 86–23) and were approved by the Animal Care and Use Committee of Wake Forest University School of Medicine, an AAALAC-accredited institution.

Housing environment and surgical preparation

Five cats were deprived of visual and visual-nonvisual experience by rearing them in a light-tight housing environment (Wallace et al., 2004). Auditory experience, and all other sensory experience, was not constrained. Daily care was accomplished with the aid of binocular infrared goggles, and an infrared closed circuit TV system allowed animals to be observed in their home environment from an adjacent room. Surgical implantation, routine veterinary procedures, and transport to the electrophysiological recording room within a tightly-masked carrier were conducted only after animals were anesthetized and either blindfolded or had their eyelids taped shut with paper tape in the dark room. When the animals matured (7–12 months), they were implanted with stainless-steel recording chambers using previously described techniques (McHaffie and Stein, 1983). Briefly, each animal was first anesthetized with a mixture of ketamine hydrochloride (20mg/kg, i.m.) and acepromazine maleate (0.4 mg/kg, i.m.), fitted with opaque contact lenses or the eyelid tape, intubated through the mouth and then placed in a stereotaxic apparatus. Surgical anesthesia was maintained with isoflurane (1.5–3%). During surgery, expiratory CO2, blood pressure and heart rate were continuously monitored using a Digital Vital Signs Monitor (VetSpecs VSM7), and body temperature was maintained at 37 ~ 38 °C with a heating pad. A recording chamber was stereotaxically placed over a craniotomy to provide access to the SC via the overlying cortex and attached to the skull with stainless steel screws and dental acrylic. After surgery, postsurgical analgesics (butorphanol tartrate; 0.1–0.2 mg/kg and ketoprofen 1mg/kg) were administered as needed, and antibiotics (cephazolin sodium; 25mg/kg) were administered twice a day for 7 days.

Electrophysiological recording procedures

Weekly recording sessions began after a postsurgical recovery period of at least 7 days. In the very first experiments, multisensory neurons were isolated and tested with combinations of visual, auditory, and visual-auditory stimuli to assess the multisensory integration capabilities of these neurons in their “naïve” state. However, each subsequent recording session began with an “exposure” period followed by a “testing” period (see below). For each session the animal was anesthetized with a combination of ketamine hydrochloride (20mg/kg, i.m.) and acepromazine maleate (0.4 mg/kg, i.m.), intubated, and ventilated mechanically. The animal rested comfortably in a recumbent position while its head was fixed on a stereotaxic apparatus through two horizontal stainless steel bars that were inserted into the openings in the recording chamber. Respiratory rate and volume were modulated to keep the end tidal CO2 at ~ 4.0%. CO2, heart rate, and blood pressure were monitored continuously to ensure the maintenance of a continued state of clinical anesthesia. Paralysis was induced with an injection of pancuronium bromide (0.06 mg/kg, i.v.) to prevent ocular drift. The pupil of the right eye was dilated with ophthalmic atropine sulfate (1%), and fitted with a contact lens to correct for refraction. The left eye was occluded with an opaque contact lens. The optic disc of the right eye was mapped using an ophthalmoscope and reverse-projection onto a tangent screen. Anesthesia, paralysis, and hydration were maintained by continuous intravenous infusion of ketamine hydrochloride (4–6 mg kg−1 h−1), pancuronium bromide (0.15–0.2 mg kg−1 h−1), and 5% dextrose in sterile physiological saline (3–6 ml/h). Body temperature was kept at 37 ~ 38 °C. Conventional methods were used for single neuron electrophysiological recording. A glass-coated tungsten electrode (tip diameter: 1–3μm, impedance: 1–3MΩ at 1 KHz) was positioned and lowered to the superficial layers of the SC via a hydraulic microdrive. The daily exposure trials were then conducted (see below), after which the electrode was advanced to search for single neurons in the multisensory (i.e., deep) layers. Once identified, single units were presented with test stimuli (see below). Single-unit neural activity was recorded, amplified, and routed to an oscilloscope, audio monitor, and computer for online and offline analyses as in the past (see below). At the end of the recording session, the anesthesia and paralysis were terminated. The eyes were covered and the animal was returned to its home cage in the dark room once stable respiration and locomotion were reinstated.

Stimulus exposure procedure

At the beginning of each recording day each animal was exposed to repeated presentations of sensory stimuli (1800/day). Different animals were exposed to different stimulus configurations. In two animals, a pair of spatiotemporally congruent visual and auditory stimuli (i.e., cross-modal stimuli) were presented unilaterally in either macular (0°, Cat 1) or contralateral extra-macular (30°, Cat 2) visual space, as shown in figure 1, at a 6s interstimulus-interval (3hrs/day). Two other animals were exposed to these same visual and auditory component stimuli, in these same spatial locations and for the same number of iterations, but these modality-specific stimuli were presented independently in an interleaved fashion (Cat 3 at 0°, cat 4 at 30°) with a 4s inter-stimulus interval (4hrs/day). A fifth animal (Cat 5) was also exposed to these modality-specific stimuli, but in random order and at random locations in space. The auditory stimulus consisted of a brief (100ms) 65 dB burst of broadband noise (20 to 20, 000 Hz) against the ambient background noise of 51.2–52.0 dB. The visual stimulus was a moving bar (width × height: 10°× 2°) of light (13.67 cd/m2 on a background luminance of 0.16 cd/m2), with a speed of 100°/s, and a duration of 100ms.

Figure 1
The two experimental exposure sites included one macular (left) and one extra-macular (right)

Search strategy and receptive field mapping

Sensory-responsive neurons were identified using a variety of visual and auditory search stimuli. Visual search stimuli consisted of moving bars of light back-projected from an LC 4445 Philips projector onto a tangent screen located 45cm from the front of the animal. Auditory stimuli consisted of broadband (20–20,000 Hz) noise bursts, hisses, clicks, and taps. Stimuli were controlled using custom software operating a NIDAQ digital controller (National Instruments) connected to a personal computer. When a visual-auditory neuron was isolated, its modality-specific visual and auditory receptive fields were mapped as in the past (Alvarado et al., 2008). The visual receptive field was mapped with moving light bars and the auditory receptive field was mapped with broadband noise bursts from any of 16 hoop-mounted speakers placed 15° apart and 15cm from the head on a rotating hoop that permitted adjustments in elevation. Custom software acquired raw data waveforms and impulses from single neurons (after A/D conversion) identified using a threshold criterion of 3x elevation of the action potential amplitude above background noise

Testing procedure

Once a multisensory neuron was identified, a short series (15–20 trials) of pre-tests were conducted using the visual, auditory, and spatiotemporally concordant visual-auditory stimuli identified above. The purpose of these tests was to choose stimuli that would maximize the likelihood of exposing the multisensory integration capabilities of the neuron under study. Thus, individual modality-specific component stimuli were chosen to be weakly effective for the particular neuron under study, and were presented in spatial congruence (Stanford et al., 2005) within the most responsive (i.e., “best”) areas of its visual and auditory receptive fields (Kadunce et al., 2001). These regions were identified whenever apparent and identified as the receptive fields in subsequent text. Lastly, although many naïve neurons preferred simultaneous visual and auditory stimuli (Wallace and Stein, 1997), often neurons in normal adults preferred an offset between these two stimuli (Meredith et al., 1987). In all the neurons studied, these pretest trials were used to determine the best stimulus offset asynchrony (SOA) to be used in subsequent tests of multisensory integration.

Data acquisition and analysis

Impulse times (1ms resolution) were recorded for each trial and analyzed offline. The response window was defined using the algorithm identified in earlier studies (Rowland and Stein, 2007; 2008). The mean spontaneous firing rate for each condition was calculated in the 500ms window preceding the stimulus. The magnitude of each response was identified as the mean number of impulses occurring in the response window minus the expected number given the spontaneous firing rate. The response to the combined cross-modal stimuli was compared with the response evoked by the more effective modality-specific stimulus to create the multisensory integration index (MSI): MSI = [(CM − SMmax)/SMmax] × 100%, where CM represents the mean magnitude of multisensory response, SMmax represents the magnitude of the response evoked by the more effective modality-specific stimulus. Data were compared statistically to determine significant differences using SPSS 11.5 (SAS), and t-tests, Mann-Whitney U tests, ANOVA and X2 tests where appropriate. All of the data were expressed as mean ± standard deviation and the minimum criterion for statistical significance was p <0.05.


A total of 434 multisensory (visual-auditory) neurons were studied in the present experiments. Of these, 26 neurons were identified as naïve and were studied in initial experiments (i.e., prior to the experiments containing exposure trials) in order to provide a baseline for comparison. These naïve neurons had response properties very similar to those previously reported in earlier dark rearing studies (Wallace et al., 2004). Their receptive fields were large, and thus the visual and auditory maps they formed likely had lower than normal spatial resolution. Although the neurons were responsive to visual and auditory stimuli, they were incapable of integrating spatiotemporally concordant visual-auditory stimuli to produce the enhanced multisensory response that characterizes their counterparts in the SC of animals raised in normal environments (Meredith and Stein, 1986; Jiang et al., 2001; Kadunce et al., 2001; Alvarado et al., 2007b). The unisensory and multisensory responses of a typical naïve multisensory neuron from a dark reared animal and a typical multisensory neuron from a normally reared animal are shown in figure 2.

Figure 2
Multisensory integration did not develop in the neurons of dark reared animals in the absence of cross-modal experience

Following these initial experiments, animals were exposed to the controlled repeated presentation of different sensory stimuli in the experimental room (i.e., “exposure trials”) at the beginning of each recording session. It is important to note that the animals were otherwise kept in their dark housing facility so that the experimental circumstance was their sole source of experience with cross-modal stimuli, and that they could accumulate exposure trials over multiple experimental sessions.

A total of 408 multisensory neurons were studied in this fashion: 186 in animals provided spatiotemporally coincident cross-modal exposure trials (Cat 1, n=98; Cat 2, n=88); 147 in animals given interleaved modality-specific exposure trials (Cat 3, n = 75; Cat 4, n = 72); and 75 in the animal (Cat 5) exposed to randomly interleaved modality-specific stimuli at random locations. In the first experimental sessions containing exposure trials, neuronal response properties closely resembled those of the naïve population studied prior to the exposure trials; that is, they responded to both visual and auditory stimuli, but did not produce enhanced responses to these stimuli when presented together in spatiotemporal register. However, over the course of multiple experimental sessions in which the animals were provided repeated exposure to these spatiotemporally concordant cross-modal stimuli (i.e., Cats 1 and 2), there was a striking increase in the probability of identifying neurons capable of multisensory integration.

The visual, auditory, and multisensory responses recorded from three different neurons after different numbers of cross-modal exposure trials are shown in figure 3. The first two neurons were recorded from the animal with cross-modal exposure in extra-macular space, the third from the animal with exposure in macular space. In each case, the visual and auditory receptive fields of each neuron encompassed the exposure site. This was the first neuron identified that was capable of multisensory integration. It was encountered after only 3,600 cross-modal exposure trials, which roughly equated to 6 hours of experience with controlled cross-modal stimuli. Although the neuron’s multisensory response was comparatively modest, its magnitude exceeded that of its response to the most effective unisensory component stimulus (visual) by a significant margin. This was not apparent in any of the previously identified neurons. The neurons depicted in B and C, were encountered, respectively, after 37,800 and 50,400 exposure trials. In these cases their multisensory responses were comparatively larger and exceeded the sum of their unisensory component responses. Such superadditive responses are characteristic of multisensory responses to weak modality-specific stimuli (Stanford et al., 2005), but also appeared to reflect the additional cross-modal experience they obtained (see below). It should be noted, however, that given the nature of the sampling procedure it was possible only to determine that a given neuron required fewer than the number of exposure trials received at the time of recording.

Figure 3
Exposure of dark reared animals to repeated presentations of cross-modal stimuli led to the development of multisensory integration

Repeated exposure to equivalent amounts of randomly-interleaved modality-specific visual and auditory stimuli in the same locations did not have the same effect, and did not yield a substantial proportion of neurons with the capacity for multisensory integration. Instead, the multisensory response properties of neurons in these animals closely resembled those recorded from neurons prior to the exposure trials. Figure 4 provides three example neurons recorded after different numbers of modality-specific exposure trials, and unlike the neurons shown in figure 3, none show enhanced responses to spatiotemporally concordant stimuli presented within its receptive fields. Presumably, the absence of temporal concordance eliminated the link between the cross-modal stimuli, and thus did not provide the essential cross-modal experience necessary for the development of this capacity.

Figure 4
Interleaved modality-specific (visual, auditory) exposure did not initiate the development of multisensory integration

As shown in figure 5, the increases in multisensory response magnitude with exposure were not due to a concomitant decrease in the comparator unisensory responses (the response to the most effective of the component stimuli). This is an important factor as it eliminates the possibility that smaller comparator responses could have inadvertently been used in later samples. Had this happened, proportionately larger multisensory responses could have been obtained, not as a consequence of increased cross-modal exposure, but because of the principle of inverse effectiveness (Stein et al., 2009). The relative constancy of the unisensory comparator responses was due to the fact that stimulus efficacies were adjusted for each neuron to maximize its potential for generating an enhanced multisensory response (see Methods). However, despite the constancy of the unisensory comparator response magnitudes, there were obvious increases in the multisensory response magnitudes after repeated exposure to spatiotemporally concordant cross-modal stimuli (A and B). Similar increases were not seen in neurons recorded from animals that were given modality-specific exposure (C, D, and E). Their multisensory and comparator unisensory responses were equivalent regardless of the amount of sensory exposure.

Figure 5
The proportionate vigor of multisensory responses gradually increased with increasing cross-modal exposure

After only hours of experience with spatiotemporally concordant cross-modal stimuli these multisensory responses reached the criterion for multisensory enhancement (i.e., multisensory integration) as shown in figure 6A and B. This was true regardless of whether the cross-modal exposure occurred in macular or extra-macular space, as long as the exposure site was encompassed by the neuron’s receptive fields. Indeed, there were no significant differences in the data based on exposure site, although the first few neurons exhibiting multisensory integration were obtained in the animal with extra-macular exposure. The first neuron exhibiting multisensory integration was obtained after approximately 6 hours of cross-modal exposure and the incidence of these neurons increased rapidly thereafter. There was also a rapid increase in the magnitude of the multisensory enhancement in these neurons, as quantified by the MSI. This was contrary to our expectation based on observations in normally reared animals (Wallace and Stein, 1997), because it indicated that multisensory integration had not achieved an adult-like profile as soon as it appeared. Rather, the magnitude of the observed multisensory enhancements increased with increasing numbers of exposure trials.

Figure 6
Cross-modal exposure led to the development of an ability to integrate cross-modal inputs and produce multisensory enhancement

The maximum incidence of exposure-induced multisensory integration was achieved after approximately 27,000 trials (15 experimental sessions), which roughly equates to 47 hours of experience with these cross-modal stimuli. At this point the incidence of neurons with multisensory integration capabilities and their MSIs were now no lower than that found in normally reared adults (Meredith and Stein, 1986; Jiang et al., 2001; Perrault et al., 2003; Stanford et al., 2005; Alvarado et al., 2007a). Yet, as noted above, neurons in animals exposed to the modality-specific stimulus components independently did not develop multisensory integration (Fig. 6C, D, and E). Instead, their multisensory responses were rarely significantly above their unisensory comparator response, and there were no significant changes in this pattern as a result of increasing numbers of modality-specific exposure trials.

To better summarize these developmental trends, the experimental series was divided and the data grouped into three stages (Fig. 7): a first stage of <10,800 exposure trials, a second stage of 10,800–27,000 exposure trials, and a third and final stage of >27,000 exposure trials. The data were combined across animals based on whether their exposure consisted of cross-modal or modality-specific exposure. In the first stage, it was only the rare neuron (13.4%, 4/28) that manifested the ability to integrate its cross-modal inputs and produce an enhanced response. Thus, the mean MSI in the cross-modal (15.93% ± 28.27%) and modality-specific (10.91% ± 20.24%) exposure groups were still quite similar. But in the second stage, neurons exhibiting multisensory integration in the cross-modal group increased to more than half the population of neurons sampled (59.4%, 19/32), a significant increase over the first stage (X2 test, df = 1, X2 = 12.84, P < 0.001), and higher than that (7.0%, 4/66) obtained from the modality-specific exposure groups at the same stage (X2 test, df = 1, X2 = 34.11, P < 0.001). Similarly, the mean MSI (50.61% ± 60.27%) had significantly increased (Mann-Whitney U Statistic= 304.000; P = 0.033) and was higher than the second stage (15.55% ± 30.17%) in the modality-specific group (Mann-Whitney U Statistic= 592.000; P = 0.006).

Figure 7
Stages in the development of multisensory integration following sensory exposure

In the third stage, SC multisensory integration capabilities in the cross-modal exposure group had reached the mature level, comprising 90 % (63/70) of the sampled population and having a mean MSI of 124.2% (± 70.2%) which represents a significant increment above the second stage in both incidence (X2 test, df = 1, X2 =13.07, P < 0.001) and magnitude (Mann-Whitney U Statistic= 450.50; P = <0.001). In contrast, only 9.6% (8/83) of multisensory neurons in animals with modality-specific exposure paradigms became capable of integrating visual-auditory inputs in the third stage (X2 test, df = 1, X2 = 98.60, P < 0.001) and their MSI was very low (18.27% ± 32.80%).

For the cross-modal experience to initiate the maturation of multisensory integration capabilities in any given neuron, the neuron’s visual and auditory receptive fields had to encompass the exposure site (Fig. 8). These neurons were categorized as Type I. Neurons in which one or both of its receptive fields did not encompass the exposure site were combined (there was no difference between these groups) and categorized as Type II (10 had one receptive field encompassing the exposure site, 18 had neither). Whereas the vast majority (90%, 63/70) of Type I neurons developed multisensory integration capabilities (Fig. 8E), very few (14.3% 4/28) Type II neurons developed this capability. Similarly, the mean MSI of Type I neurons (124.2% ± 70.2%) was significantly higher than that of the Type II neurons (10.2% ± 35.3%) (Mann-Whitney U Statistic = 78.50, P < 0.001). Thus, it appeared that the simple exposure of an animal to cross-modal stimuli at a given spatial location was insufficient to initiate the development of multisensory integration in the entire population of SC. Rather, a given SC neuron generally had to have both its receptive fields encompassing the exposure site in order to have access to the cross-modal experience.

Figure 8
In order to initiate multisensory integration maturation, both cross-modal component stimuli had to fall within their respective receptive fields

However, once a neuron acquired the ability to engage in multisensory integration as a consequence of cross-modal exposure at one location within its receptive fields, it appeared to generalize this capacity to other receptive field sites. This was the case in nearly all of the neurons under study (94%, 31/33). Two typical examples are provided in figure 9. In each case multisensory integration capabilities were acquired after cross-modal exposure at a single site (0° or 30°), but were also evident at a non-exposure test site, albeit, on average, with lower magnitude. The data for the population are shown in figure 10. Here it is apparent that in nearly all cases (91%, 30/33) the MSI was higher at the exposure than the non-exposure site. This was the case regardless of the location of the exposure site, and the group means at these sites were significantly larger than those evoked from their respective non-exposure sites (Paired t-test, P<0.001). This difference held regardless of whether the exposure site was macular and the non-exposure site was extra-macular (125.8 % ± 125.0 % versus 76.0 % ± 74.0%, Fig. 10A), or the reverse (126.1 % ± 103.9 % versus 90.1% ± 78.1%, Fig. 10B).

Figure 9
Once a neuron acquired multisensory integration at one receptive field location, it was generalized to other locations
Figure 10
MSI was higher at exposure than at non-exposure sites


Presumably, the normal period for multisensory development in SC neurons is protracted so that the brain can accommodate to the statistics of cross-modal events, then use this information to link the modalities to one another and organize the principles by which their information will be integrated. Normally this takes place very early in life (Stein et al., 1973; Wallace and Stein, 1997, , 2001, , 2007) when brain plasticity is believed to be greatest (Schoop et al., 1997; Katz, 1999; Berardi et al., 2000; Hensch, 2004; de Villers-Sidani et al., 2007; Galtrey and Fawcett, 2007). However, in the present experiments these cross-modal experiences were delayed until long after the capability for multisensory integration is normally acquired in SC neurons. That these neurons were still able to acquire this capability indicates that the brain retains the ability to use experiences with cross-modal cues to initiate its development far longer than previously suspected, an observation that is consonant with previous speculations (Wallace et al., 2004).

That cross-modal experiences were critical for the development of multisensory integration was apparent from its absence in dark-reared animals and the inability of equivalent exposure to the component stimuli, either individually or in interleaved fashion, to produce comparable results. An important factor was that both modality-specific components of the cross-modal stimulus be contained within a neuron’s receptive field. Only then would a neuron be directly privy to that cross-modal information. The development of multisensory integration appeared to involve several stages. On first appearance, the relative magnitude of integrated responses was far lower than is characteristic of normal animals, but gradually increased with continued cross-modal exposure. Although it appears likely that the continued accumulation of cross-modal experience was instrumental in this process, the possibility that, once initiated, multisensory integration capabilities can continue to develop in the absence of further experience cannot yet be excluded.

One can also infer that the acquisition of multisensory integration is initially closely linked to the physical properties of the stimuli, but is then generalized to other stimulus configurations. This inference is based on two observations. First, all the neurons that acquired multisensory integration capabilities did so at the receptive field position corresponding to the site of cross-modal exposure, but not all of these neurons did so at a non-exposure site. Second, those that did exhibit the capacity for multisensory integration at a non-exposure site exhibited lower magnitudes of response enhancement there than at the actual exposure site. Although these observations are consistent with the idea that a multisensory principle has been learned (i.e., the visual and auditory cues are linked at a given receptive field location) and then generalized, support for this concept would be strengthened considerably if future observations can demonstrate a temporal interval between these two presumptive stages, and/or show that changes in the multisensory responses to other, non-spatial, properties of the cross-modal stimulus exhibit a similar pattern.

It is particularly interesting to note that the cross-modal experiences that led to the development of multisensory integration in SC neurons were acquired during anesthesia and in the absence of any specific reward contingency. Apparently, exposure to the spatiotemporal coincidence of the stimuli was sufficient for them to become bound in the anesthetized brain so that they were then able to elicit an enhanced multisensory response in SC neurons. The experience in this case is unlike that usually referred to by this term, as it did not appear to involve higher order cognitive processes. Relatively simple Hebbian learning rules such as spike-timing dependent plasticity (Markram & Tsodyks, 1996; Gerstner et al., 1996; Fu et al., 2002; Dan & Poo 2004; Young et al., 2007; Dahmen et al., 2008) have been shown to be operative in other areas of the brain, and even brain slices, and are likely to be sufficient to encode the physical correlations between the two inputs in the anesthetized brain in a manner that would replicate the present observations (Yu et al., 2009). According to this principle, presynaptic (i.e., tectopetal) afferents whose activity repeatedly precedes postsynaptic spiking are strengthened. If operative in circumstances in which spatiotemporally concordant visual and auditory stimuli are presented, STDP should lead to the preferential strengthening of the afferents activated at approximately the same time, because they cooperate in activating the postsynaptic (SC) neuron. Consequently, the strongest inputs to a given SC neuron should be those that respond to visual and auditory stimuli that are approximately aligned in time and space.

The exceptional sensitivity of SC neurons to cross-modal stimulus exposure was reflected in the speed with which they used this experience to develop the capacity to integrate those inputs. Indeed, SC neurons accomplished this feat far more rapidly than would have been predicted from their normal developmental chronology. The first SC neuron capable of multisensory integration was encountered after several hours of visual-auditory experience rather than after many weeks as in the normally developing animal (Wallace and Stein, 1997). Furthermore, experience with this cross-modal stimulus configuration was sufficient to achieve a normal complement of integrating multisensory neurons. Presumably, this seemingly speeded acquisition reflects the high iterative rate and constancy of the cross-modal experiences that were provided here, as well as the presence of a neural substrate that had become mature enough to incorporate these experiences. Although it is assumed that the developmental principles observed here will generalize to other cross-modal stimulus combinations, at present only the visual-auditory combination has been studied.

That surprisingly little cross-modal exposure can have a substantial impact on the acquisition of multisensory integration capabilities, and that this high degree of experiential sensitivity is retained during early adulthood, suggest that multisensory integration is subject to continual modification throughout life. Although this possibility requires far more empirical support, it is consistent with a recent observation that in normal adult animals the responses of multisensory SC neurons to sequential visual and auditory stimuli change substantially after only minutes of exposure (Yu et al., 2009), that dark reared animals can manifest multisensory integration at the behavioral level after comparatively short training periods (Gingras et al., 2007), and that intense training with cross-modal stimuli can reverse hemineglect in cats (Jiang et al., 2008; McHaffie et al., 2008) and hemianopia in humans (Bolognini et al., 2005). It is also likely to be of substantial import in the context of the current rehabilitative efforts directed at inducing hearing or sight via cochlear or retinal prostheses or other intervention strategies such as the removal of congenital cataracts (Putzar et al., 2007). Although these strategies are generally directed specifically toward the compromised sensory modality, providing the experiences with cross-modal stimuli that would be necessary to develop an ability to use these senses in concert may require little additional effort.


We thank Nancy London for technical assistance. This work was supported by National Institutes of Health grant NS036916 and EY016716, as well as a grant from the Wallace Foundation.


  • Alvarado JC, Stanford TR, Vaughan JW, Stein BE. Cortex mediates multisensory but not unisensory integration in superior colliculus. J Neurosci. 2007a;27:12775–12786. [PubMed]
  • Alvarado JC, Vaughan JW, Stanford TR, Stein BE. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J Neurophysiol. 2007b;97:3193–3205. [PubMed]
  • Alvarado JC, Rowland BA, Stanford TR, Stein BE. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res 2008 [PMC free article] [PubMed]
  • Anooshian LJ, Bryan JM., Jr The effects of early auditory deprivation on temporal perceptions: a comparison of hearing and hearing-impaired children on temporal pattern matching tasks. J Speech Hear Res. 1979;22:731–746. [PubMed]
  • Berardi N, Pizzorusso T, Maffei L. Critical periods during sensory development. Curr Opin Neurobiol. 2000;10:138–145. [PubMed]
  • Bolognini N, Rasi F, Coccia M, Ladavas E. Visual search improvement in hemianopic patients after audio-visual stimulation. Brain. 2005;128:2830–2842. [PubMed]
  • Carlson S, Pertovaara A, Tanila H. Late effects of early binocular visual deprivation on the function of Brodmann’s area 7 of monkeys (Macaca arctoides) Brain Res. 1987;430:101–111. [PubMed]
  • Dahmen JC, Hartley DE, King AJ. Stimulus-timing-dependent plasticity of cortical frequency representation. J Neurosci. 2008;28:13629–13639. [PMC free article] [PubMed]
  • Dan Y, Poo MM. Spike timing-dependent plasticity of neural circuits. Neuron. 2004;44:23–30. [PubMed]
  • de Villers-Sidani E, Chang EF, Bao S, Merzenich MM. Critical period window for spectral tuning defined in the primary auditory cortex (A1) in the rat. J Neurosci. 2007;27:180–189. [PubMed]
  • Fu YX, Djupsund K, Gao H, Hayden B, Shen K, Dan Y. Temporal specificity in the cortical plasticity of visual space representation. Science. 2002;296:1999–2003. [PubMed]
  • Galtrey CM, Fawcett JW. The role of chondroitin sulfate proteoglycans in regeneration and plasticity in the central nervous system. Brain Res Rev. 2007;54:1–18. [PubMed]
  • Gerstner W, Kempter R, van Hemmen JL, Wagner H. A neuronal learning rule for sub-millisecond temporal coding. Nature. 1996;383:76–81. [PubMed]
  • Gingras G, Rowland BA, Stein BE. The impact of dark rearing on learing a lateralization task using stimuli of different modalities. Soc Neurosci Abst. 2007;33:614.1.
  • Hensch TK. Critical period regulation. Annu Rev Neurosci. 2004;27:549–579. [PubMed]
  • Hubel DH. Effects of deprivation on the visual cortex of cat and monkey. Harvey Lect. 1978;72:1–51. [PubMed]
  • Jiang H, Stein BE, McHaffie JG. Cross-modal rehabilitative training reinstates lost visual responsiveness to deep superior colliculus (SC) neurons. Soc Neurosci Abst. 2008;34 Program No. 457.6.
  • Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol. 2001;85:506–522. [PubMed]
  • Kadunce DC, Vaughan JW, Wallace MT, Stein BE. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp Brain Res. 2001;139:303–310. [PubMed]
  • Katz LC. What’s critical for the critical period in visual cortex? Cell. 1999;99:673–676. [PubMed]
  • Markram H, Tsodyks M. Redistribution of synaptic efficacy between pyramidal neurons. Nature. 1996;382:807–810. [PubMed]
  • McHaffie JG, Stein BE. A chronic headholder minimizing facial obstructions. Brain Res Bull. 1983;10:859–860. [PubMed]
  • McHaffie JG, Stein BE, Jiang H. Cross-modal rehabilitative training ameliorates visuomotor deficits in visual cortex lesioned cats. Soc Neurosci Abst. 2008;34 Program No. 457.8.
  • Meredith MA, Stein BE. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol. 1986;56:640–662. [PubMed]
  • Meredith MA, Nemitz JW, Stein BE. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J Neurosci. 1987;7:3215–3229. [PubMed]
  • Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT. Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol. 2003;90:4022–4026. [PubMed]
  • Putzar L, Goerendt I, Lange K, Rosler F, Roder B. Early visual deprivation impairs multisensory interactions in humans. Nat Neurosci. 2007;10:1243–1245. [PubMed]
  • Rowland BA, Stein BE. Multisensory integration produces an initial response enhancement. Front Integr Neurosci. 2007;1:4. [PMC free article] [PubMed]
  • Rowland BA, Stein BE. Temporal profiles of response enhancement in multisensory integration. Front Neurosci. 2008;2:218–224. [PMC free article] [PubMed]
  • Schoop VM, Gardziella S, Muller CM. Critical period-dependent reduction of the permissiveness of cat visual cortex tissue for neuronal adhesion and neurite growth. Eur J Neurosci. 1997;9:1911–1922. [PubMed]
  • Stanford TR, Quessy S, Stein BE. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci. 2005;25:6499–6508. [PMC free article] [PubMed]
  • Stein BE, Meredith MA. The merging of the senses. Cambridge, MA: MIT; 1993.
  • Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci. 2008;9:255–266. [PubMed]
  • Stein BE, Labos E, Kruger L. Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. J Neurophysiol. 1973;36:667–679. [PubMed]
  • Stein BE, Stanford TR, Ramachandran R, Perrault TJ, Jr, Rowland BA. Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res. 2009;198:113–126. [PMC free article] [PubMed]
  • Wallace MT, Stein BE. Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci. 1997;17:2429–2444. [PubMed]
  • Wallace MT, Stein BE. Sensory and multisensory responses in the newborn monkey superior colliculus. J Neurosci. 2001;21:8886–8894. [PubMed]
  • Wallace MT, Stein BE. Early experience determines how the senses will interact. J Neurophysiol. 2007;97:921–926. [PubMed]
  • Wallace MT, Perrault TJ, Jr, Hairston WD, Stein BE. Visual experience is necessary for the development of multisensory integration. J Neurosci. 2004;24:9580–9584. [PubMed]
  • Welsh LW, Welsh JJ, Healy MP. Early sound deprivation and long-term hearing. Ann Otol Rhinol Laryngol. 1996;105:877–881. [PubMed]
  • Young JM, Waleszczyk WJ, Wang C, Calford MB, Dreher B, Obermayer K. Cortical reorganization consistent with spike timing-but not correlation-dependent plasticity. Nat Neurosci. 2007;10:887–895. [PubMed]
  • Yu L, Stein BE, Rowland BA. Plasticity of multisensory neurons in adult superior colliculus: effects of repeated sequential visual and auditory stimuli. Soc Neurosci Abst. 2009:847.3.
  • Zahar Y, Reches A, Gutfreund Y. Multisensory enhancement in the optic tectum of the barn owl: spike count and spike timing. J Neurophysiol. 2009;101:2380–2394. [PubMed]