|Home | About | Journals | Submit | Contact Us | Français|
From birth, infants detect associations between the locations of static visual objects and sounds they emit, but there is limited evidence regarding their sensitivity to the dynamic equivalent when a sound-emitting object moves. In four experiments involving 36 2-month-olds, 48 5-month-olds and 48 8-month-olds, we investigated infants’ ability to process this form of spatial co-location. Whereas there was no evidence of spontaneous sensitivity, all age groups detected a dynamic co-location during habituation, and looked longer at test trials in which sound and sight were dislocated. Only 2-month-olds showed clear sensitivity to the dislocation relation, although 8-month-olds did so following additional habituation. These results are discussed relative to the intersensory redundancy hypothesis and work suggesting increasing specificity in processing with age.
Our experience of the world is based largely on multi-sensory information. For instance, when we manipulate objects we typically see and touch them simultaneously. Also, the sight of a person and the sound of his or her voice are co-located in space, something that also applies to sound-emitting objects in general. Further, when someone speaks, speech sounds correlate in an orderly way with facial movements. When objects or people move they typically produce a sound accompanying their movement and sound is produced when, for instance, a ball bounces on a surface or rolls across a hard floor. The ability to detect the matches and correlations existing in information from separate senses is thus a necessary condition for an integrated multisensory awareness of the world, and vital questions arise regarding the developmental origins of this ability.
Auditory-visual spatial co-location refers to the cross-modal association between the location of a sound and a visual event and, of course, depends on infants being able to localise sounds. Here there is some disagreement in the literature, with some work indicating a discontinuity or U-shaped developmental function, with auditory localisation hard to elicit at around 2 months (Clifton, Morrongiello, Kulig, & Dowd, 1981; Field, Muir, Pilon, Sinclair, & Dodwell, 1980; Muir, Clifton, & Clarkson, 1989), and other work suggesting a more or less linear increase in localisation ability with age (Morrongiello, 1988; Morrongiello, Fenwick, & Chance, 1990). It appears likely that this disagreement relates to differences in the dependent measure. When visual orienting to sound is measured there is a dip in responsiveness at 2 months that may reflect a change over in the neural system mediating the orienting response (Muir et al., 1989). However, a linear increase in localisation ability emerges when the response to sound position change does not involve orienting to it. This conclusion is in line with evidence from the spatial co-location literature, which suggests this ability is present at birth (Morrongiello, Fenwick, & Chance, 1998) and in older infants, including 2-month-olds (Morrongiello, Fenwick, & Nutley, 1998). We should note, however, that even at 6 to 7 months, accuracy of auditory localisation discrimination is approximately one tenth of adult resolution (Ashmead, Clifton, & Perris, 1987).
Spatial co-location is fundamental to everyday perception and so extending our knowledge of the conditions under which infants reveal sensitivity to auditory-visual spatial co-location should be a priority. A current view is that temporal synchrony between sound and sight is initially more salient than spatial co-location (Morrongiello, et al., 1998). However, spatial co-location is also a ubiquitous feature of intersensory information from the world; there are frequent cases in which objects produce sounds that are consistently co-located with their visual manifestation. Examples include people, who frequently talk while stationary or in motion and generally produce some sound of footfall when in motion, mechanical mobile toys within the home, and many forms of mechanised transport in the wider environment.
Many studies of spatial co-location (e.g. Fenwick & Morrongiello, 1998; Morrongiello et al., 1998) involve events at two fixed places: although the events themselves may be dynamic (for instance, the visual stimulus may move up and down to gain and maintain the infant’s attention), co-location of sound and sight occurs at static locations. However, there is some work that investigates detection of dynamic auditory-visual correspondences for movements in the near-far plane (Pickens, 1994; Walker-Andrews & Lennon, 1985). These studies indicate an ability to form such correspondences at 4 to 5 months. In these cases, however, the correspondence is not a direct spatial one, because auditory ‘distance’ is specified by sound intensity. Also, Pickens demonstrated that infants showed an association between changing sound amplitude and changing size of the visual stimulus in the absence of cues for movement in depth. This could mean that synaesthetic correspondence between visual size and sound amplitude explains part of the effect in these cases, a possibility made more plausible by evidence for synaesthetic correspondences at 4 months (Walker, Bremner, Mattock, Mason, Spring, Slater, & Johnson, 2010) as well as in toddlers (Maurer, Pathman, & Mondloch, 2006; Mondloch & Maurer, 2004).
It thus appears important to investigate dynamic auditory-visual co-location where visual and auditory locations are more directly specified. In this respect, lateral movement is a good candidate for investigation, because it is possible to provide veridical auditory information for changing location. Also there is evidence that infants are sensitive to a bounce illusion in which two objects that move smoothly through each other appear to bounce when a sound co-occurs with their fusion (Scheier, Lewkowicz, & Shimojo, 2003), which suggests that intermodal information is likely to be processed in the case of lateral movements. Surprisingly, however, to our knowledge there is no work that investigates dynamic co-location in lateral movements, though this should be relatively easy to investigate. Suppose, for example, infants are habituated to an event sequence in which a sounding object moves back and forth on a horizontal path. It is then possible to test for dishabituation when the object moves as usual but the sound is dislocated, so that, for instance, as the object moves left the sound moves right. If infants show recovery of looking, we can conclude that they have detected the invariant dynamic relation between locus of sight and sound, and note when this is violated. We can also investigate whether any such effect is limited to the case of co-location by habituating infants to a dislocation relation and testing for recovery of looking to the co-location relation. In addition to filling an important gap in the literature, studies of this sort carry the dual advantage of tapping into dynamic events that typically occur in the world (moving objects typically make a sound due to their movement).
It is not clear what one would predict regarding emergence of this dynamic form of spatial co-location. On the one hand, we might expect quite young infants to reveal this ability. We know that newborns detect spatial co-location in the case of static positions (Morrongiello et al., 1998), and presentation of dynamic information might, if anything, enhance this ability. On the other hand, adults are quite poor at detecting departure from dynamic spatio-temporal co-location of a moving object and a moving sound, there being a tendency to perceive a sound as moving with the visual object even when it is not (Soto-Faraco, Kingstone, Lyons, Gazzaniga, & Spence, 2002). If this tendency exists in infants it could act as a barrier to detection of dislocation between sight and sound.
The work reported here is a systematic investigation of circumstances under which infants detect violation of amodal auditory-visual relations in dynamic events involving sounding objects. We employ well-tested techniques used successfully to investigate object unity (Johnson, Bremner, Slater, & Mason, 2000) and trajectory perception (Bremner, et al., 2005; 2007) in infancy, and report four experiments that investigate the conditions under which infants detect changes in spatio-temporal co-location and dislocation between moving visual and auditory stimuli. In all four experiments, as the visual stimulus we use an image of a ball, moving on a horizontal trajectory, and as the auditory stimulus we use an attractive sound, stereophonically produced so as to create the impression (to adults) that it moved with or in the opposite direction to the object. Thus in co-located displays there was redundant dynamic co-location of sound and sight, but the relation between the nature of the sound and the nature of the object was arbitrary. The latter choice was made partly because piloting indicated that it was important to ensure that both the auditory and visual stimulus were salient, and more ‘realistic’ sounds, such as that of a ball rolling, did not appear to recruit attention. Additionally, however, many of the sound-emitting physical objects that infants encounter produce sounds that are arbitrarily related to their visual appearance; this is particularly true of infant toys. Thus, for both methodological and theoretical reasons we chose a sound that was completely arbitrary relative to the visual object.
Dynamic auditory-visual spatial co-location is an amodal intersensory relation involving redundant presentation of information across the senses, and there is good evidence that redundant presentation of this sort recruits infants’ attention and enhances learning (Bahrick, Flom, & Lickliter, 2002; Bahrick & Lickliter, 2000; Bahrick, Lickliter, & Flom, 2004). It should be noted that in all displays used in this series, there was also temporal redundancy consisting of common onset and offset, and hence duration of visual and auditory events. However, although stimulus onset generally happened when infants were fixating the screen, offset generally occurred when infants were looking away. Also, there were no discontinuities in auditory and visual stimuli during trials. Thus, only common onset information was liable to be consistently perceived by infants. However, given the argument that temporal synchrony is initially more salient than spatial co-location (Morrongiello et al., 1998), it is possible that common onset is salient enough to cue that sound and sight are linked. Thus, this information in itself may be sufficient to link auditory and visual information that is spatially dislocated.
In the experiments reported here we investigate dynamic auditory spatial relations in three ways. In Experiment 1 we investigate whether infants between 2 and 8 months of age demonstrate a spontaneous sensitivity to co-location and detect departures from this relation. In Experiment 2 we investigate infants’ response to departure from co-location following exposure to co-location during habituation trials. And in Experiments 3 and 4 we exposed infants to a dislocated relation between sight and sound in which the sound appears to move in the opposite direction to the visual object, and measured their post habituation response to a co-located event. Thus, in addition to investigating spontaneous sensitivity to dynamic co-location, this series investigated whether asymmetries existed in the degree to which infants could detect departures from co-location and dislocation relations following repeated exposure. Given that co-location is a good inter-modal cue to ‘objecthood’, we might expect greater sensitivity to co-location than dislocation.
Many experiments on cross-modal perception obtain clear effects with no prior habituation, and it is possible that dynamic spatial co-location is so salient that infants would detect its violation immediately with no prior familiarisation with or learning of the event. In other words, if we present infants with a series of test trials, half in which the sound and object move in co-location and half in which they move in dislocation, a strong predisposition towards co-location might be revealed in longer looking at one or other of the test trials. It is not clear which test trial we would expect to attract longer looking. On the one hand, work on detection of temporal synchrony demonstrates that infants look longer at the visual display that is in synchrony with the soundtrack (Spelke, 1981), which would lead us to expect that infants would look longer at the co-located test trials. On the other hand, the literature on core knowledge in infancy (reviewed in Spelke & Kinzler, 2007) provides numerous examples of infants’ tendency to look longer at events that violate their expectations. If departure from dynamic auditory-visual spatial co-location is perceived as a violation of normal reality, it is possible that infants would look longer at the display that violated their expectation. The important point is that a clear result in either direction would be informative, indicating sensitivity to dynamic co-location, in comparison to a null preference, which would provide no evidence of sensitivity to dynamic co-location. Thus, in our first experiment, we measured infants’ visual attention on four test trials, two of which presented sound and sight moving in co-location, and two of which presented sound and sight moving in dislocation.
Thirty-six infants took part, 12 2-month-olds (M = 62.92 days; range 54 – 71 days; 5 girls and 7 boys), 12 5-month-olds (M = 156.8 days; range 140 – 167 days; 6 girls and 6 boys), and 12 8-month-olds (M = 243.8 days; range 233 – 262 days; 6 girls and 6 boys). A further 6 infants were not included due to fussiness (2 2-month-olds and 3 8-month-olds) or technical problems (1 8-month-old). Throughout the series, infants took part in only one experiment. In all experiments, participants were recruited by personal contact with parents in the maternity unit when the baby was born, followed up by telephone contact near test age to those parents who volunteered to take part. Infants with reported health problems including visual and hearing deficits and those born two weeks or more before due date were omitted from the sample. The majority were from Caucasian, middle class families.
The room in which testing took place was partitioned with black curtaining, creating a cubicle measuring 3 metres square. In this way, the experimenter and the equipment, other than a plasma screen and camera, were out of view of the infant. The plasma screen created one of the ‘walls’ of the cubicle, and was blanked off to leave a visible screen 135cm wide by 17cm high, with the mid-point on the vertical dimension approximately at the infant’s eye level. A Macintosh G4 computer and the plasma screen were used to present stimuli and collect looking time data. A JVC, low-light, black and white camera was sited above the plasma screen, and was connected to a monitor that the observer used to record looking times. The observer’s judgments were input with a key press on the computer keyboard. Infants were recorded on videotape for later independent coding for inter-observer reliability measurement. The computer presented displays and recorded looking time judgments. Order and length of display presentation was managed using HABIT software (Cohen, Atkinson, & Chaput, 2004).
Animations were created using InfiniD software and Sound Edit files. The finished movies showed a green ball (6.7cm) going back and forth at a speed of 18 cm/sec across the screen (see Figure 1). The sound was presented stereophonically via two hidden speakers mounted at the ends of the visual display, creating the impression of a sound moving at the same speed as the ball. This was achieved through varying the balance from equal volume at each speaker (and hence equal intensity at both ears of the listener) smoothly to two extremes when sound only came from one or the other speaker. Given the placement of the speakers directly at the extremes of visual object motion, and assuming the use of inter-aural intensity difference to locate the sound, this provides objective co-location of visual and auditory information at the mid point and extremes, and the smooth alteration in balance may be assumed to maintain a smooth change between midpoint and extremes. On each trial both the ball and the sound originated at the center of the display and, depending on trial type, moved in the same or opposite directions. A full cycle (for example, center to left to right to center) took 15 seconds. The sound was a complex pulsating tone sequence that had been used in previous research as an attention-getter, so its suitability to attract and sustain attention had been verified. The object and sound cycled continuously with no breaks or static components for the duration of each trial. Background texture for the animations was provided by a light blue cloudy diffuse image that covered the entire visible screen.
Infants were given time to adjust to their new surroundings and during this time an explanation of the procedure was given to the parent and informed consent obtained. The parent was asked to sit, with the baby on his or her lap, on an adjustable wheeled office chair that could be positioned accurately so that the infant was one meter from the plasma screen. The parent was asked to remain quiet throughout, and to look either at the screen or at the top of the infant’s head. Given that, when questioned, most parents were unaware that the relation between sound and visual object varied across test trials, it is unlikely that they influenced infants’ looking. The need to place the chair precisely relative to the screen was stressed, and in this and subsequent experiments, no parent attempted to move the chair. Testing took place with the main room lights off.
Following initial acclimatisation, four test trial animations were presented in two blocks each consisting of a co-located event in which object and sound appeared to move together, and a dislocated event in which the sound and object travelled in the opposite directions. Thus, co-located and dislocated trials were presented in alternation. Order of presentation was counterbalanced. Prior to each trial, an attention-getter visual stimulus was presented to ensure infant fixation, and the test trial commenced as soon as fixation was achieved. Each trial continued either until 60 seconds had elapsed or the infant had looked away for 2 seconds.
A second observer used the recordings of infant gaze to second score the looking times for reliability purposes. The mean correlation between the scores of the two observers was .988 (SD = .018). Neither observer was aware of the hypothesis under investigation, or the display being presented at any point in the session. Although the primary observer could hear the sound, localisation was not possible due to the physical barriers presented by the equipment and screening.
Preliminary analyses revealed no reliable main effects or interactions concerning sex of participants, so data were collapsed across this variable in all experiments. Looking time data in many cells were positively skewed, violating assumptions of homogeneity of variance required by ANOVA; therefore scores were log-transformed prior to analysis in all the experiments (data in the figures are based on raw scores). ANOVAs based on raw scores revealed broadly the same patterns of significance. Figure 2 shows looking times to co-located versus dislocated test trials subdivided by age group and trial block. In all but the 8-month-olds’ second trial block there was a small preference for the dislocated test display. However, this was not a reliable effect, because a 3 (age group) × 2 (test trial order) × 2 (co-location: co-located vs. dislocated) × 2 (test trial block) mixed ANOVA revealed neither a significant effect of co-location, F (1, 30) = .33, p = .7, ηp2 = .005, nor a significant interaction between co-location and age, F (2, 30) = .19, p = .83, ηp2 = .01. The only effect to reach significance was trial block, F (1, 30) = 11.47, p = .002, ηp2 = .28, due to a general reduction in looking across trial blocks.
The lack of an effect of co-location was confirmed by the fact that 6/12 2- and 5-month-olds (binomial p = .61), and 7/12 8-month-olds (binomial p = .39) looked longer overall at the dislocated test display.
No evidence emerged in any age group of an affect of violation of dynamic spatial co-location in a version of the task that did not involve prior habituation to a co-located dynamic event. This could mean either that infants were incapable of processing departures from dynamic auditory-visual spatio-temporal co-location, or that this form of co-location is not so immediately salient that infants respond spontaneously to its violation. The lack of a trial effect cannot be attributed to a ceiling effect, because although looking times were relatively high, they were below ceiling, particularly on the second trial block.
The possibility that it takes some time to detect dynamic spatial co-location is very much in keeping with current theory of intersensory perception. According to the intersensory redundancy hypothesis (Bahrick, Flom, & Lickliter, 2002; Bahrick & Lickliter, 2000; Bahrick, Lickliter, & Flom, 2004), early in development, attention to and detection of amodal properties is enhanced when these properties are presented redundantly across two or more senses, as opposed to when they are presented non-redundantly to one sense. Thus, in the present case this hypothesis would predict that infants are likely to attend selectively to dynamic information for object movement after presentation of a series of trials in which this information is consistently presented redundantly across visual and auditory senses. In contrast, in Experiment 1, although all trials contained multimodal information, two trials provided redundant co-location information and two provided non-redundant information in the sense that auditory and visual information were dislocated.
Consequently, in Experiment 2 we repeated Experiment 1 with the difference that infants were first habituated to the co-located display prior to the same test trials as before, the aim of prior habituation being to ensure that infants were given extensive redundant presentation of information about object movement prior to test trials. In this case, our research question was whether, following exposure to a co-located dynamic auditory-visual event, infants would demonstrate detection of co-location by looking longer at a dislocation event during test trials.
Thirty-six infants took part, 12 2-month-olds (M = 70.75 days; range 55 – 82 days; 5 girls and 7 boys), 12 5-month-olds (M = 153.4 days; range 142 – 169 days; 6 girls and 6 boys), and 12 8-month-olds (M = 249.4 days; range 235 – 258 days; 6 girls and 6 boys). A further 7 infants were not included due to fussiness (two 2-month-olds and two 8-month-olds), failure to habituate (one 8-month-old), or technical problems (two 8-month-olds). Infants were recruited from the same population and through the same procedures as Experiment 1.
The same apparatus and displays were used as in Experiment 1. Prior to test trials, infants were habituated to the co-located test display; otherwise the procedure was identical to that of Experiment 1. Parents were told that part of the process of habituation would involve the baby getting bored, that they should not be concerned if the baby looked away from the screen, and that they should not try to direct the baby’s attention towards it. Progress to habituation was recorded through a sliding window measure that summed the looking times over the current and preceding three trials. Subject to a maximum of 12 trials, the habituation display was presented until looking time declined across four consecutive trials, from the second trial on, adding up to less than half the total looking time during the first four trials. Prior to each trial an attention-getting display was presented and as soon as the infant’s attention was on the display the trial commenced. A trial was terminated when the infant looked away for two seconds or 120 s had elapsed. The computer presented displays, recorded looking time judgments, calculated the habituation criterion for each infant, and changed displays after the criterion was met. Between trials, a beeping target was shown to attract attention back to the screen. There were two different animations used: one where the ball started in the middle of the screen and moved to the left, the other where the ball started in the middle of the screen and moved to the right. The order of these trials was randomly determined with no more than two the same presented consecutively. Once habituation trials ended, the four test trials followed immediately, with the same administration conditions as in Experiment 1. Infants in this and subsequent experiments who did not habituate to criterion within 12 trials proceeded to test trials, but their data were excluded.
Figure 3 shows mean looking times at the two types of test trial subdivided by age group, and trial block (mean looking time for the final habituation trial is included for comparison). For all ages, this indicates a tendency to look longer at the dislocated (unexpected) test trial, a tendency that is stronger on the second block of trials. A 3 (age group) × 2 (test trial order) × 2 (co-location) × 2 (test trial block) mixed ANOVA yielded a significant main effect of co-location, F (1, 30) = 17.43, p = .001, ηp2 = .37. There was also a significant interaction between co-location and test trial block, F (1, 30) = 8.5, p = .007, ηp2 = .22. On the first test trial block the effect of co-location was not quite significant, F (1,30) = 3.34, p = .078, ηp2 = .1, whereas on the second block it was highly significant, F (1,30) = 27.38, p = .001, ηp2 = .48. This interaction was qualified by a significant four-way interaction between co-location, test trial block, test trial order, and age group, F (2, 30) = 5.46, p = .009, ηp2 = .27. To investigate this, further analyses were carried out for each age group separately. For 2-month-olds, there was a significant interaction between co-location, test trial block, and test trial order, F (1, 10) = 7.88, p = .019, ηp2 = .44. Further analysis indicated that on the first test trial block there was a significant interaction between co-location and test trial order, F (1, 10) = 6.62, p =. 027, ηp2 = .4, that arose from infants looking longer at the test trial presented second, whereas on the second test trial block infants looked significantly longer at the dislocated test display, F (1, 10) = 8.52, p = .015, ηp2 = .46. For 5-month-olds, there was a significant effect of co-location, F (1, 10) = 14.48, p = .003, ηp2 = .59, qualified by a significant interaction between co-location and test trial block, F (1, 10) = 6.44, p = .03, ηp2 = .39. On the first block, the effect of co-location was not significant, F (1, 10) = 2.83, p = .12, ηp2 = .22, whereas on the second block it was highly significant, F (1, 10) = 43.4, p = .001, ηp2 = .81. Thus, for both of the younger groups, the co-location effect only emerged by the second test trial block. For 8-month-olds, there was a significant effect of co-location, F (1, 10) = 6.44, p = .029, ηp2 = .39, but the interaction between co-location and test trial block was not significant, F (1, 10) = 3.34, p = .097, ηp2 = .25. Thus, at this age the co-location effect was evident in both test trial blocks.
The same general pattern emerges from analysis of the number of infants looking longer at the dislocated test display. On the first block of trials, 7/12 2-month-olds (binomial p = .39), 8/12 5-month-olds (p = .19) and 7/12 8-month-olds looked longer at the dislocated trial, whereas on the second block, 10/12 2-month-olds (p = .02), 12/12 5-month-olds (p = .001) and 11/12 8-month-olds (p = .006) did so.
Finally, a comparison of mean looking time data with Experiment 1 revealed a significant interaction between co-location and experiment, F (1,60) = 9.39, p = .003, ηp2 = .23, reflecting an effect of co-location in Experiment 2 but not in Experiment 1.
By the second block of test trials all age groups showed a significant preference for the dislocated test display, and in the case of the 8-month-olds, this was evident on the first test trial block as well. Thus, it appears that from 2 months infants are capable of detecting a dynamic auditory-visual spatial co-location relation during habituation trials and respond to its violation on test trials. For the 2- and 5-month-olds, the lack of a significant effect on the first test trial block may be evidence that the effect of violating dynamic co-location is relatively subtle, such that younger infants take some trials to detect it clearly. And there is reason to expect that the effect would be relatively subtle. As indicated in the introduction, adults have a tendency to perceive a sound as moving with the visual object even when it is not (Soto-Faraco et al., 2002), and although the same effect has not been investigated in infants, 5-month-olds are subject to visual dominance in syllable detection (the McGurk effect), in some cases giving priority to visual information in perceiving the syllable presented (Rosenblum, Schmuckler, & Johnson, 1997). Thus, it is possible that infants are subject to the same effect as adults, though only to the extent of partially suppressing sensitivity to auditory-visual dislocation.
In contrast to the null results obtained in Experiment 1, infants who were first habituated to the co-located display, and who thus had repeated experience of redundant presentation of information about object movement, showed sensitivity to violation of auditory-visual matching of this information (dynamic spatial co-location). According to the intersensory redundancy hypothesis, redundant presentation of information enhances attention to the stimulus property concerned, and on this basis we would expect this form of presentation during habituation trials would be conducive to detecting an auditory visual relation. However, it is possible that the more restricted redundant information presented by common onset is sufficient to support detection of a relation even if location information is presented non-redundantly between the senses. If this is the case, infants may reveal similar effects when the task is ‘run backwards’. Specifically, following habituation to the dislocated display, infants may show a looking preference for the co-located display, the opposite result from that obtained in Experiment 2. Although this form of habituation seems likely to present less favourable conditions for forming a cross-modal association, the change from non-redundant to redundant presentation of location information on co-located tests trials may in itself cue attention. Thus, in Experiment 3 we repeated Experiment 2 with the difference that, prior to test trials, infants were first habituated to the dislocated display. In this case, our research question was whether, following exposure to a dislocated dynamic auditory-visual event, infants would demonstrate detection of this relation by looking longer at a co-located event during test trials.
Thirty-six infants took part, 12 2-month-olds (M = 62.2 days; range 56 – 67 days; 5 girls and 7 boys), 12 5-month-olds (M = 159.6 days; range 142 – 167 days; 7 girls and 5 boys), and 12 8-month-olds (M = 246.8 days; range 239 – 253 days; 5 girls and 7 boys). A further 8 infants (four 2-month-olds, two 5-month-olds and two 8-month-olds) were not included due to fussiness. Infants were recruited from the same population and through the same procedures as in previous experiments.
The same apparatus and displays were used as in Experiments 1 and 2. The same procedure was followed as in Experiment 2, except that infants were habituated to the dislocated display rather than the co-located display prior to test trials.
Figure 4 shows mean looking times at the two types of test trial subdivided by age group, and trial block (mean looking time for the final habituation trial is included for comparison). Although there is a preference for the co-located test trial at 2 and 5 months it is very small at 5 months and the 8-month-olds show a slight preference in the opposite direction. A 3 (age group) × 2 (test trial order) × 2 (co-location) × 2 (test trial block) mixed ANOVA revealed neither a significant effect of co-location, F (1, 30) = 2.14, p = .154, ηp2 = .15, nor a significant interaction between co-location and age group, F (2, 30) = 1.47, p = .247, ηp2 = .09. There was a significant effect of age group, F (2, 30) = 6.09, p = .006, ηp2 = .29, with 2-month-olds looking significantly longer than 5-month-olds, Newman-Keuls p = .005, and 8-month-olds, Newman-Keuls p = .02. There was also a significant interaction between test trial order and age group, F (2, 30) = 4.57, p = .018, ηp2 = .234, qualified by a significant interaction between co-location, test trial block, test trial order, and age group, F (2, 30) = 4.66, p = .017, ηp2 = .237. Further investigation indicated that these effects were explained by the 8-month-olds’ data, which revealed a significant effect of test trial order, F (1, 10) = 6.25, p = .031, ηp2 = .38, qualified by a significant interaction between co-location, test trial block and test trial order, F (1, 10) = 6.04, p = .034, ηp2 = .38. The effect of test trial order was due to longer looking at both test trials when the dislocated trial was presented first, and the interaction reflects the fact that the exception to this is the first dislocated test trial, for which looking is unaffected by test trial order. There is no clear explanation for either of these effects. No significant main effects or interactions involving these factors were obtained in other age groups.
The lack of an effect of co-location is confirmed by the fact that 8/12 2-month-olds and 5-month-olds (binomial p = .19) and 7/12 8-month-olds (p = .39) looked longer overall at the co-located test display. However, 2-month-olds showed longer looking to the co-located test trial on both blocks and the magnitude of this difference was not much smaller than the opposite effect in experiment 2 for this age group. Although at 2 months the effect of co-location was not significant, F (1,10) = 2.73, p = .13, ηp2 = .21, closer inspection of the individual data revealed that two 2-month-olds showed extremely long looking times during habituation and went on to look longer at the dislocated test trials. Thus we replicated Experiment 3 with a new group of 2-month-olds (N = 12, M 65.1 days , range 54-74 days). An additional 4 infants did not complete testing, 2 due to fussiness and 2 due to sleepiness. The results are displayed in Figure 4. This time, there was substantially longer looking at the co-location test trial and analysis yielded a significant effect of co-location, F (1,10) = 24.4, p = 001, ηp2 = .71, and no other significant effects or interactions.
Finally, in order to make a direct comparison of the results of Experiments 2 and 3, we carried out a 2 (experiment) × 3 (age Group) × 2 (co-location) mixed ANOVA on the dataset for both experiments combined (using the 2-month-old replication data, and defining test trial novelty relative to the habituation display infants were exposed to). This yielded a significant effect of co-location, F (1,60) = 28.47, p = .001, ηp2 = .32, qualified by a significant interaction between co-location, experiment, and age, F (2,60) = 5.36, p = .007, ηp2 = .15. To investigate this interaction, separate analyses were carried out for each age group. For 2-month-olds, there was a significant effect of co-location, F (1,20) = 22.14, p = .001, ηp2 = .52, and no significant interaction between experiment and co-location, F (1,20) = 2.83, p = .11, ηp2 = .12, indicating that infants showed recovery of looking to the novel test trial whether they had been habituated to co-location or dislocation. For 5-month-olds, there was also a significant effect of co-location, F (1,20) = 8.52, p = .008, ηp2 = .29, and no significant interaction between experiment and co-location, F (1,20) = 1.47, p = .24. ηp2 = .07. Thus, for this age group, although there was no significant co-location effect in Experiment 3, the trend was not significantly different from the significant effect in Experiment 2. For 8-month-olds, there was a near significant effect of co-location, F (1,20) = 3.88, p = .063, ηp2 = .16, and a significant interaction between experiment and co-location, F (1,20) = 5.83, p = .025, ηp2 = .23, due to a significant preference for the dislocation display in Experiment 2 following habituation to the co-location display, F (1,10) = 6.44, p = .029, ηp2 = .39, but no significant preference for either display in Experiment 3 following habituation to the dislocation display, F (1,10) = .19, p = .67, ηp2 = .02.
The replication with 2-month-olds yielded a positive result, this age group showing a novelty preference for dynamic co-location following exposure to dynamic dislocation. In contrast, there was no evidence of such an effect at 5 or 8 months. Note, however, that the trend with 5-month-olds was in the same direction as the effect with 2-month-olds, and did not differ significantly from the significant effect with this age group in Experiment 2. Because there was auditory-visual dislocation during habituation, the only redundant information likely to have been detected was common onset, and presentation of movement information was explicitly incongruent in presenting different information to the two senses. Thus, given evidence that infants did not learn intersensory relations when information was congruent at one level but incongruent at another (Bahrick, 1988), these trials should have constituted relatively unfavourable conditions for intersensory learning. The result for 2-month-olds is striking and suggests the presence of an early ability that functions even when the level of redundancy across the senses is low. As already noted, longer looking to co-located test trials following dislocated habituation may simply indicate that infants have registered non-redundant presentation of auditory and visual information during habituation and have detected the shift to redundant presentation. In contrast, the lack of an effect with older infants, particularly 8-month-olds, suggests that infants’ accumulating experience of events in which auditory-visual spatial co-location is the norm, leads infants to expect co-location, making it difficult to elicit novelty preferences for co-located events and possibly reducing infants’ ability to detect dislocation events. Other evidence of the constraining effects of experience can be seen in Morrongiello, Fenwick and Nutley’s (1998) finding that, when sound and object were spatially dislocated, 2- to 4-month-olds formed sound-object associations on the basis of temporal synchrony, but 6- and 8-month-olds did not. And at a more general level this effect is similar to that detected in other domains such as perception of speech sounds (Werker & Tees, 1984) and, later in infancy, perception of animacy (Rakison & Poulin-Dubois, 2001), form-function relations (Madole & Cohen, 1995), and symbolic reference (Namy & Waxman, 1998), in which an initial general ability becomes increasingly constrained through experience.
Our final question is whether older infants might show a novelty preference for a co-located event if they had more prior experience of a dislocated event. As already noted, accumulated everyday experience of co-location in older infants may result in a reduced tendency to treat a co-location event as novel following experience of a dislocation event. However, it is possible that provision of more extensive experience of a dislocation event may counteract this experience-based constraint. Thus, in Experiment 4, we repeated Experiment 3 with 5- and 8-month-olds, providing more habituation experience, to reveal whether following this additional experience older infants would show a novelty preference for the co-located test trial. .
Twenty-four infants took part, 12 5-month-olds (M = 152.8 days; range 141 – 168 days; 6 girls and 6 boys), and 12 8-month-olds (M = 250.7 days; range 241 – 261 days; 5 girls and 7 boys). A further 3 infants (one 5-month-old and two 8-month-olds) were not included due to fussiness. Infants were recruited from the same population and through the same procedures as in previous experiments.
The same apparatus and displays were used as in Experiments 1 to 3. The same procedure was followed as in Experiment 3, with the exception of a modification in the habituation procedure to ensure fuller habituation. In contrast to the sliding window method used in Experiments 2 and 3 in which the total of four trials was computed for every block of four from trial 2 to 5 onwards, infants were habituated according to a fixed window method in which the total was only sampled across trials 5 to 8, and if the criterion was not met, over trials 9 to 12. As previously, this total was compared to the total looking time across the first four trials, and the habituation criterion was met when total looking across a block of four trials fell to half the total across the first four trials. This meant that, in contrast to the sliding window criterion in which habituation is possible from trial 5 onwards, habituation trials could end only after 8 or 12 trials.
First, to check that infants in Experiment 4 did indeed undergo lengthier habituation than 5-and 8-month-olds in Experiment 3, we compared number of habituation trials and accumulated habituation times between the two experiments. In the case of number of habituation trials, a 2 (experiment) × 2 (age group) ANOVA yielded a significant main effect of experiment, F (1, 44) = 19.12, p = .001, ηp2 = .3, with more trials to habituate in Experiment 4 than in Experiment 3, M = 10.00 (SD = 2.04) vs. M = 7.41 (SD = 2.22). There was also a significant main effect of age group, F (1, 44) = 4.48, p = .04, ηp2 = .09, 8-month-olds having more habituation trials than 5-month-olds, M = 9.33 (SD = 2.78) vs. M = 8.08 (SD = 2.02). The interaction between experiment and age group was not significant, F (1,44) = 1.61, p = .21, ηp2 = .03. In the case of habituation time, a 2 (experiment) × 2 (age group) ANOVA yielded a significant main effect of experiment, F (1, 44) = 4.29, p = .044, ηp2 = .09, habituation times being longer in Experiment 4 than in Experiment 3, M = 229.24 s (SD = 147.58) vs. M = 160.59 s (SD = 72.79). There was no significant effect of age group, F (1, 44) = 1.26, p = .27, ηp2 = .03, or significant interaction between experiment and age group, F (1,44) = 1.96, p = .17, ηp2 = .04.
Figure 5 shows mean looking times at the two types of test trial subdivided by age group, and trial block (mean looking time for the final habituation trial is included for comparison). The 5-month-olds showed no consistent pattern across test trial blocks, whereas the 8-month-olds showed a looking preference for the co-located test trials on both blocks. A 2 (age group) × 2 (test trial order) × 2 (co-location) × 2 (test trial block) mixed ANOVA revealed a significant effect of co-location, F (1, 20) = 7.15, p = .017, ηp2 = .31, qualified by a significant interaction between co-location and age group, F (1, 20) = 6.61, p = .02, ηp2 = .29. Although 5-month-olds showed no looking preference for either test trial, F (1, 10) = .01, p = .93, ηp2 = .001, 8-month-olds showed a significant looking preference for the co-located test display, F = (1,10) = 9.48, p = .015, ηp2 = .54.
This age related pattern is confirmed by the fact that 6/12 5-month-olds (p = .61) and 10/12 8-month-olds (p = .02) showed an overall looking preference for the co-located test display.
Although providing more habituation experience had no effect on 5-month-olds’ performance, it led to a significant effect for 8-month-olds, indicating that with additional experience of an event that violated dynamic spatial co-location, this age group showed the reverse of the effect obtained in Experiment 2. Thus, 8-month-olds were able to detect when an event changed from a dislocation to a co-location relation. It is possible that this is due to increased experience of dislocation counteracting older infants’ expectation of co-location. Also, the fact that in both experiments 3 and 4, 8-month-olds took more trials to habituate than 5-month-olds may reflect the increased novelty of a dislocation event relative to accumulated experience of co-location. However, the fact that only 8-month-olds benefited from additional dislocation experience suggests an alternative interpretation of their longer looking during habituation, namely that this age group was actively attempting to process an intersensory relation that was novel relative to their accumulated experience.
The main aim of this series of studies was to establish the conditions under which infants are capable of detecting dynamic auditory-visual spatial co-location and responding to events in which co-location is abolished. In Experiment 1, in which no habituation trials were administered prior to test, infants showed no differences in looking at co-located and dislocated test trials, indicating that in none of the age groups tested was there evidence of a spontaneous expectation that sight and sound should move together. It is possible, however, that dynamic spatial co-location is rather too subtle to be picked up immediately; the fact that few adults noted the change from co-located to dislocated trials is in keeping with this. In Experiment 2, following habituation to the co-located event, infants at all ages showed a looking preference for the dislocated test event, though for the younger groups this was only firmly established by the second block of test trials. This is clear indication that, from as early as 2 months of age, infants are capable of detecting the dynamic association between a horizontally moving object and a similarly moving sound. To our knowledge, this is the first demonstration of this ability, other than in cases of movements in the depth plane in which sound position was represented by sound amplitude. Unlike the case of sound amplitude, which varies for a number of reasons other than sound distance, and which may bear a synaesthetic relation to visual stimulus properties (Walker et al., 2010), stereophonic presentation of sound in the horizontal plane is unambiguous information about sound location.
Although this work demonstrates that even very young infants are capable of processing dynamic information for spatial co-location, the work is silent regarding the processes underlying this ability. In principle it is possible that infants sample instances from the display and detect co-location or dislocation in these ‘snapshots’. However, it must be noted that in our experiments neither the auditory nor the visual stimulus was ever static. Additionally, such a process is very much the antithesis of arguments emerging from ecological theory regarding the dynamic nature of perception (Gibson, 1979), and it seems likely that we are tapping into a truly dynamic process.
In Experiment 3, following habituation to the dislocated event, only 2-month-olds looked significantly longer at the co-located test display, although in Experiment 4, following more thorough habituation to the dislocated event, 8-month-olds (but not 5-month-olds) showed a looking preference for the co-located test event. Spatial dislocation specifies lack of co-location between sound and sight, whereas common onset specifies a relation, and so this event presents contradictory information regarding the sight-sound relation, There is evidence that young infants do not detect changes in rhythm in mechanical events (Bahrick & Lickliter, 2000) and affect in social stimuli (Flom & Bahrick, 2007) when the auditory and visual components are presented out of synchrony, so the outcome with 2-month-olds is striking. It is possible that infants simply note the shift from non-redundant to redundant presentation of location information. In other words, the onset of redundancy cues attention. However, if this was the case, why did 5-month-olds infants not show a similar pattern of response, and 8-month-olds only after fuller exposure to the dislocation event? A possible interpretation is that, in common with developmental changes in other domains such as speech perception and perception of animacy, experience leads infants to expect co-location events and to treat dislocation events as unexpected. Thus it becomes increasingly difficult to obtain novelty preferences for co-location events following exposure to dislocation (Experiments 3 & 4) and increasingly easy to obtain novelty preferences for dislocation events following exposure to collocation events (Experiment 2).
On the other hand, the fact that the additional habituation in Experiment 4 was sufficient for 8-month-olds to show a novelty preference to a co-location event may be evidence of developing flexibility in perceptual learning that, at this age, begins to counteract experience-based perceptual selectivity. Although dislocation events involving single objects do not produce information for objecthood, infants are bombarded with events in which new arbitrary intersensory relations are encountered. For example, in the case of speech perception, although older infants become insensitive to speech sounds that they are not exposed to, this process is reversible when they are exposed to a new language, and there are advantages to being able to ‘re-learn’ these sounds after relatively short exposure. The same perceptual flexibility that supported 8-month-olds’ performance in Experiment 4 is likely to serve them well in processing new intersensory relations.
We believe that, taken together, the results of this series of studies add to our understanding of the development of intersensory perception by providing evidence regarding the conditions under which infants are sensitive to dynamic spatial co-location and dislocation. Dynamic spatial co-location is ubiquitous in the natural world, but previous work investigating this phenomenon has focused on the case of movement in depth with auditory ‘distance’ mediated by sound intensity. In contrast, our work specifies auditory position more directly through stereophonic presentation that translates into inter-aural intensity difference, a primary cue to auditory localization. Our finding that 2-month-olds apparently detected a change from dislocation to co-location under circumstances in which older infants did not raises the likelihood that future theorising and research will need to pay more attention to the ways in which, during development, early intersensory perception changes due to accumulated experience. Also, although this must currently be a very tentative suggestion, our finding that additional experience of dislocation events led only the oldest infants to treat co-location as novel may indicate a later transition as infants begin to break out of the constraints of experience to learn new intersensory relations such as those arbitrary relations existing between objects and their names.
This work was supported by Economic & Social Research Council Grant R000239979 and NIH Grants HD40432 and HD 48733. The authors acknowledge the assistance of staff of University Hospitals of Morecambe Bay NHS Trust in recruitment, and are grateful to parents and infants who took part in the work. The authors are also grateful to the editor and three anonymous reviewers for their useful contribution to improving the interpretation of the work.
J. Gavin Bremner, Lancaster University.
Alan M. Slater, University of Exeter.
Scott P. Johnson, University of California, Los Angeles.
Uschi C. Mason, Lancaster University.
Jo Spring, Lancaster University.
Maggie E. Bremner, Lancaster University.