Viewing hand gestures during face-to-face communication affects speech perception and comprehension. Despite the visible role played by gesture in social interactions, relatively little is known about how the brain integrates hand gestures with co-occurring speech. Here we used functional magnetic resonance imaging (fMRI) and an ecologically valid paradigm to investigate how beat gesture – a fundamental type of hand gesture that marks speech prosody – might impact speech perception at the neural level. Subjects underwent fMRI while listening to spontaneously-produced speech accompanied by beat gesture, nonsense hand movement, or a still body; as additional control conditions, subjects also viewed beat gesture, nonsense hand movement, or a still body all presented without speech. Validating behavioral evidence that gesture affects speech perception, bilateral nonprimary auditory cortex showed greater activity when speech was accompanied by beat gesture than when speech was presented alone. Further, the left superior temporal gyrus/sulcus showed stronger activity when speech was accompanied by beat gesture than when speech was accompanied by nonsense hand movement. Finally, the right planum temporale was identified as a putative multisensory integration site for beat gesture and speech (i.e., here activity in response to speech accompanied by beat gesture was greater than the summed responses to speech alone and beat gesture alone), indicating that this area may be pivotally involved in synthesizing the rhythmic aspects of both speech and gesture. Taken together, these findings suggest a common neural substrate for processing speech and gesture, likely reflecting their joint communicative role in social interactions.
gestures; speech perception; auditory cortex; magnetic resonance imaging; nonverbal communication
Although stuttering is regarded as a speech-specific disorder, there is a growing body of evidence suggesting that subtle abnormalities in the motor planning and execution of non-speech gestures exist in stuttering individuals. We hypothesized that people who stutter (PWS) would differ from fluent controls in their neural responses during motor planning and execution of both speech and non-speech gestures that had auditory targets. Using fMRI with sparse sampling, separate BOLD responses were measured for perception, planning, and fluent production of speech and non-speech vocal tract gestures. During both speech and non-speech perception and planning, PWS had less activation in the frontal and temporoparietal regions relative to controls. During speech and non-speech production, PWS had less activation than the controls in the left superior temporal gyrus (STG) and the left pre-motor areas (BA 6) but greater activation in the right STG, bilateral Heschl’s gyrus (HG), insula, putamen, and precentral motor regions (BA 4). Differences in brain activation patterns between PWS and controls were greatest in the females and less apparent in males. In conclusion, similar differences in PWS from the controls were found during speech and non-speech; during perception and planning they had reduced activation while during production they had increased activity in the auditory area on the right and decreased activation in the left sensorimotor regions. These results demonstrated that neural activation differences in PWS are not speech-specific.
Stuttering; Speech perception; planning; production; non-speech; functional magnetic resonance imaging (fMRI); auditory-motor interaction; forward model
This study examined the communicative profiles of children with autism spectrum disorders (ASD) in the second year of life.
Communicative acts were examined in 125 children 18 to 24 months of age: 50 later diagnosed with ASD; 25 with developmental delays (DD); and 50 with typical development (TD). Precise measures of rate, functions, and means of communication were obtained through systematic observation of videotaped Behavior Samples from the Communication and Symbolic Behavior Scales Developmental Profile (Wetherby & Prizant, 2002).
Children with ASD communicated at a significantly lower rate than children with DD and TD. The ASD group used a significantly lower proportion of acts for joint attention and a significantly lower proportion of deictic gestures with a reliance on more primitive gestures compared to DD and TD. Children with ASD who did communicate for joint attention were as likely as other children to coordinate vocalizations, eye gaze, and gestures. Rate of communicative acts and joint attention were the strongest predictors of verbal outcome at age 3.
By 18 to 24 months of age, children later diagnosed with ASD showed a unique profile of communication, with core deficits in communication rate, joint attention, and communicative gestures.
► The emergence of first gestures in girls with RTT is not necessarily delayed. ► The repertoire of communicative gestures, however, is restricted. ► Although girls with RTT have difficulties in their verbal communicative domain, gestures do not constitute a compensatory mechanism. ► A limited repertoire of gestures and qualitative peculiarities in other speech-language domains might be characteristic for a severe neurodevelopmental disorder like RTT.
We studied the gestures used by children with classic Rett syndrome (RTT) to provide evidence as to how this essential aspect of communicative functions develops. Seven participants with RTT were longitudinally observed between 9 and 18 months of life. The gestures used by these participants were transcribed and coded from a retrospective analysis of a video footage. Gestures were classified as deictic gestures, play schemes, and representational gestures. Results of the analysis showed that the majority of gestures observed were of deictic character. There were no gestures that could be classified as play schemes and only two (e.g., head nodding and waving bye bye) that were coded as representational or symbolic gestures. The overall repertoire of gestures, even though not necessarily delayed in it's onset, was characterized by little variability and a restricted pragmatic functionality. We conclude that the gestural abilities in girls with RTT appear to remain limited and do not constitute a compensatory mechanism for the verbal language modality.
Communication; Gesture; Interaction; Language; Language impairment; Pointing; Rett; Speech; Video analysis
Everyday communication is accompanied by visual information from several sources, including co-speech gestures, which provide semantic information listeners use to help disambiguate the speaker’s message. Using fMRI, we examined how gestures influence neural activity in brain regions associated with processing semantic information. The BOLD response was recorded while participants listened to stories under three audiovisual conditions and one auditory-only (speech alone) condition. In the first audiovisual condition, the storyteller produced gestures that naturally accompany speech. In the second, she made semantically unrelated hand movements. In the third, she kept her hands still. In addition to inferior parietal and posterior superior and middle temporal regions, bilateral posterior superior temporal sulcus and left anterior inferior frontal gyrus responded more strongly to speech when it was further accompanied by gesture, regardless of the semantic relation to speech. However, the right inferior frontal gyrus was sensitive to the semantic import of the hand movements, demonstrating more activity when hand movements were semantically unrelated to the accompanying speech. These findings show that perceiving hand movements during speech modulates the distributed pattern of neural activation involved in both biological motion perception and discourse comprehension, suggesting listeners attempt to find meaning, not only in the words speakers produce, but also in the hand movements that accompany speech.
discourse comprehension; fMRI; gestures; semantic processing; inferior frontal gyrus
Abstractness and modality of interpersonal communication have a considerable impact on comprehension. They are relevant for determining thoughts and constituting internal models of the environment. Whereas concrete object-related information can be represented in mind irrespective of language, abstract concepts require a representation in speech. Consequently, modality-independent processing of abstract information can be expected. Here we investigated the neural correlates of abstractness (abstract vs. concrete) and modality (speech vs. gestures), to identify an abstractness-specific supramodal neural network. During fMRI data acquisition 20 participants were presented with videos of an actor either speaking sentences with an abstract-social [AS] or concrete-object-related content [CS], or performing meaningful abstract-social emblematic [AG] or concrete-object-related tool-use gestures [CG]. Gestures were accompanied by a foreign language to increase the comparability between conditions and to frame the communication context of the gesture videos. Participants performed a content judgment task referring to the person vs. object-relatedness of the utterances. The behavioral data suggest a comparable comprehension of contents communicated by speech or gesture. Furthermore, we found common neural processing for abstract information independent of modality (AS > CS ∩ AG > CG) in a left hemispheric network including the left inferior frontal gyrus (IFG), temporal pole, and medial frontal cortex. Modality specific activations were found in bilateral occipital, parietal, and temporal as well as right inferior frontal brain regions for gesture (G > S) and in left anterior temporal regions and the left angular gyrus for the processing of speech semantics (S > G). These data support the idea that abstract concepts are represented in a supramodal manner. Consequently, gestures referring to abstract concepts are processed in a predominantly left hemispheric language related neural network.
gesture; speech; fMRI; abstract semantics; emblematic gestures; tool-use gestures
In a natural setting, speech is often accompanied by gestures. As language, speech-accompanying iconic gestures to some extent convey semantic information. However, if comprehension of the information contained in both the auditory and visual modality depends on same or different brain-networks is quite unknown. In this fMRI study, we aimed at identifying the cortical areas engaged in supramodal processing of semantic information. BOLD changes were recorded in 18 healthy right-handed male subjects watching video clips showing an actor who either performed speech (S, acoustic) or gestures (G, visual) in more (+) or less (−) meaningful varieties. In the experimental conditions familiar speech or isolated iconic gestures were presented; during the visual control condition the volunteers watched meaningless gestures (G−), while during the acoustic control condition a foreign language was presented (S−). The conjunction of the visual and acoustic semantic processing revealed activations extending from the left inferior frontal gyrus to the precentral gyrus, and included bilateral posterior temporal regions. We conclude that proclaiming this frontotemporal network the brain's core language system is to take too narrow a view. Our results rather indicate that these regions constitute a supramodal semantic processing network.
From very early in life, expressive behavior is multimodal, with early behavioral coordinations being refined and strengthened over time as they become used for the communication of meaning. Of these communicative coordinations, those that involve gesture and speech have received perhaps the greatest empirical attention, but little is known about the developmental origins of the gesture-speech link. One possibility is that the origins of speech-gesture coordinations lie in hand-mouth linkages that are observed in the everyday sensorimotor activity of very young infants who do not yet use the hand or mouth to communicate meaning. In this article, I review evidence suggesting that the study of gesture-speech links and developmentally prior couplings between the vocal and motor systems in infancy can provide valuable insight into a number of later developments that reflect the cognitive interdependence of gesture and speech. These include aspects of language development and delay, the infant origins of the adult speech-gesture system, and early signs of autism spectrum disorder. Implications of these findings for studying the development of multimodal communication are considered.
gesture; language development; vocalization; motor development
Given that the linguistic articulators for sign language are also used to produce co-speech gesture, we examined whether one year of academic instruction in American Sign Language (ASL) impacts the rate and nature of gestures produced when speaking English. A survey study revealed that 75% of ASL learners (N = 95), but only 14% of Romance language learners (N = 203), felt that they gestured more after one year of language instruction. A longitudinal study confirmed this perception. Twenty-one ASL learners and 20 Romance language learners (French, Italian, Spanish) were filmed re-telling a cartoon story before and after one academic year of language instruction. Only the ASL learners exhibited an increase in gesture rate, an increase in the production of iconic gestures, and an increase in the number of handshape types exploited in co-speech gesture. Five ASL students also produced at least one ASL sign when re-telling the cartoon. We suggest that learning ASL may (i) lower the neural threshold for co-speech gesture production, (ii) pose a unique challenge for language control, and (iii) have the potential to improve cognitive processes that are linked to gesture.
gesture; American Sign Language; bilingualism
Italian children are immersed in a gesture-rich culture. Given the large gesture repertoire of Italian adults, young Italian children might be expected to develop a larger inventory of gestures than American children. If so, do these gestures impact the course of language learning? We examined gesture and speech production in Italian and US children between the onset of first words and the onset of two-word combinations. We found differences in the size of the gesture repertoires produced by the Italian vs. the American children, differences that were inversely related to the size of the children’s spoken vocabularies. Despite these differences in gesture vocabulary, in both cultures we found that gesture + speech combinations reliably predicted the onset of two-word combinations, underscoring the robustness of gesture as a harbinger of linguistic development.
Early communication; gesture and culture; gesture in Italian; gesture-word combinations; two-word speech
Reciprocal Imitation Training (RIT) is a naturalistic behavioral intervention that teaches imitation to children with autism within a social-communicative context. RIT has been shown to be effective at teaching spontaneous, generalized object and gesture imitation. In addition, improvements in imitation were associated with increases in verbal imitation and spontaneous language.
This study used a modified multiple-baseline design across four children to examine whether adding gesture imitation training improves the overall rate of appropriate language use in children with ASD who have already been participating in object imitation training.
Three of the four children showed greater improvements in their use of appropriate language after gesture imitation was begun. Further, the children were more likely to use verbal imitation during gesture than object imitation training.
These findings suggest adding gestural imitation training to object imitation training can lead to greater gains in rate of language use than object imitation alone. Implications for both language development and early intervention are discussed.
To investigate, by means of fMRI, the influence of the visual environment in the process of symbolic gesture recognition. Emblems are semiotic gestures that use movements or hand postures to symbolically encode and communicate meaning, independently of language. They often require contextual information to be correctly understood. Until now, observation of symbolic gestures was studied against a blank background where the meaning and intentionality of the gesture was not fulfilled.
Normal subjects were scanned while observing short videos of an individual performing symbolic gesture with or without the corresponding visual context and the context scenes without gestures. The comparison between gestures regardless of the context demonstrated increased activity in the inferior frontal gyrus, the superior parietal cortex and the temporoparietal junction in the right hemisphere and the precuneus and posterior cingulate bilaterally, while the comparison between context and gestures alone did not recruit any of these regions.
These areas seem to be crucial for the inference of intentions in symbolic gestures observed in their natural context and represent an interrelated network formed by components of the putative human neuron mirror system as well as the mentalizing system.
Previous studies have shown that iconic gestures presented in an isolated manner prime visually presented semantically related words. Since gestures and speech are almost always produced together, this study examined whether iconic gestures accompanying speech would prime words and compared the priming effect of iconic gestures with speech to that of iconic gestures presented alone. Adult participants (N = 180) were randomly assigned to one of three conditions in a lexical decision task: Gestures-Only (the primes were iconic gestures presented alone); Speech-Only (the primes were auditory tokens conveying the same meaning as the iconic gestures); Gestures-Accompanying-Speech (the primes were the simultaneous coupling of iconic gestures and their corresponding auditory tokens). Our findings revealed significant priming effects in all three conditions. However, the priming effect in the Gestures-Accompanying-Speech condition was comparable to that in the Speech-Only condition and was significantly weaker than that in the Gestures-Only condition, suggesting that the facilitatory effect of iconic gestures accompanying speech may be constrained by the level of language processing required in the lexical decision task where linguistic processing of words forms is more dominant than semantic processing. Hence, the priming effect afforded by the co-speech iconic gestures was weakened.
co-speech gestures; cross-modal priming; lexical decision; language processing
Previous research has found that people with aphasia produce more spontaneous iconic gesture than control participants, especially during word-finding difficulties. There is some evidence that impaired semantic knowledge impacts on the diversity of gestural handshapes, as well as the frequency of gesture production. However, no previous research has explored how impaired semantic knowledge impacts on the frequency and type of iconic gestures produced during fluent speech compared with those produced during word-finding difficulties.
To explore the impact of impaired semantic knowledge on the frequency and type of iconic gestures produced during fluent speech and those produced during word-finding difficulties.
Methods & Procedures
A group of 29 participants with aphasia and 29 control participants were video recorded describing a cartoon they had just watched. All iconic gestures were tagged and coded as either “manner,” “path only,” “shape outline” or “other”. These gestures were then separated into either those occurring during fluent speech or those occurring during a word-finding difficulty. The relationships between semantic knowledge and gesture frequency and form were then investigated in the two different conditions.
Outcomes & Results
As expected, the participants with aphasia produced a higher frequency of iconic gestures than the control participants, but when the iconic gestures produced during word-finding difficulties were removed from the analysis, the frequency of iconic gesture was not significantly different between the groups. While there was not a significant relationship between the frequency of iconic gestures produced during fluent speech and semantic knowledge, there was a significant positive correlation between semantic knowledge and the proportion of word-finding difficulties that contained gesture. There was also a significant positive correlation between the speakers' semantic knowledge and the proportion of gestures that were produced during fluent speech that were classified as “manner”. Finally while not significant, there was a positive trend between semantic knowledge of objects and the production of “shape outline” gestures during word-finding difficulties for objects.
The results indicate that impaired semantic knowledge in aphasia impacts on both the iconic gestures produced during fluent speech and those produced during word-finding difficulties but in different ways. These results shed new light on the relationship between impaired language and iconic co-speech gesture production and also suggest that analysis of iconic gesture may be a useful addition to clinical assessment.
Gesture; Aphasia; Semantic knowledge
Impairments in language and communication are core features of Autism Spectrum Disorder (ASD), and a substantial percentage of children with ASD do not develop speech. ASD is often characterized as a disorder of brain connectivity, and a number of studies have identified white matter impairments in affected individuals. The current study investigated white matter integrity in the speech network of high-functioning adults with ASD. Diffusion tensor imaging (DTI) scans were collected from 18 participants with ASD and 18 neurotypical participants. Probabilistic tractography was used to estimate the connection strength between ventral premotor cortex (vPMC), a cortical region responsible for speech motor planning, and five other cortical regions in the network of areas involved in speech production. We found a weaker connection between the left vPMC and the supplementary motor area in the ASD group. This pathway has been hypothesized to underlie the initiation of speech motor programs. Our results indicate that a key pathway in the speech production network is impaired in ASD, and that this impairment can occur even in the presence of normal language abilities. Therapies that result in normalization of this pathway may hold particular promise for improving speech output in ASD.
•We used diffusion tensor imaging to measure white matter (WM) tracts in autism.•Autistic participants were high-functioning individuals with normal language skills.•WM between left supplementary motor and premotor areas is impaired in autism.•This tract is believed to be involved in the initiation of speech articulation.•Speech production may be impaired in the absence of language deficits in autism.
Autism; ASD; Speech; Diffusion tensor imaging; Tractography; Communication
To explore the specificity of impaired praxis and postural knowledge to autism by examining three samples of children, including those with autism spectrum disorder (ASD), attention-deficit hyperactivity disorder (ADHD), and typically developing (TD) children.
Twenty-four children with ASD, 24 children with ADHD, and 24 TD children, ages 8–13, completed measures assessing basic motor control (the Physical and Neurological Exam for Subtle Signs; PANESS), praxis (performance of skilled gestures to command, with imitation, and tool use) and the ability to recognize correct hand postures necessary to perform these skilled gestures (the Postural Knowledge Test; PKT).
Children with ASD performed significantly worse than TD children on all three assessments. In contrast, children with ADHD performed significantly worse than TD controls on PANESS but not on the praxis examination or PKT. Furthermore, children with ASD performed significantly worse than children with ADHD on both the praxis examination and PKT, but not on the PANESS.
Whereas both children with ADHD and children with ASD show impairments in basic motor control, impairments in performance and recognition of skilled motor gestures, consistent with dyspraxia, appear to be specific to autism. The findings suggest that impaired formation of perceptual-motor action models necessary to development of skilled gestures and other goal directed behavior is specific to autism; whereas, impaired basic motor control may be a more generalized finding.
imitation; motor learning; procedural learning; premotor cortex; inferior parietal lobe
It has been suggested that children with autism are particularly deficient at imitating novel gestures or gestures without goals. In the present study, we asked high-functioning autistic children and age-matched typically developing children to imitate several types of gestures that could be either already known or novel to them. Known gestures either conveyed a communicative meaning (i.e., intransitive) or involved the use of objects (i.e., transitive). We observed a significant interaction between gesture type and group of participants, with children with autism performing known gestures better than novel gestures. However, imitation of intransitive and transitive gestures did not differ across groups. These findings are discussed in light of a dual-route model for action imitation.
How does gesturing help children learn? Gesturing might encourage children to extract meaning implicit in their hand movements. If so, children should be sensitive to the particular movements they produce and learn accordingly. Alternatively, all that may matter is that children move their hands. If so, they should learn regardless of which movements they produce. To investigate these alternatives, we manipulated gesturing during a math lesson. We found that children required to produce correct gestures learned more than children required to produce partially correct gestures, who learned more than children required to produce no gestures. This effect was mediated by whether children took information conveyed solely in their gestures and added it to their speech. The findings suggest that body movements are involved not only in processing old ideas, but also in creating new ones. We may be able to lay foundations for new knowledge simply by telling learners how to move their hands.
Language impairment is a hallmark of autism spectrum disorders (ASD). The origin of the deficit is poorly understood although deficiencies in auditory processing have been detected in both perception and cortical encoding of speech sounds. Little is known about the processing and transcription of speech sounds at earlier (brainstem) levels or about how background noise may impact this transcription process. Unlike cortical encoding of sounds, brainstem representation preserves stimulus features with a degree of fidelity that enables a direct link between acoustic components of the speech syllable (e.g., onsets) to specific aspects of neural encoding (e.g., waves V and A). We measured brainstem responses to the syllable /da/, in quiet and background noise, in children with and without ASD. Children with ASD exhibited deficits in both the neural synchrony (timing) and phase locking (frequency encoding) of speech sounds, despite normal click-evoked brainstem responses. They also exhibited reduced magnitude and fidelity of speech-evoked responses and inordinate degradation of responses by background noise in comparison to typically developing controls. Neural synchrony in noise was significantly related to measures of core and receptive language ability. These data support the idea that abnormalities in the brainstem processing of speech contribute to the language impairment in ASD. Because it is both passively-elicited and malleable, the speech-evoked brainstem response may serve as a clinical tool to assess auditory processing as well as the effects of auditory training in the ASD population.
auditory brainstem; autism spectrum disorder; speech; language; evoked potentials
Electroencephalogram was recorded as healthy adults viewed short videos of spontaneous discourse in which a speaker used depictive gestures to complement information expressed through speech. Event-related potentials were computed time locked to content words in the speech stream and to subsequent related and unrelated picture probes. Gestures modulated event-related potentials to content words co-timed with the first gesture in a discourse segment, relative to the same words presented with static freeze frames of the speaker. Effects were observed 200–550λms after speech onset, a time interval associated with semantic processing. Gestures also increased sensitivity to picture probe relatedness. Effects of gestures on picture probe and spoken word analysis were inversely correlated, suggesting that gestures differentially impact verbal and image-based processes.
auditory comprehension; audio-visual processing; discourse; gesture; multisensory integration; semantics; speech
Language and communicative impairments are among the primary characteristics of autism spectrum disorders (ASD). Previous studies have examined auditory language processing in ASD. However, during face-to-face conversation, auditory and visual speech inputs provide complementary information, and little is known about audiovisual (AV) speech processing in ASD. It is possible to elucidate the neural correlates of AV integration by examining the effects of seeing the lip movements accompanying the speech (visual speech) on electrophysiological event-related potentials (ERP) to spoken words. Moreover, electrophysiological techniques have a high temporal resolution and thus enable us to track the time-course of spoken word processing in ASD and typical development (TD). The present study examined the ERP correlates of AV effects in three time windows that are indicative of hierarchical stages of word processing. We studied a group of TD adolescent boys (n=14) and a group of high-functioning boys with ASD (n=14). Significant group differences were found in AV integration of spoken words in the 200–300ms time window when spoken words start to be processed for meaning. These results suggest that the neural facilitation by visual speech of spoken word processing is reduced in individuals with ASD.
In typically developing (TD) individuals, behavioural and event-related potential (ERP) studies suggest that audiovisual (AV) integration enables faster and more efficient processing of speech. However, little is known about AV speech processing in individuals with autism spectrum disorder (ASD). The present study examined ERP responses to spoken words to elucidate the effects of visual speech (the lip movements accompanying a spoken word) on the range of auditory speech processing stages from sound onset detection to semantic integration. The study also included an AV condition which paired spoken words with a dynamic scrambled face in order to highlight AV effects specific to visual speech. Fourteen adolescent boys with ASD (15–17 years old) and 14 age- and verbal IQ-matched TD boys participated. The ERP of the TD group showed a pattern and topography of AV interaction effects consistent with activity within the superior temporal plane, with two dissociable effects over fronto-central and centro-parietal regions. The posterior effect (200–300ms interval) was specifically sensitive to lip movements in TD boys, and no AV modulation was observed in this region for the ASD group. Moreover, the magnitude of the posterior AV effect to visual speech correlated inversely with ASD symptomatology. In addition, the ASD boys showed an unexpected effect (P2 time window) over the frontal-central region (pooled electrodes F3, Fz, F4, FC1, FC2, FC3, FC4) which was sensitive to scrambled face stimuli. These results suggest that the neural networks facilitating processing of spoken words by visual speech are altered in individuals with ASD.
Auditory; ASD; ERP; Language; Multisensory; Visual
Evidence is reviewed for the existence of a core system for moment-to-moment social communication that is based on the perception of dynamic gestures and other social perceptual processes in the temporal-parietal occipital junction (TPJ), including the posterior superior temporal sulcus (PSTS) and surrounding regions. Overactivation of these regions may produce the schizophrenic syndrome. The TPJ plays a key role in the perception and production of dynamic social, emotional, and attentional gestures for the self and others. These include dynamic gestures of the body, face, and eyes as well as audiovisual speech and prosody. Many negative symptoms are characterized by deficits in responding within these domains. Several properties of this system have been discovered through single neuron recording, brain stimulation, neuroimaging, and the study of neurological impairment. These properties map onto the schizophrenic syndrome. The representation of dynamic gestures is multimodal (auditory, visual, and tactile), matching the predominant hallucinatory categories in schizophrenia. Inherent in the perceptual signal of gesture representation is a computation of intention, agency, and anticipation or expectancy (for the self and others). The neurons are also tuned or biased to rapidly detect threat-related emotions. I review preliminary evidence that overactivation of this system can result in schizophrenia.
Including gesture in instruction facilitates learning. Why? One possibility is that gesture points out objects in the immediate context and thus helps ground the words learners hear in the world they see. Previous work on gesture’s role in instruction has used gestures that either point to or trace paths on objects, thus providing support for this hypothesis. Here we investigate the possibility that gesture helps children learn even when it is not produced in relation to an object but is instead produced “in the air.” We gave children instruction in Piagetian conservation problems with or without gesture and with or without concrete objects. We found that children given instruction with speech and gesture learned more about conservation than children given instruction with speech alone, whether or not objects were present during instruction. Moreover, children who received instruction in speech and gesture were more likely to give explanations for how they solved the problems that they were not taught during the experiment; this advantage was found only when objects were absent during instruction. Gesture in instruction can thus help learners learn even when those gestures do not direct attention to visible objects, suggesting that gesture can do more for learners than simply ground arbitrary, symbolic language in the physical, observable world.
Gesture; Instruction; Action; Learning; Teaching
This study used eye-tracking methodology to assess audiovisual (AV) speech perception in 26 children ranging in age from 5-15 years, half with autism spectrum disorders (ASD) and half with typical development (TD). Given the characteristic reduction in gaze to the faces of others in children with ASD, it was hypothesized that they would show reduced influence of visual information on heard speech. Responses were compared on a set of auditory, visual and audiovisual speech perception tasks. Even when fixated on the face of the speaker, children with ASD were less visually influenced than TD controls. This indicates fundamental differences in the processing of AV speech in children with ASD, which may contribute to their language and communication impairments.
The gestures children produce predict the early stages of spoken language development. Here we ask whether gesture is a global predictor of language learning, or whether particular gestures predict particular language outcomes. We observed 52 children interacting with their caregivers at home, and found that gesture use at 18 months selectively predicted lexical versus syntactic skills at 42 months, even with early child speech controlled. Specifically, number of different meanings conveyed in gesture at 18 months predicted vocabulary at 42 months, but number of gesture+speech combinations did not. In contrast, number of gesture+speech combinations, particularly those conveying sentence-like ideas, produced at 18 months predicted sentence complexity at 42 months, but meanings conveyed in gesture did not. We can thus predict particular milestones in vocabulary and sentence complexity at age 3 1/2 by watching how children move their hands two years earlier.