Using the cross-modality paradigm, in the present study we investigated pre-attentive processing of face expressions by recording the expression-related visual mismatch negativity (expression MMN, EMMN). To observe genuine memory-comparison-based EMMN, stimulus-specific refractoriness was eliminated by using equiprobable control block. As expected, two kinds of EMMNs, oddball-EMMN in deviant-standard comparison in oddball block and controlled-EMMN in deviant-control comparison, were found with posterior scalp distribution. In particular, the oddball-EMMN was bigger and earlier than the controlled-EMMN because the face-associated N170 did not contribute to the latter.
Similar to auditory MMN studies, a suitable experimental approach for visual MMN would involve maintaining the subject's attention to a continuous task in which the task stimuli are independent of irrelevant ones. In the present study we adopted the cross-modality method proposed by Maekawa et al [28
], in which subjects were instructed to focus their attention on a story delivered binaurally through earphones, while looking at the center of the monitor, and press a button as soon as they recognize a visual target stimulus. The visual target task could make subject to really look at the center of monitor so that the visual MMN could be elicited by no-target stimuli. Obviously, it is possible that subjects, indeed, attend the visual modality. However, in Maekawa et al' study [28
], the authors found that only when irrelevant deviants were close resemblance to the target they did capture the attention of the subject, reflecting by the fact that deviants interfered with target in terms of the RT results. In contrast, irrelevant deviants did not interfere with target identification when they were very different from the target. Similarly, in the present study, the visual target stimulus (i.e., white circle) was very different from non-target standard and deviant stimuli and the subjects were asked to attention the auditory story throughout the experiments. In addition, the attention specific N2b-P3 complex [36
] was elicited only by visual target stimuli but not by deviant faces (see Figure ). As proposed by Maekawa et al [28
], therefore, these results support the idea that subject's attention was almost shifted away from the non-target deviant stimuli. In consistent with our previous study [5
], compared with the ERP waveform of standard neutral faces as well as control sad faces, a clear negative shift over posterior scalp was elicited by deviant sad faces, regardless of control and oddball conditions, that is, a valid EMMN was observed. We believed that the EMMN reported here reflected pre-attentive expressional information processing.
Compared with standard neutral faces in oddball block, infrequent sad faces elicited posterior negativities between 100 ms and 350 ms, which is highly consistent with previous facial expression oddball study [5
]. The posterior negativity observed in the deviant-versus-standard comparison, i.e., oddball-EMMN, indeed consisted of two subsequent posterior negativities, early negativity (100-200 ms) based on N170 difference and late negativity (200-350 ms) due to P2 difference. In line with the present N170 effect in oddball sequence, using similar cross-modality oddball paradigm one recent study showed that deviant expressional faces elicited larger N170 than frequent neutral faces, resulting in the occipital negativity at the latency of 150-180 ms, and the authors considered that the N170 could be suitable counterpart of visual EMMN [21
]. However, some visual MMN studies indicated that the traditional MMN in the oddball sequence indeed confounds standard stimuli refractoriness reflected by the changes of early visual ERP components such as the temporo-occipital N1 component (N170 component in the present study) [13
]. Recently, Kimura et al. [13
] investigated directly the underlying processes of the visual MMN and considered that the early posterior negativity peaked at around 100-150 ms reflects visual N1 refractory effect, while the late negativity peaked at around 200-250 ms reflects memory-comparison-based change detection effect (that is, visual MMN). Indeed, the present EMMN time-course and distribution was also (partly) similar to the Kimura et al' vMMN study [13
], e.g., occipital, right-hemispheric preponderant distribution, latency in the N1 to P2 range. Therefore, in the present study as well as previous EMMN studies the early part of oddball-EMMN based on N170 component changes could reflect stimulus-specific refractoriness. Supporting this notion, in deviant-versus-control comparison the N170 was not modulated at all and the controlled-EMMN was evident between 200-350 ms post stimuli onset. In the present experiment, control stimuli should not active change-specific neural populations and the state of refractoriness for control stimuli should be similar. Thus, the controlled-EMMN eliminated absolutely N170 refractory effect and reflects the memory-comparison-based change detection effect (i.e., genuine EMMN). At the same time, our data further support the suggestion that classical visual MMN in deviant-versus-standard comparison contains a contribution associated with neuron populations in different states of refractoriness responding to the standard and deviant stimuli [13
Interestingly, in contrast to the present N170 effect on expression processing in deviant-versus-control condition, there is evidence that the face-associated N170 component was not totally insensitive to the facial emotional content even under non-consciousness condition (see [37
] for a review). In those studies the comparison among different facial expressions was performed, while in the present deviant-control comparison the N170s elicited by expression stimuli with same physical features were compared. Since the N170 reflects an early activation of a domain-specific mechanism for visual analysis of faces [38
], the present results suggest that pre-attentive memory-comparison-based processing of facial expressions occur after the perceptual processing relevant to the N170.
The topography analysis of the controlled-EMMN showed the occipito-temporal distribution (larger at right than at left hemisphere sites). In line with the present hemispheric specialization for change detection of sad expression, several recent EMMN studies showed that automatic processing of negative affect (e.g., fearful, angry) was associated with right hemisphere, while processing of positive affect (e.g., happy) evoked larger MMN in the left hemisphere [[5
], but see [41
]]. In addition, the source analysis of controlled-EMMN indicated a current source primarily involved in posterior areas including superior temporal gyrus, postcentral gyrus, inferior parietal lobule as well as the insula. Using sLORETA method like the present study, one recent source study of visual MMN elicited by orientation changes found that current sources of the visual N1 reflecting refractoriness effect were located in the occipital lobe (BA 17-19) and the visual MMN reflecting memory-comparison-based processing involved in neural activations of the occipital lobe (BA 19) and the frontal lobe (BA 47 and BA 11) [35
]. Our results extend the vMMN study for simple visual features (e.g., color) to for complex visual information (i.e., facial expressions). The distinction between generators for expression MMN and visual MMN might indicate the specificity for visual MMN elicited by expression changes over by simple visual feature changes and further within-group studies are necessary to determine the distinction between vMMN and EMMN.
Particular relevant to present study, Kimura et al [40
] found that for both fearful and happy faces, the neural generators of EMMN were located in temporal, occipital, limbic and frontal lobes. Obviously, there were partly diverging between Kimura et al and our present findings, in particular, we did not find the fontal activations of EMMN. In Kimura et al's study [40
], the violated alternating pattern (i.e., perceptual learning pattern) was adopted to investigate the prediction error responses. Converging evidence indicated that prefrontal area plays an important role in error processing [42
] and hence, it is possible that the EMMN generator is located in frontal lobe like in Kimura et al's study [40
]. However, the EMMN in the present experiment was elicited in a simple random pattern (i.e., oddball sequence) that is related to sensory memory-based comparison not to repeating prediction processing. On the other hand, we did not use the fearful and happy expressions in the present study. Although converging evidence from functional imaging studies suggested that there is certain distinction of neural correlates for processing of different emotional facial expression, only a few studies explored the neural response to sad expressions [2
]. For example, using a sex discrimination task, Blair et al [43
] found that increasing intensity of sad facial expression was associated with enhanced activity in the left amygdala and right temporal region. It has been widely accepted that the areas including amygdale, cingulate cortex, and basal ganglia were activated during the sad expression recognition [44
]. Interestingly, the present current sources of EMMN elicited by sad expression located in the insula with a maximum of current density. In contrast to this finding, insular activation has been selectively reported during processing of disgusted and angry faces in brain imaging studies [45
]. As our knowledge, however, there were no brain-imaging studies about processing sad faces under the pre-attentive memory-based condition. Considering the methodological distinction between ERPs and functional imaging as well as the variety of valence and intensity of facial expressions, the neuro basis of processing sad faces under non-attentional condition awaits further investigation.