We are able to explore and select information in the environment in a flexible way and usually do not experience any limits or difficulty when doing so. In a cluttered visual scene, we can mentally select and extract visual information and even relate objects that have nothing in common. This ability appears to be impaired in patients with schizophrenia, and may impact on how they adapt to the visual environment. It has been related to a more general difficulty at organizing information that is expressed at a clinical level (Silverstein and Keane, 2011
). However, the mechanisms of these impairments are still debated, and especially the relative contribution of automatic grouping mechanisms vs. high-level, top-down mechanisms. Our aim is twofold. Understanding how patients with schizophrenia explore the visual environment should help us to understand the mechanisms underlying their difficulties when attempting to adapt to an ever changing environment. More generally it might contribute to objectify and better define the difficulties of patients at organizing information. Second, patients’ results lead to questions regarding the mechanisms of the mental selection of objects in healthy volunteers and how these mechanisms tie in with automatic grouping. This question is not fully resolved in healthy controls. For this reason we went back and forth from fundamental knowledge to clinically related issues. We explored the ability to mentally relate objects in healthy volunteers, and we use this same paradigm here in patients with schizophrenia. The results will be used to discuss first their significance for patients and second what they reveal about mental selection and visual organization in healthy volunteers.
There is already considerable knowledge regarding visual processing in healthy volunteers. Form processing is known to involve a number of steps, from the extraction of primitives (local orientation, color, luminance, etc.), to the integration of the form contour and surface filling-in that sub-tend object recognition (Boucart et al., 1994
; Humphreys, 2003
; Grossberg et al., 2007
). The integration of contour information involves Gestalt rules like grouping by collinearity, proximity, similarity or common fate, and the use of segmentation cues in order to correctly separate object parts, objects from the background, and objects from one another (Boucart et al., 1994
; Kovács, 1996
; Beck and Palmer, 2002
; Spillmann, 2006
). Similar rules apply when considering the relationship between distinct objects, even though the pathways sub-tending the coding of relations between objects are distinct from those sub-tending the coding of the relations within objects (Humphreys, 1998
; Davis, 2001
). Grouping between individual items allows to identify global forms that emerge from the way local elements are organized (Kimchi, 2000
; Kimchi et al., 2005
). Information at the global and local levels are processed by specialized neural pathways, and structure the visual environment in a hierarchical manner (Delis et al., 1986
; van Kleeck, 1989
; Hübner and Volberg, 2005
). A number of studies suggest that grouping mechanisms can occur automatically under conditions of inattention (Driver et al., 2001
; Müller et al., 2010
). It has been shown also, however, that attention can interact with grouping (Driver et al., 2001
), and can be directed either toward the local or the global level (Robertson et al., 1993
; Humphreys, 1998
Here, we question what happens when attention is directed toward object pairs that are unrelated and do not form a global shape. We argue this question is not resolved by usual mechanisms of grouping, and we suspect it might be crucial to understand the impairments in patients with schizophrenia (van Assche and Giersch, 2011
). We explore it by using a paradigm designed by Beck and Palmer (2002
). Beck and Palmer (2002
), see also Palmer and Beck (2007
) built visual search tasks with a setting which can be considered as a simplified version of a visual scene. Squares and circles represent simple objects and are displayed on a horizontal row. Squares and circles alternate on the row, except for two shapes sharing the same form and displayed one beside the other (Figures A–C). The task of the subjects is to spot these two identical and adjacent shapes, which represent the targets, and to discriminate their form, i.e., to decide whether they are two circles or two squares. An additional manipulation allows us to evaluate the effect of grouping. The objects on the row are grouped by pairs on the basis of classical rules like proximity, or the presence of connectors linking the shapes. As a consequence of this grouping manipulation, the two identical shapes are either part of the same pair of related figures (i.e., grouped by proximity or connecters), or part of different pairs (i.e., unrelated). As can be expected, it is easier for subjects to find the targets if they are part of a pair of related shapes (i.e., grouped), than if they are unrelated and part of different pairs. This effect reflects the advantage provided by grouping. Interestingly, this advantage is modulated by contextual information, i.e., the percentage of related vs. unrelated targets within an experimental block. Beck and Palmer (2002
) used three experimental blocks, one with 75% unrelated and 25% related targets, one with equal proportions of unrelated and related targets, and one with 25% unrelated and 75% related targets. The advantage for related targets increases when related targets are the majority, and decreases when they are the minority. This modulation could not be explained by repetition effects, i.e., facilitated search for a target pair when it follows a trial with a pair in the same condition (e.g., related targets following related targets or unrelated targets following unrelated targets). Indeed, Beck and Palmer (2002
) observed probability effects for both repeated and non-repeated trials. The performance modulation rather reflects the prioritization of one type of pairs (related or unrelated) according to the contextual information provided by the frequency of these pairs within a given experimental block. On each trial subjects must process visual information in order to locate the related pairs, and then can direct their attention to the prioritized pairs. Since the prioritization relies on the estimation of a frequency across different trials and is not provided by information in a single trial, it can thus be considered as a top-down effect. This does not mean that subjects provide a conscious effort to prioritize related or unrelated pairs. When subjects are not informed about the proportion modulation and cannot report it after the tasks, the effect is nonetheless identical to the prioritization obtained when subjects are informed (data obtained in unpublished pilot studies). All in all the modulation is considered top-down because it results from a global and automatic probability estimation rather than from a local priming effect.
Figure 1 Illustration of the stimuli used to explore visual grouping and re-grouping. (A–C) stimuli used in the original paradigm of Beck and Palmer (2002), with a manipulation of grouping by proximity (A,B) and by connecters (C). Only two shapes shared (more ...)
A top-down modulation of grouping does not imply that unrelated targets can be prioritized, and as a matter of fact, the possibility to prioritize unrelated targets is not straightforward. In the results of Beck and Palmer (2002
), the modulation effects for related and unrelated shapes were usually symmetrical. This means that each time performance was improved for related pairs, there was a symmetrical cost for unrelated pairs, and the reverse. Such results can be interpreted as a modulation of the prioritization of connected pairs, and the performance variations for unrelated shapes might be an automatic consequence of the varying prioritization of connected pairs. The more subjects would focus on connected pairs the less they would spend on unrelated shapes. In other words, the results do not imply that unrelated shapes are prioritized selectively. As a matter of fact, the advantage for related over unrelated shapes shows that selecting two unrelated shapes at the same time entails some difficulties (Beck and Palmer, 2002
). The literature on multiple object tracking confirms these difficulties, even though it shows it is possible to select distinct shapes. During multiple object tracking tasks, subjects select several unrelated objects efficiently enough to track them when they move in distinct directions among distracters (reviews in Pylyshyn, 2001
; Cavanagh and Alvarez, 2005
; Alvarez, 2011
). It has been proposed that such ability is sub-tended by goal-directed re-grouping of the separate objects (Yantis, 1992
; Alvarez, 2011
). However, this ability is severely impacted when the selected objects are automatically grouped with distracters (Scholl et al., 2001
; Suganuma and Yokosawa, 2006
). This suggests that it is very difficult to select distinct objects when each one is part of a different group. Even with a simpler visual search paradigm and static objects, object-centered attention can be expected to induce difficulties when trying to focus on two shapes that belong to different pairs of objects. Object-centered attention implies that when attention is focused on an element of a group, then attention spreads to the whole group (Duncan, 1984
; Egly et al., 1994
; Matsukura and Vecera, 2006
). This means that when distracters are grouped with target information, attention directed toward the target will spread to distracter information, and attention is not drawn on target information in a selective way anymore. Despite this, is it really the case that we cannot attend selectively to two items when they belong to different sets of objects? In every day life, it can happen that we pick up detail information in different sets of objects and compare them or associate them mentally. In fact, it happens each time information is hierarchical, and we wish to associate mentally details from different hierarchical objects sets (e.g., flowers from different houses, leaves from different trees, fruits from different piles). Yet we are usually able to compare two details from different houses, fruit piles, or trees without experiencing any noticeable difficulty. Our own results (Giersch and Rhein, 2008
; van Assche et al., 2012
) confirm we can attend to such details and associate them selectively.
We will call “re-grouping” the ability to attend selectively to figures that are not only separate but also part of different sets of objects. We obtained some evidence of “re-grouping” by deriving new paradigms from the one elaborated by Beck and Palmer (2002
). We observed in two different studies that healthy volunteers are able to focus selectively on unrelated pairs, even when they belong to different pairs of figures. (Giersch and Rhein, 2008
; van Assche et al., 2012
). In van Assche et al. (2012
), targets were circles and squares like in the original paradigm, but they were arranged in alternation on a circle around a fixation point. Subjects decided whether the two identical shapes located one beside the other were two circles or two squares, as in the typical experiment. The presence of connecters led to the perception of pairs of figures (Figure ), and as in the previous experiments, targets were either part of the same connected pair, or belonged to two different pairs. We manipulated the frequency of connected and unconnected targets in three different experimental blocks. Contrary to previous experiments however, subjects were instructed to look at the central fixation point throughout the experiment, and this was checked by continuous eye-tracking. In case of an ocular saccade out of central area, the trial was stopped, and was presented again at the end of the experimental block. Hence, subjects could not visually sweep across the stimuli. Because eye movements were not allowed, subjects could not compare nearby figures through ocular exploration, and had to relate them mentally. This might explain why this procedure helped us to evidence “re-grouping” of unrelated figures more easily than previous paradigms. As a matter of fact, the results showed that subjects became significantly faster (by no less than 123
ms) at finding unconnected targets when those targets were the majority, as compared to the block with an equal proportion of connected and unconnected targets. Despite this large improvement, performance for connected targets remained stable across these two blocks, suggesting that focalization on unconnected targets cannot be explained by an inhibition of connected targets and rather involve a selective “re-grouping” of unconnected targets.
Figure 2 Example of the stimuli used with an arrangement of figures around a central fixation point. Subjects had to fixate the central point throughout the trials, and this was checked with continuous eye-tracking (Cambridge Research System, 50Hz). Connecters (more ...)
All in all, the results suggest that unrelated stimuli can be isolated and “re-grouped” efficiently, even if they belong to different objects groups. This mechanism bypasses object-centered attention and cannot be accounted for by global/local processing. Local and global information correspond to individual shapes and pairs of related figures, respectively, but the pairs of unrelated figures correspond to neither, and may require higher-level cognitive operations. A late mechanism would be dissociated from usual mechanisms of visual grouping and would rather involve attentional selection mechanisms. It should be noted that our data does not allow us to distinguish between a simultaneous selection of two stimuli and the possibility that each figure is attended to in turn very fast, i.e., that items are selected sequentially rather than simultaneously (Hogendoorn et al., 2010
). In the latter case (sequential selection), subjects would not be conscious of alternating between items. Hence, this possibility is beyond the scope of this paper, since both possibilities allow for a selective focalization on the two figures during a period of time. As such, both possibilities, simultaneous selection or fast serial selection, lead to questions regarding object-based attention, and conflict with usual, automatic grouping. We explored this question further by studying to which extent the outputs of automatic grouping and “re-grouping” differ.
Even if the mechanisms underlying this ability are clearly different to the ones underlying classical grouping, one might wonder whether they have comparable end-products.
We used the cost of across-hemifield presentation as a tool to contrast the impact of automatic grouping and “re-grouping.”
We observed that connecters between targets, or physical arrangement leading to automatic grouping, erased the cost of across-hemifield presentation (van Assche et al., 2012
). The benefit provided by connecters is akin to what had been described in patients with parietal lesions, who display a difficulty to perceive stimuli in the contra-lesional hemifield (Driver, 1995
; Gilchrist et al., 1996
; Pavlovskaya et al., 1997
; Boutsen and Humphreys, 2000
; Brooks et al., 2005
). The benefit of grouping contrasted with “re-grouping,” which was without effect on this cost. Even when attention prioritization led to a large improvement of performance for unconnected targets, the cost of across-hemifield presentation remained high. In fact, it was as high as when unconnected targets were the minority. These results suggest that in addition to taking different routes, grouping and “re-grouping” also differ in their output. In other words, outputs for “re-grouping” and automatic grouping would differ. This would sub-tend our subjective experience suggesting that automatic grouping provides background information and that our mental exploration is akin to playing with such information at the foreground. However, as emphasized above, we perceive only one unique outer world, implying that “re-grouping” must be somehow tied in with automatic grouping.
The literature and our own results suggest that understanding the role of “re-grouping” and how it ties in with automatic grouping might be crucial in patients with schizophrenia. Conversely, the results in patients might shed light on this question. A number of studies has shown that patients with schizophrenia have a difficulty to organize visual information (review in Silverstein and Keane, 2011
), and our own initial studies suggested a selective difficulty to “re-group” unconnected items (Giersch and Rhein, 2008
; van Assche and Giersch, 2011
). A selective difficulty at “re-grouping” would be an argument in favor of a complete dissociation between automatic grouping and “re-grouping.” Recently however, we used a working memory task, and results suggested that patients can re-group items when incited to, but then experience a conflict between usual grouping and “re-grouping,” which contrasts with results in controls. This suggests that the difficulty at “re-grouping” also impacts on the ability of the patients to use automatic grouping processes. If grouping and “re-grouping” are found to be competing in patients but not in controls, this would confirm that the usual preservation of automatic grouping is not as straightforward as believed. It would call for explanations on how healthy subjects avoid this competition and make “grouping” and “re-grouping” coexist.
The results to date were obtained in a memory task, however, and the competition between representations of related and unrelated figures might have been specific to this memory task. To test this possibility, we checked whether similar results could be obtained in a perception task.
In order to test the possibility of a competition between grouping and re-grouping in patients with schizophrenia, we used again figures arranged in a circle around a central fixation point, as already described. We chose this arrangement because it had been particularly efficient in showing the effect of a prioritization of unrelated figures in healthy subjects (van Assche et al., 2012
). If patients are unable to re-group items, then we should see no effect of prioritization in patients, i.e., less variation in performance than in controls when the proportion of related and unrelated figures is manipulated. These results would then be similar to those observed in our first study (Giersch and Rhein, 2008
). If on the contrary the task is efficient in inciting patients to re-group unrelated figures, then we should observe performance variations across blocks. Most importantly, if patients can maintain the link between related figures while re-grouping information, then performance for related figures should be preserved. If in contrast patients can re-group unrelated figures only at the expense of the link between related figures, as we observed recently (Giersch et al., 2011
) then we should observe a cost for related figures that is symmetrical to the gain for unrelated figures. This would indicate a competition between the access to related and unrelated figures in patients, and would reinforce our hypothesis that specific mechanisms are at work to enable the coexistence of the two types of groupings.
In addition, we contrasted within- and across-hemifield presentations, and this was expected to further our understanding of the mechanisms at work, and especially to what extent automatic grouping and “re-grouping” are dissociated. Our previous work has shown that our configuration leads to a large RT cost in case of unconnected targets displayed across-hemifields. Interhemispheric transfer is believed to be impaired in patients (Schwartz et al., 1984
; Mohr et al., 2008
, but see David, 1993
), and we wondered if this explains the difficulties of patients with schizophrenia at “re-grouping.” In that case patients with schizophrenia should be impaired relative to controls mainly in case of across-hemifield presentation. On the other hand, if patients can focus on re-grouped figures, the comparison of the effects of hemifield presentation in patients and controls was expected to give some indications on the mechanisms at work in the two groups. The idea was that early mechanisms of “re-grouping” were expected to be sensitive to the cost of interhemispheric transfer, whereas later and lateralized mechanisms (Kosslyn, 1987
; van der Ham et al., 2011
; Stevens et al., 2012
) would be less sensitive to this cost. Our previous results confirmed that even though the presentation across-hemifields globally slowed down healthy subjects when the targets were unconnected, it was without effect on the prioritization induced by the manipulation of the percentage of connected vs. unconnected figures. If patients with schizophrenia re-group and prioritize pairs the same way as controls, then their pattern of responses should be similar. In contrast, a difference in the effect of across-hemifield presentation might reveal a difference in the mechanisms at work in the two groups.