Appreciation of the functional organization of the mammalian visual system (Ungerleider & Haxby, 1994
; Ungerleider & Mishkin, 1982
) has led to the widely accepted view that working and short-term memory1
for objects (“what”) and locations (“where”) are computed by at least partially discrete neural systems in monkeys (Wilson, O’Scalaidhe, & Goldman-Rakic, 1993
) and humans (e.g., Courtney, Ungerleider, Keil, & Haxby, 1996
; McCarthy et al., 1996
; Mecklinger & Muller, 1996
). And behavioral studies in humans confirm that working memory for these two domains of information are supported by distinct visually based mental codes representing object identity and location, respectively (e.g., Della Sala, Gray, Baddeley, Allamano, & Wilson, 1999
; Hecker & Mapperson, 1997
; Smith et al., 1995
; Tresch, Sinnamon, & Seamon, 1993
). The purpose of the experiments presented in this report, however, was to explore whether spatial and object working memory may also differ along an axis orthogonal to that dictated by the functional organization of the visual system: Object working memory may automatically, obligatorily engage verbal coding mechanisms, whereas spatial working memory may not.
This theoretical proposition arose from our experience with spatial/object manipulations in memory tasks—delayed recognition (Postle & D’Esposito, 1999
; Postle, Jonides, Smith, Corkin, & Growdon, 1997
), conditional-associative learning (Postle, Locascio, Corkin, & Growdon, 1997
), and the n
-back task (Postle, Stern, Rosen, & Corkin, 2000
). This research has indicated that participants often adopt a strategy of verbally encoding stimuli in the object condition, although our tasks have used relatively “nonverbalizable” Attneave shapes. Participants in these studies of object memory were never instructed to use verbal coding to mediate performance on the tests, yet post-test debriefing suggested that the tendency to do so was strong and was consistent across age groups (from late teens to 80s), neurological status (healthy, Parkinson’s disease, stroke, and medial temporal-lobe amnesia), and testing environment (behavioral laboratory and fMRI laboratory). We had not observed a similar tendency in tests of spatial memory that were procedurally identical to the object tasks and that were administered in the same session.
The idea that the representations of objects in working memory include a semantic code can be seen as an extension of the simultaneous multiple encoding theory of Wickens (1972 Wickens (1973)
, which holds that words can be encoded according to their semantic attributes, the physical characteristics of their presentation at the time of encoding, and other attributes (such as language, frequency, representing symbol, and imageability). Thus, Wickens argued that words are encoded not only according to their central function—the conveyance of meaning—but according to attributes and contextual factors that can vary independent of a word’s semantic content. We, in turn, are proposing that the mnemonic representation of visually presented object stimuli may encode not only the visual features of the object (e.g., size, color, texture, shape), but also verbal information associated by the individual with the visual stimulus. Furthermore, we propose that this association occurs automatically, such that a verbal code is an inherent part of the working memory representation of a visually presented object. Initially, we could not specify whether this code is semantic or only lexical (as would be the case with a name devoid of meaning), although the results of the present study helped us to sharpen this aspect of our claim. Furthermore, our empirical observations led us to assert that the automaticity of verbal coding does not extend to all visually presented information. Specifically, we posited that it does not apply to the mnemonic representation of locations in space.
The theoretical and empirical literatures germane to the mnemonic coding of visually presented objects and locations, however, are incomplete. It is widely assumed that locations in extrapersonal space are represented mentally in a nonverbal analog code (Attneave, 1972
). Posner reported (Posner & Konick, 1966
) and replicated (Posner, 1967
) data consistent with an important role for rehearsal in spatial working memory: Reproduction of a visually guided movement showed minimal forgetting across an unfilled 20-sec delay interval but substantial forgetting when the delay was interpolated with a digit classification task. Posner (1967)
posited that the term rehearsal
need not be restricted to verbal processing but could apply to any case in which information retention required central processing capacity. Posner and Konick emphasized introspective reports of participants in concluding that the memory code employed to remember a spatial location was largely nonverbal, but they did not test this idea directly.
Familiar objects can be represented in many different codes. For example, tests of comparison of serially presented representational object stimuli (e.g., line drawings of buildings and photographs of cars) have revealed evidence for multiple levels of representational codes, including a visual object code and a nonvisual semantic code that may or may not be verbal (Bartram, 1976
). Other work has suggested that this semantic code is, in fact, verbally mediated and that participants’ expectations about the nature of a probe stimulus (either a picture of or the name for a well-learned schematic face) in a delayed-recognition test was a more important determinant of the accessibility of the verbal code versus the pictorial code associated with an item than was the modality (verbal or pictorial) in which the target stimulus had been presented (Tversky, 1969
Most relevant to the present question are investigations using abstract object stimuli that do not inherently represent objects in the real world. Cermak (1977)
demonstrated that (13-sec) delayed-comparison memory for abstract outline figures was sensitive to the ease with which target and probe stimuli could be interpreted as representing the same objects (e.g., a hippopotamus head or a human face). Although these results demonstrated that semantic interpretation of an object can have an important influence on working memory, they did not address whether the use of semantic codes is fundamental to the short-term retention of information about objects or whether it is strategic. This is because Cermak’s experimental procedures deliberately induced subjects to establish semantic codes for studied items.
More recently, a study by Simons (1996)
directly compared working memory for spatial or object characteristics of complex pictures with those of objects in an array. Simons reported that object memory can be considerably worse than spatial memory, and that only object memory was sensitive to experimental manipulations intended to block verbal labeling. On the basis of these results, Simons proposed that memory for objects and for spatial layout are mediated by fundamentally different mechanisms, with successful working memory for objects requiring verbal encoding, but that for spatial layouts requiring automatic encoding without verbal mediation. Simons’s data, however, do not satisfactorily address the present question concerning the role of verbalization in object versus spatial working memory. The problem is that Simons’s data contain a difficulty confound: Performance in object conditions was significantly lower than that in spatial conditions in each of his experiments. His data, therefore, are equally consistent with the alternative view that verbalization strategies in working memory are dependent on task difficulty rather than on stimulus material.
The experiments presented here were designed to test conclusively two related hypotheses: that a verbal code is fundamental to the representation of objects in working memory and that the representation of locations in working memory does not require a verbal code. We used a dual-task procedure for these studies, in which a primary working memory task was performed in parallel with a series of distractor trials. The logic was that interactions between distractors featuring different domains of information (e.g., motion vs. words) and working memory tasks also containing different stimulus information domains would reveal some of the mental codes that support spatial and object working memory performance (Crowder, 1993
; Posner, 1978
We used the n
-back task, a continuous performance working memory task in which participants view the serial presentation of stimuli and judge whether each is a repetition of the stimulus that appeared n
stimuli previously. For example, in a two-back test, the third item of the series A B A is a two-back match, whereas the third item of the series C B A is not. This task is believed to engage several mental operations, including: (1) encoding a stimulus into working memory; (2) maintenance of this mnemonic representation despite the subsequent presentation of additional interfering, attentionally salient stimuli; (3) shifting attention back to this mnemonic representation when necessitated by task contingencies; (4) discriminating between this mnemonic representation and the stimulus on the screen, and guiding behavior with the outcome of this discrimination; and (5) retagging each still-relevant mnemonic representation with a new positional code to reflect the updating of the contents of working memory that must happen with the appearance of each new stimulus (Jonides et al., 1997
; Postle et al., 2000
To simplify the interpretation of predicted primary task–secondary task interactions, we designed this version of the procedure in an interleaved manner such that each distraction trial occurred during the interstimulus interval (ISI) of the n
-back task. That is, the distractor onset occurred after the offset of the preceding n
-back stimulus, and the distractor offset occurred prior to the onset of the next n
-back stimulus. In this way, we sought to limit the effects of distractors on processes (2) and (5) from the previous paragraph. Thus, any interactions between distractor-task and n
-back performance could be ascribed primarily to effects on the maintenance and/or control of working memory representations2
, as opposed to encoding-related or response-related processes.