|Home | About | Journals | Submit | Contact Us | Français|
The modified Stroop effect (MSE), in which participants show delayed color naming to trauma specific words is one of the most widely cited findings in the literature pertaining to cognitive bias in posttraumatic stress disorder (PTSD). The current study used a novel approach (Dissertation Abstract Review; DAR) to review the presence of the MSE in dissertation abstracts. A review of dissertations that used the modified Stroop task in a PTSD sample revealed that only 8% of the studies found delayed reaction times to trauma-specific words in participants with PTSD. The most common finding (75%) was for no PTSD-specific effects in color naming trauma-relevant words. This ratio is significantly lower than ratios found in the peer reviewed literature, but even in the peer reviewed literature only 44% of controlled studies found the modified Stroop effect (MSE). These data suggest that a re-evaluation of the MSE in PTSD is warranted.
Modified versions of the original Stroop (1935) task have been widely used to demonstrate biases in the processing of information in a range of psychological disorders. In these modified Stroop tasks, participants are shown a series of printed words that may or may not be relevant to their psychological disorder. The color that the words are printed in vary, and the participants’ primary task is to name the color the word is printed in. There have been multiple studies, across a range of psychological disorders, that have reported delays in naming the color of words related to aspects of psychological disorders (Williams, Mathews, MacLeod, 1996; MacLeod, 2005). Authors refer to this delay in color naming as the interference or modifed Stroop effect (MSE). It has been argued that the MSE represents an attentional bias (increased selective attention) to the processing of disorder-relevant information or an inability to inhibit disorder-relevant information even when such information is incidental to a task (MacLeod, 2005; McNally, 1998)
Such a bias in information processing, if it exists, is accepted to be a meaningful cognitive aspect of psychopathology. It has been argued, for example, that increased attention to disorder-relevant information could underpin and sustain cognitive biases in depressive, anxiety, and eating disorders (Dagleish & Power, 2004; MacLeod, 2005). In such models, selective attention to disorder-specific information would enhance and prioritize processing of such stimuli and thus further exacerbate the condition. In an individual with depression, for example, an automatic or unsuppressed bias for depressive stimuli would only worsen a negative world view. For these reasons, cognitive biases, such as those demonstrated during the emotional Stroop task, have been proposed to play a role in the etiology and maintenance of a range of psychological disorders.
It is broadly accepted that the modified Stroop effect also exists in posttraumatic stress disorder (PTSD), an anxiety disorder that is precipitated by a psychological trauma and results in symptoms of re-experiencing, avoidance/numbing, and hyperarousal (APA, 2004). There is considerable evidence indicating that individuals with PTSD show a color-naming interference effect to trauma-related words, but not to words unrelated to their trauma (Beck, Freeman, Shipherd, Hamblen & Lackner, 2001; Bryant & Harvey, 1995; Cassiday, McNally, & Zeitlin, 1992; Kaspi, McNally, & Amir, 1995; Foa, Feske, Mordock, Kozak, & McCarthy, 1991; Harvey, Bryant, & Rapee, 1996; McNally, Kaspi, Riemenn, & Zeitlin, 1990; Moradi et al., 1999; Thrasher, Dagleish, & Yule, 1994). Further, such interference is thought to be present in individuals with PTSD but not in individuals who have been traumatized but do not have PTSD—thus demonstrating the specificity of the effect to the disorder (PTSD) and not the traumatic experience itself. These studies, in general, conclude that individuals with PTSD show trauma-specific interference effects representative of attention bias towards trauma related stimuli.
Other studies have produced more mixed results. Dubner and Motta (1999) found delayed color naming in a sexual but not a physical abuse sample. Litz et al. (1996) found interference to trauma words, but also to generally threatening words. Other studies have found no PTSD-related effects in the color naming of trauma words (Devineni, Blanchard, Hickling, & Buckley, 2004; Freeman & Beck, 2000; Naidich & Motta, 2000; Suozzi & Motta, 2004; Shin et al., 2001; Bremner et al., 2004).
Despite decidedly mixed evidence, the general consensus in the literature from the early work to the present seems to be that the Stroop interference is robust in PTSD. For example, Buckley and colleagues (2003) state that a “very reliable phenomenon across numerous studies with traumatized individuals is that patients with PTSD show delayed vocal response latencies to trauma relevant stimuli relative to threat stimuli that are not trauma related (p. 60).” Emelien et al. (2000) state the “use of the emotional Stroop test has shown that trauma-related words are a sensitive measure of clinical state in PTSD (p. 3).” McNally (1998) reports that these “studies have repeatedly confirmed that PTSD participants exhibit more interference for trauma words than for other words, and exhibit more interference for trauma words than do trauma exposed people without the disorder (p. 972).” Likewise, negative effects are often interpreted as being in conflict with the existing literature. For example, Bremner and colleagues (2004) did not find the Stroop interference effect and report “Unlike prior studies, this study did not find a difference in response time to emotional versus color Stroop in PTSD versus control subjects (p. 618).”
This study undertook a novel approach to examining the robustness of the MSE in PTSD. In particular, we reviewed all dissertation abstracts that covered the topics of “Stroop” and “PTSD”—a process we refer to as Dissertation Abstract Review (DAR). Given that all dissertation abstracts are published, DAR has the distinct advantage of avoiding publication bias or the “file drawer effect”—a well-documented tendency for studies with non-significant findings to not make it to press (Rosenthal, 1979; Rosenthal, 1986; Dickersin, 1990). Historically, a number of reasons have been identified for publication bias including the tendency for authors to only submit significant findings for publication, journal editor’s biases towards positive findings, and the fact that commercially sponsored research is more likely to be published if the results are supportive of the product (Gilbody, Song, Eastwood, & Sutton, 2000).
DAR represents the possibility of estimating the “base rate” of a finding prior to the pressures of publication bias. Dissertations, therefore, represent a unique and underutilized recourse for reviewing a literature. Of greatest importance, is the fact that all dissertation abstracts are published by Dissertation Abstracts International© regardless of their outcome or significance. What little selection or review may occur in dissertations will appear at the level of the dissertation committee—a valuable source of quality control that would presumably reject a dissertation for methodological flaws or a serious misunderstanding or misrepresentation of the data.
The process of DAR allows for two estimates of the “file drawer effect.” The first estimate is based on the discrepancy between the “base rates” of an effect in dissertations and the published rates found in the peer reviewed literature. The second estimate is based on tracking the progress of dissertations into the peer-reviewed literature. Not all dissertations make it to press and the percentage of dissertations with “negative” findings that make it to press—particularly in comparison to those with “positive” findings—can be an approximation of the number of studies that get “shelved” based on their outcomes.
In addition to DAR, the study proposes an analysis of the “top drawer effect”—a term coined to refer to the fact that positive findings are more likely to be published in journals with higher impact. Publication bias is only one mechanism that may distort perception on a particular topic. A misperception of an effect may partly arise in that positive findings are more likely to be published in more widely read and widely cited journals. This may lead to a greater sense of an effect’s potency than exists in reality. By comparing the impact ratings of journals that published articles with positive MSE’s in comparison to the impact ratings of journals that published articles with negative MSE’s would give some indication of differential exposure based on outcome.
There are three primary goals to this paper. The first is to establish the “base rate” of the modified Stroop effect in PTSD by evaluating dissertations for the presence of PTSD-specific effects in the emotional Stroop. Second, we will look at the presence of the modified Stroop effect in the peer reviewed literature as an estimate of the pressures applied by the file drawer effect. Finally, we will compare journal impact factors for studies that report positive versus negative MSE findings.
Dissertations abstracts were found for review using PsycInfo® and Medline® and the keywords “PTSD” and “Stroop.” The search was limited to document type “Dissertation Abstract.” There was no limit placed on the years of the search.
The search identified 25 dissertation abstracts that met the initial requirements of the search terms. Of these 25, only 12 ran a modified Stroop task in which traumatized participants with PTSD were compared to traumatized participants without PTSD. The other 13 dissertations were eliminated from further analyses either because 1) they only alluded to the “Stroop” in their Introduction but never used a Stroop task in their dissertation, 2) they did not run the modified Stroop task but rather ran the standard Stroop paradigm, 3) they looked at the modified Stroop in descendants of individuals with PTSD, or 4) they did not have a trauma control group. Of the 12 qualifying studies, a study was considered to support the MSE if participants with PTSD showed significantly longer reaction times to trauma relevant words compared to the non PTSD group. If a dissertation showed a general slowing to all study stimuli, they were classified as not supporting the MSE. If a dissertation had measures of PTSD, used the Stroop but did not report significant group effects, it was also classified as not supporting the MSE. One dissertation reported longer reaction times to all negatively valenced stimuli not just trauma relevant stimuli. This was classified as “No.” Studies were classified as showing “Partial” support if one trauma sample (i.e. sexual abuse) showed the MSE, but a second trauma sample (i.e., physical abuse) did not. Therefore, all studies were classified as “Yes,” “No,” or “Partial.”
The same search criteria were applied to peer reviewed journal articles except no limits were placed on the document type in the literature search. This search produced 41 citations. Eighteen studies remained after applying the same exclusion criteria used for dissertations as to the use of a trauma control group. In addition to the exclusion criteria listed for dissertations, some papers that met the original search criteria were not included because they were review articles and did not present original data.
In addition to comparing the ratio of positive MSEs in dissertations to those in peer reviewed journals, dissertations could also be directly “tracked” from dissertation to publication (if applicable) to see if a disproportionate number of positive MSEs from dissertations were published in peer reviewed journals relative to dissertations with negative MSEs. Only dissertation abstracts published before 2004 were used in the analysis. This is because dissertation abstracts dated 2004 or after were considered too recent and might be counted as “not published” when in fact they may be “in press.” Four of the 12 dissertation abstracts were published after 2004 and thus were excluded from further analyses. This left eight dissertations which could be tracked from abstract to publication. Dissertations were tracked from dissertation to press by using the first author’s last name and the keyword “PTSD” (or “posttraumatic stress disorder”). The document type was limited to “Article.” In total, 5 of the 8 dissertations had been published.
All impact factors and total citation values came from the Science Citation Index published by Thomson publishing.* Each article was assigned an impact factor based on the 2006 impact factor of the journal in which it was published in. Any journal article that did not have the modified Stroop as its central focus (i.e., Stroop was given as part of a larger treatment study or was somehow peripheral to the larger study) was not included in the primary analysis. This is due to the likelihood that the Stroop was fundamentally irrelevant to the “impact” of the study. Two studies were published in low impact journals that did not have impact factors listed. They were arbitrarily assigned an impact factor of .10 and 200 total citations, values that correspond with an average of the last 10 journals listed in the Science Citation Index for Psychology.
Twelve dissertation abstracts, ranging from 1992 to 2007, were eligible for analysis. One of the 12 dissertation abstracts (8.3%) supported a PTSD-specific MSE. Two (16.7%) were classified as “Partial”—both Dubner (1999) and Baxt (2004) found the effect in one sample with PTSD, but not a second. The majority of studies (75.0%) found no group differences between PTSD and no-PTSD participants (see Table 1).
Eighteen journal articles, ranging from 1990 to 2004, were eligible for analysis. Eight of the 18 (44%) supported the MSE, two of 18 (12%) provided partial support, and eight of the 18 (44%) did not support the MSE (see Table 2).
A 2 × 2 chi square analysis (one tailed) was run to analyze whether the observed counts were significantly different from expected counts for “Yes,” and “No” categories for the dissertations and journal articles. Studies that were deemed “Partial” were not included in the analyses. The analysis revealed a statistical difference [χ2(2)=4.35, p<.05] in which dissertations had relatively more in the “No” category relative to the “Yes” category compared to journal articles.
Only five of the 8 (62.5%) of the dissertations were in press at the time of this writing. The one dissertation that found the modified Stroop effect was published (100%). Three of the six dissertations that did not support the MSE were published (50%). The one dissertation to find “Partial” effects was also published (100%).
Two studies were excluded given that modified Stroop reaction times were not central to the study (Bremner et al., 2004; Shin et al., 2001). In both cases, for example, results for the MSE were not reported in the abstract. The eight studies that found the MSE for PTSD had a mean impact rating 3.02 (1.53) and the remaining 6 studies not finding the effect had a mean rating of 1.41 (1.03). A two-tailed independent samples t test revealed this difference to be significant [t(12)=2.21, p<.05). An analysis of total citations from those journals revealed a similar pattern with journal articles reporting positive MSE effects being published in journals more frequently cited (M=6,311.6, SD=3725) than those studies not finding an MSE effect (M=855.2, SD=561.0). This difference was significant as well [t(12)=7.42, p<.01).
This study presents data suggesting that our current understanding of the modified Stroop effect in PTSD may be distorted by a combination of “file drawer” (i.e., publication bias) and a “top drawer” effects. Together, these effects make it likely that a traditional review of the literature may overestimate the presence and robustness of the MSE for PTSD. In fact, our data suggest that the MSE is not a reliable finding in persons with PTSD. Only one of the 12 dissertations found the MSE in PTSD. This rate is only slightly greater than chance. The majority of dissertations found no effect at all. The evidence from the dissertations seems to indicate that the modified Stroop effect is extremely weak or extremely subtle, if it exists at all. Alternative explanations, such as a general negative valence effect, do not emerge from the dissertations either as these effects were equally rare.
One might consider criticizing the source upon which this review was based. Dissertations are almost certainly quite variable in quality, and they undergo a review process that is not anonymous. It is no more possible to confirm the quality and expertise of the dissertation committees than it is to confirm the quality and expertise of anonymous peer reviewers for a journal. However, dissertation committees certainly provide a level of protection against poor quality work. They are typically developed with, supervised by, and ultimately approved by a committee of which at least one member is an expert on the topic and the method. In the PTSD MSE literature, many of the dissertations were supervised by investigators who have peer reviewed publications on the topic. Of obvious importance, dissertations are not subject to the same publication bias as are peer reviewed journal articles, as they are published regardless of the significance vs. non-significance of their findings. Upon completion of the goals outlined in the proposal and a successful defense, all dissertation abstracts are published by Dissertation Abstract International—essentially eliminating the “file drawer effect.” The entire dissertation procedure, from proposal to abstract publication, in fact mimics procedures that have been suggested for the peer reviewed literature that are intended to minimize publication bias (Bornstein, 1990; Dickersin 1994). Finally, the lack of findings in the dissertations is not due to a corresponding lack of power. In fact, the average sample size in the dissertation was 63.9, while the average sample size in the peer reviewed literature was 49.0.
While empirical support for the MSE for PTSD in the peer reviewed literature is considerably stronger than it is in dissertations, it is still decidedly equivocal. Despite a general characterization of the effect as “robust,” only 44% of the published studies support the MSE in PTSD participants as compared to non PTSD participants. The 36% increase over dissertation abstracts suggests the influence of publication bias. Even with such a bias present, the peer reviewed literature still does not overwhelmingly support the MSE. This is somewhat surprising given the general characterization of the strength of the effect in the literature. There may be two factors that influence this. First, it appears that articles that report a positive MSE may be published in higher impact journals and are cited more frequently. Second, there are at least seven studies in the peer reviewed literature that report “positive” MSE’s in PTSD samples, but have no trauma control group to allow for separation of the effects of the trauma itself (Buckley, Blanchard, & Hickling, 2002; Constans, McCloskey, Vasterling, Brailey, & Mathews, 2004; Field et al., 2001; McNeil, Tucker, Miranda, Lewin, & Nordgren (1999); Metzger, Orr, Lasko, McNally, & Pitman (1997); Moradi, Taghavi, Heshat, Yule, & Dagleish (1999); Paunovic, Lundh, & Ost (2002). These studies are often cited as supporting the modified Stroop effect in PTSD. In fact, the following types of designs are frequently reported as support for the MSE despite the lack of a trauma control group: 1) studies with only PTSD subjects that show delayed reaction times to trauma words, but not other types of words; 2) studies in which PTSD participants show delayed reaction times to trauma words relative to individuals with other disorders; and 3) studies that report a positive MSE effect in PTSD participants relative to no trauma/no PTSD control group. While these studies are valuable, the lack of a trauma control group limits what they can conclude with respect to PTSD.
A no-PTSD trauma control group is necessary to argue that the MSE exists in PTSD. In studies without a trauma control group, the differences in reaction time between the trauma/PTSD group and any other groups almost certainly caused by increased attention to a self relevant event (the trauma). Given that self-relevant material garners additional attentional resources, it is an unjustified conceptual leap to say that “positive” MSE’s are caused by PTSD and not by the traumatic experience itself unless appropriately controlled for. This interpretation is supported by the data indicating that carefully controlled dissertations or peer reviewed journal articles do not show much support for a PTSD specific MSE. Certainly “positive” MSE studies appearing in peer review journals may represent the highest quality work. However, DAR suggests that these “positive” studies are only a small percentage of the total studies that examine the MSE in PTSD and may be the studies, simply by chance, that happened to find the effect.
Non significant MSE findings were sometimes published, but were published as adjuncts to a larger study that had something else “worthy” of publication. Bremner et al. (2004) and Shin et al. (2001) embedded the Stroop within studies that investigated anterior cingulate activation using a functional MRI. Devineni et al. (2004) reported their non significant MSE as part of a larger treatment study. Others reported trauma (not PTSD) related MSE’s, overall slowing, or trauma (not PTSD) related MSE’s in children of trauma survivors (Naidich & Motta, 2000; Souzzi & Motta, 2004; Vrana, Roodman, & Beckham, 1995). In many of these studies, the non significant Stroop findings were not a major part of the manuscript, thus minimizing the salience of these “negative” modified Stroop findings.
Analyzing the attrition rate from dissertation to publication was not effective in furthering understanding for the file drawer effect for the MSE. Because only one dissertation found the MSE, there was no good basis for comparison between positive and negative studies. In addition there were only six dissertations that had been completed prior to 2004 that did not find the MSE. This small and one sided sample can not provide a fair estimate of the file drawer effect from dissertation to publication.
The above paragraph highlights one of the limitations of this study—there were only 12 dissertations that met study criteria. The case of the MSE in PTSD may have tested the limits of DAR’s utility. Only one of the 12 dissertations found a positive effect thus making comparisons to studies that found negative effects impossible. Similarly, the inclusion and exclusion criteria, as well as the operational definitions for “positive,” “negative,” and “partial,” had an impact on the review. For example, McNally, Amir, and Lipke (1996), found a complex effect in which reaction times were affected by stimulus type and time. Litz et al. (1996) found delayed reaction times in PTSD relative to No PTSD military veterans to all high threat words including military words. Some might argue these studies support the MSE in PTSD, but were classified here as “No MSE.” This effectively lowered the overall rate of the MSE in the peer reviewed literature, but at the same time worked against the prediction relevant to the file drawer effect—that there would be higher rates of MSEs in the peer reviewed studies than in the dissertations.
The findings presented here should not be generalized to other anxiety disorders, and it is not clear to what extent a similar pattern exists in parallel literatures. The MSE has been found, with varying reliability, in generalized anxiety disorder, depression, phobia, and obsessive compulsive disorder. This study can not speak to the robustness of the MSE in those literatures, however biases in publication are pervasive suggesting DAR is warranted in those literatures as well.
As noted above, it is commonly accepted that individuals with PTSD have an underlying, automatic attentional bias that may play a role in the etiology and maintenance of cued symptoms in PTSD such as intrusive memories, flashbacks, avoidance, and psychological and physiological reactivity. The MSE has been considered to be at the core of the empirical work that has supported such a bias. While there may be support for information processing attention bias present in other experimental paradigms, it is not clear that the current MSE literature can support such an assertion. In fact, the MSE data casts doubt about whether trauma-relevant material automatically interferes with primary task activity in people with PTSD any more than it does for trauma survivors without PTSD.
As the field of PTSD matures it is having to grapple with a number of conceptual and phenomenological questions central to our understanding of posttraumatic reactions (McNally, 2003; Rosen & Frueh, 2007; Spitzer, First, & Wakefield, 2007). The MSE in PTSD seems to be an issue that warrants reconsideration. Reevaluating certain findings in PTSD, or in the broader literature, can be assisted through the process of DAR. As evidenced by the review of the MSE in PTSD, DAR point toward a very different conclusion than a traditional literature review or a meta-analysis based on published studies. A traditional literature review on the MSE would result in the comments like those frequently found in the literature today: that the MSE is a common finding in PTSD. In combination with DAR however, one arrives at a very different conclusion: the MSE effect in PTSD may be exaggerated by a pronounced publication bias and a top drawer effect.
*An impact factor is reported by Thomson publishing as … “a measure of the frequency with which the “average article” in a journal has been cited in a particular year or period” (retrieved from http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor/ on July 24, 2007).
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.