PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Dev Cogn Neurosci. Author manuscript; available in PMC 2014 April 1.
Published in final edited form as:
PMCID: PMC3617931
NIHMSID: NIHMS455260

Preliminary data suggesting the efficacy of attention training for school-aged children with ADHD

Abstract

A pilot randomized clinical trial was conducted to examine the initial efficacy of Pay Attention!, an intervention training sustained, selective, alternating, and divided attention, in children diagnosed with Attention-Deficit/Hyperactivity Disorder (ADHD). After a diagnostic and baseline evaluation, school-aged children with ADHD were randomized to receive 16 bi-weekly sessions of Pay Attention! (n = 54) or to a waitlist control group (n = 51). Participants completed an outcome evaluation approximately 12 weeks after their baseline evaluation. Results showed significant treatment effects for parent and clinician ratings of ADHD symptoms, child self-report of ability to focus, and parent ratings of executive functioning. Child performance on neuropsychological tests showed significant treatment-related improvement on strategic planning efficiency, but no treatment effects were observed on other neuropsychological outcomes. Treatment effects were also not observed for teacher ratings of ADHD. These data add to a growing body of literature supporting effects of cognitive training on attention and behavior, however, additional research is warranted.

Keywords: ADHD, Attention training, Cognitive training, Executive functioning, Intervention

1. Introduction

Attention training, also known as cognitive training, is an intervention in which the various components of attention are viewed as skills that can be enhanced by training. Attention in this conceptualization is defined as the appropriate allocation of processing resources to relevant stimuli and is thought to comprise several sub-processes (Coull, 1998). Specifically, attention can be fractionated into (1) attention orientation (direction of attention to a stimulus), (2) selective attention (prioritizing one stimulus in favor of another), (3) sustained attention (attending to one stimulus over time), and (4) divided attention (dividing attention between two plus stimuli) (Coull, 1998). These sub-processes are thought to be the building blocks underlying executive functioning and to have unique neuronal correlates (Coull, 1998). Theoretically, more efficient attentional sub-processes should give rise to enhanced executive processes (Paelecke-Habermann et al., 2005), such as inhibition, task management, planning, monitoring/working memory, and coding. Further, more efficient executive functioning should translate into improvements in behavior – for example, improved inhibition might lead to better self-regulation.

The rationale for attention training is based on the concept that efficiency increases after repetitive practice of specific cognitive operations of attention (Posner and Raichle, 1994), theoretically because practice produces adaptations in the underlying neuroanatomical networks linked to these processes (Kerns et al., 1999). Initial neuroimaging research comparing brain activation before and after different forms of cognitive training supports this contention (Beauregard and Levesque, 2006; Brefczynski-Lewis et al., 2007; Erickson et al., 2007; Kim et al., 2008; Thimm et al., 2006). Furthermore, controlled trials investigating the efficacy of attention training programs report gains in both trained and untrained assays of attention components such as sustained attention and executive function (Ethier et al., 1989; Finlayson et al., 1987; Gray and Robertson, 1989; Mateer and Mapou, 1996; Penkman, 2004; Sohlberg and Mateer, 1987; Thompson and Kerns, 1995).

In the last decade, researchers have turned to investigate attention training as an intervention for Attention-Deficit/Hyperactivity Disorder (ADHD) since ADHD involves impairment in attention and related executive functions. There is a growing literature providing support for attention training in individuals with ADHD (Epstein and Tsal, 2010; Kerns et al., 1999; Klingberg et al., 2002, 2005; O'Connell et al., 2006; Semrud-Clikeman et al., 1999; Shalev et al., 2004; Tamm et al., 2007, 2010; Williams, 1989). These studies report improvements in the cognitive skill being trained directly but also improvements in untrained cognitive skills (Epstein and Tsal, 2010; Tamm et al., 2010; Williams, 1989), although effect sizes are typically small to moderate (Epstein and Tsal, 2010). Furthermore, studies of attention training in ADHD also provide initial support for improvements on untrained measures of attention and academic efficiency (Semrud-Clikeman et al., 1999; Williams, 1989), as well as reductions (although not consistently) in teacher and parent ratings of ADHD symptoms (Kerns et al., 1999; Klingberg et al., 2005; Semrud-Clikeman et al., 1999; Shalev et al., 2004; Tamm et al., 2010; Thomson et al., 1984) and restlessness/head movements (Klingberg et al., 2002). There is also preliminary neuroimaging data to suggest that cognitive training impacts brain function in ADHD (Hoekzema et al., 2010, 2011). After 10 days of cognitive training, children with ADHD showed increases in orbitofrontal, superior frontal, middle temporal, and inferior frontal cortex on an inhibition task and increased cerebellar activity on an attentional task, while the control group showed no such activation differences (Hoekzema et al., 2010). Furthermore, these activation differences were complemented by focal volumetric gray matter increases in bilateral middle frontal cortex and right inferior–posterior cerebellum (Hoekzema et al., 2011). Taken together, these studies suggest that attention training is a promising treatment approach for ADHD. However, to date there have been few randomized clinical trials investigating attention training in ADHD. This is critical given that replication studies have not always been successful and, more critically, studies utilizing blinded evaluators do not always find a positive effect for attention training.

One potentially promising intervention is Pay Attention! (www.lapublishing.com), an intervention targeting sustained, selective, divided, and alternating attention, constructs which have been shown to be compromised in individuals with ADHD (Manly et al., 2001). A small randomized trial of the effects of Pay Attention! in seven children with ADHD reported significant benefits of attention training in measures of sustained attention, executive functioning, and effortful processing compared to children in a video game control group (Kerns et al., 1999). However, the sample size for this study was small and participant diagnosis was not verified by the research team. We conducted an open trial of Pay Attention! with 23 children diagnosed with ADHD and showed effects on executive functioning measures (cognitive flexibility, working memory, parent ratings), as well as improvements in ADHD symptomatology (Tamm et al., 2010). However, we did not utilize a control group as the primary emphasis was to determine feasibility of the intervention in a clinically diagnosed group and in our outpatient setting.

The current trial was initiated to investigate if the Pay Attention! intervention was effective in improving trained and untrained executive functions as well as parent, teacher, and clinician ratings of attention and behavior when compared to a waitlist control group that did not receive the intervention. Consistent with the theoretical underpinnings of attention training, we hypothesized that by training attention (sustained, selective, divided, and alternating), we would see proximal treatment-related improvements in executive functioning and distal treatment-related improvements in attention and behavior (see Fig. 1).

Fig. 1
Model of training targets and outcomes.

2. Method

The study was approved by the University of Texas, Southwestern Medical Center at Dallas Institutional Review Board and informed parental consent and participant assent were obtained from all participants prior to initiating any procedures.

2.1. Participants

Participants (n = 132) were recruited from outpatient clinics at Children's Medical Center at Dallas, the community, and Shelton School, a private school for learning differences; 105 participants were randomized (see Fig. 2, CONSORT diagram). Participants ranged in age from 7 to 15 years (M = 9.3, SD = 1.4) and were predominantly Caucasian. Demographic and baseline characteristics are presented in Table 1. Randomization was stratified by gender, ADHD subtype, and medication status. Participants were asked to maintain a stable medication status (i.e., continue taking medication as directed and not initiate new medication during the study). Exclusion criteria included the following: estimated full scale IQ < 85, history of head injury, history of prenatal drug exposure, diagnosis with other congenital or acquired neurological conditions, and participating in other non-pharmacological treatment interventions for ADHD (e.g., neurofeedback, cognitive-behavioral therapy, etc.).

Fig. 2
CONSORT diagram.
Table 1
Demographic and baseline characteristics.

2.2. Design and procedure

Families meeting eligibility criteria by phone screen were invited to participate in a baseline evaluation. A semi-structured clinical interview, the Kiddie-SADS-Present and Lifetime Version or K-SADS-PL (Kaufman et al., 1997), was conducted with the primary caregiver, and with the child/adolescent separately, to determine ADHD diagnosis. Information regarding family functioning, family psychiatric history, and developmental and medical histories was also collected. In addition, the clinical interviewers provided ratings of ADHD on the Swanson, Nolan, and Pelham DSM-IV ADHD rating scale (SNAP-IV) based on the interviews, and impairment/functioning using the Clinical Global Impressions (CGI) rating scale. Parents completed several rating scales assessing attention, executive function, and behavior, and participants were administered several tests assessing a variety of executive functions including planning, behavioral inhibition, visual–spatial abilities, and working memory.

Following the baseline evaluation, participants were randomized to either receive the intervention or to a waitlist control group. Participants randomized to the intervention attended twice-weekly 30-min sessions for 8 consecutive weeks (for a total of 16 sessions). Individuals randomized to the waitlist control condition were asked to not begin any new treatment for ADHD during the wait period and were offered the opportunity to receive the intervention at the end of the wait period. All participants and their parents were invited to attend an outcome evaluation. There was variability between subjects as to how long it took them to complete the 16 treatment sessions due to parent and interventionist schedules; thus we opted to re-test all participants on the same schedule (i.e., 12 weeks after baseline). During the outcome evaluation, parents and children were re-interviewed using the K-SADS-PL, clinicians completed SNAP-IV and CGI improvement ratings, and participants completed the same battery of executive functioning measures as at baseline. Parents were also given a packet of ratings to give to teachers to complete at baseline and outcome, although few teachers returned the forms (n = 33 at baseline and n = 27 at outcome for the intervention group; n = 25 at baseline and n = 27 at outcome for the waitlist control group).

2.3. Intervention

The Pay Attention! materials are designed to train sustained, selective, alternating, and divided attention using visual and auditory stimuli. The visual stimuli include a set of cards depicting drawings of children and adults that can be distinguished by various features including age, gender, hair color, and other physical qualities; and a set of home layouts that include several rooms with objects that can be sorted by color, shape, and other characteristics. The auditory stimuli include lists of words played on a CD with the participant required to press a buzzer whenever a specific word is heard. The tasks become progressively more difficult (e.g., a distracting overlay is placed over the visual stimuli, a distracting sound is played during task completion, or participants are asked to complete two tasks simultaneously). All participants completed an initial session to establish performance levels and orient them to the materials. Participants then progressed through the four modules, beginning with the simplest sustained attention tasks. After criterion was reached (e.g., gains in speed while maintaining overall accuracy), the next module was started. Not all participants completed all four modules since they progressed at different rates, although the majority completed at least the sustained and selective attention modules. Participants were given immediate feedback regarding their performance and interventionists spent time each session discussing how the targeted attentional skill could be applied in a home or school setting. Pay Attention! is designed to be flexibly delivered so that the interventionist can tailor the treatment to the specific needs of the participant. Parents were also provided with reading materials about the attention skills being trained and met with the child and interventionist for a few minutes after each session to discuss the activities practiced at each session, which skill was being trained, and how parents could support the child's implementation of the skill in home and school activities.

2.4. Measures

2.4.1. Rating scales

  • Swanson, Nolan, and Pelham (SNAP-IV) DSM-IV ADHD Rating Scale (Swanson, 1992): Clinicians (based on parent and child interview), parents and teachers rated how well each ADHD symptom described the child on a four point Likert scale (0 = not at all, 1 = just a little, 2 = quite a bit, 3 = very much). We examined average scores for Inattention symptoms and Hyperactivity/Impulsivity symptoms. Although psychometric properties have not been investigated for the SNAP-IV, adequate reliability and validity has been established for similar DSM-IV checklists [e.g., ADHD Rating Scale-IV (DuPaul et al., 1998)].
  • Behavioral Assessment System for Children, Second Edition (BASC-II) (Reynolds and Kamphaus, 2004): Parents and teachers completed the BASC-II describing a variety of behaviors. For analytic purposes, we analyzed subscales relevant to ADHD (i.e., Externalizing, Hyperactivity, Attention Problems and Behavioral Symptoms Index). The scale has been demonstrated to have adequate reliability and validity (Reynolds and Kamphaus, 2004).
  • Clinical Global Impressions (CGI) (Leon et al., 1993): Un-blinded clinicians rated the severity of the child's impairment related to ADHD at baseline and outcome on an 7-point scale ranging from Not At All Ill to Very Severely Ill, and the level of improvement at outcome on an 7-point Likert scale ranging from Very Much Improved to Very Much Worse (4 being “no change”). The scale has adequate reliability and validity (Leon et al., 1993).
  • Attentional Control Scale (ATTC) (Derryberry and Reed, 2002): Participants rated themselves on their ability to focus and shift their attention on the ATTC. The ATTC consists of 20 items that are rated on a four-point Likert scale from 1 (almost never) to 4 (always). The ATTC yields two primary scales (attention shifting and attention focusing). This measure was added later in the study, and thus was only available for 20 participants. Analyses of content, internal and construct validity as well as reliability provided evidence of the scale's significant convergent and discriminant validity (Fajkowska and Derryberry, 2010).
  • Behavior Rating Inventory of Executive Function (BRIEF) (Gioia et al., 2000): Parents and teachers completed this rating scale assessing executive function behaviors in the home and school environments, yielding T-scores on several subscales, including Inhibit, Shift, Working Memory, Emotion Regulation, Organization, Planning, Initiate, Monitor, Behavioral Regulation Index, Metacognitive Index, and a General Executive Composite. The scale has adequate reliability and validity (Gioia et al., 2000).

2.4.2. Neuropsychological measures

  • Test of Everyday Attention for Children (TEA-Ch) – The TEA-Ch is an objective measure of attention (Manly et al., 2001). The battery has alternate forms (A and B) to allow for re-test. Subtests utilized for this study included: Sky Search, Score!, Creature Counting, Sky Search DT, Score DT, and Code Transmission. The TEA-Ch was designed to more directly assess sustained, alternating, selective, and divided attention constructs while at the same time, minimizing demands on memory, reasoning, task comprehension, motor speed, verbal ability, and perceptual ability (Manly et al., 2001). Thus, the TEA-Ch tasks more directly map onto the constructs being trained in Pay Attention! and was included as an assessment of whether the intervention was successful in training the desired target (and thus was only collected for the intervention group). Form A of the TEA-Ch has adequate reliability although Form B is not as well established; validity studies are lacking (Strauss et al., 2006).
  • Wechsler Intelligence Scale for Children – Fourth Edition (WISC-IV) (Wechsler, 2004): Matrix Reasoning (fluid reasoning/cognitive flexibility), Digit Span and Letter Number Sequencing (working memory). Vocabulary and Block Design were administered to provide an estimate of Full Scale IQ (Sattler and Dumont, 2004) at baseline only. Psychometric properties for this measure are well established (Wechsler, 2004).
  • Woodcock Johnson Tests of Achievement – Third Edition (WJ-III) (Woodcock et al., 2001): Understanding Directions (auditory working memory, listening comprehension). The subtest has high reliability (>.8) (Woodcock et al., 2001).
  • Delis–Kaplan Executive Functioning System (D-KEFS) (Delis et al., 2001): Tower Test (problem solving) and Color-Word Interference (CWI; inhibition/cognitive flexibility). Additional measures generated from the Tower subtest include Mean First Move Time (initial planning time as measured by the time spent planning the first move), Time Per Move Ratio (overall planning time as measured by the time spent planning all moves subsequent to the first move), Move Accuracy Ratio (accuracy and planning efficiency of problem solving strategy as measured by the total number of moves), and Rule Violations Per Item Ratio (establishing and maintaining cognitive set as measured by the number of rule violations). Reliability of the DKEFS subtests is in the moderate to high range and validity has been established with a number of executive functioning measures (Delis et al., 2004).
  • Quotient ADHD system – The Quotient is a continuous performance test assessing a participant's ability to pay attention which also objectively measures a patient's ability to sit still (Sumner, 2010). Children are asked to press a button each time they see an eight-sided star but to withhold their response to five-sided stars. A motion-tracking device directed at sensors on the ankles and forehead records movement simultaneously with performance of the task. Variables analyzed included omission errors (percentage of missed targets), commission errors (percentage of inaccurate responses to non-targets), latency (average amount of time to respond correctly), and variability (variation in response time to the correct target). Reliability and validity for this measure has not yet been established, although the task is very similar to the Connor's Continuous Performance Test which has adequate reliability and validity (Conners and Jeff, 1999).

2.5. Analytic strategy

Demographic and baseline clinical characteristics of the two groups were reported using mean (SD) for continuous variables and frequency (percentage) for categorical variables. To identify any differences between the characteristics of the 2 groups, we used the 2-independent sample t test for continuous variables and the χ2 test for categorical variables.

As a manipulation check of whether the Pay Attention! intervention impacted sustained, selective, divided, and/or alternating attention, we conducted repeated measures ANOVAs in SPSS for each of the 6 subtests administered of the TEA-Ch (note, alternate forms were used for this measure) for the intervention group only; this measure was not available for the waitlist control group.

To address whether the intervention improved behavioral and executive functioning ratings, and child performance on executive function tasks, multivariate and univariate generalized estimating equation (GEE) analyses were performed to test for significant mean differences between the treatment intervention and waitlist control groups at outcome using baseline scores and participant age (for non-normed measures, e.g., SNAP-IV, CGI) as control covariates (e.g., Feldon et al., 2011). The baseline scores were included as covariates to address potential group differences at baseline contributing to the pattern of findings at outcome. Multivariate GEE analyses were used to test for group mean differences among correlated subtests from the same response variable measure (e.g., BRIEF subscales), while univariate GEE analyses were used to test for mean differences among summative subscales or total scores from a response variable measure separately to avoid collinearity (e.g., BRIEF Index scores). These GEE analyses were conducted using Mplus version 6.12 (Muthén and Muthén, 1998–2010) to: (a) handle missing data via maximum likelihood estimation assuming missing at random (MAR) (e.g., Enders, 2010), (b) test for significant response variable mean differences directly using the `Model Constraint' and `New' commands, and (c) ensure that analysis assumptions of homogeneity of covariance matrices and homogeneity of covariate regression slopes were met by imposing the necessary parameter estimate constraints prior to analysis. Testing for group differences on the ATTC attention focusing and attention shifting scales was done via separate analyses of covariance in SPSS (baseline scores and age were entered as covariates) due to the reduced sample size (N = 20) available for those analyses. Given the number of response variable mean difference computations involved, the false discovery rate procedure (Benjamini and Hochberg, 1995) was used to ensure that no more than 5% of the total number of significant findings reported represent Type-I errors. The false discovery rate has been shown to produce fewer Type-I errors than per-comparison critical alpha corrections, and has been shown to better maintain statistical power compared to experiment-wise critical alpha corrections (Benjamini and Hochberg, 1995; Maxwell and Delaney, 2004). The p-values listed in Tables 24 are those generated from the false discovery rate procedure and are thus labeled “corrected p-value”. We also report effect sizes which can be particularly illuminating when there is limited power (e.g., teacher ratings, child self-report ratings), and where one wants to compare them with other research in the literature. Cohen's d effect size estimates were computed for all group mean differences.

Table 2
Intervention group results on Test of Everyday Attention for Children (TEA-Ch).
Table 4
Group differences at outcome: parent and teacher ratings of executive functioning.

Finally, additional analyses were conducted to investigate if there were any moderators of treatment response after controlling for baseline score variation. Specifically, for each response variable, separate treatment group by (1) age, (2) gender, (3) IQ, (4) ADHD subtype, and (5) medication status interaction variables were computed and tested for significance. Results of these analyses were also corrected for Type-1 error inflation via the false discovery rate procedure.

3. Results

3.1. Manipulation check

Compared to their pre-intervention performance, the group that received intervention improved significantly on the TEA-Ch Score!, Sky Search Completion Time and Attention (combined accuracy and timing score), and Creature Counting Completion Time after the intervention (Table 2).

3.2. Parent, clinician, and teacher behavioral ratings

Parents and clinicians reported significantly fewer ADHD symptoms on the SNAP-IV Inattention and Hyperactivity/Impulsivity ratings for the intervention group compared to the waitlist control group (Table 3). On the BASC-II, parents rated children in the intervention group as having fewer Attention Problems compared to the waitlist control group. Teacher ratings were not significant for the SNAP-IV or BASC-II. Clinician ratings on the CGI indicated lower severity and greater improvement for the intervention group than the waitlist control group. Participants rated themselves as having significantly improved ability to focus their attention [F(3, 19) = 6.6, p < .05, Cohen's d = 1.11] and shift their attention [F(3, 19) = 5.5, p < .05, Cohen's d = .51] on the ATTC.

Table 3
Group differences at outcome: parent, clinician and teacher behavior ratings.

Examination of effect sizes revealed that teachers were also reporting reductions in Hyperactivity/Impulsivity ratings (medium effect) for the intervention group compared to the control group. On the BASC-II, both parents and teachers reported improvements on the Externalizing, Hyperactivity, and Behavioral Symptoms Index, and teachers reported improvements on the Attention Problems subscale, for the intervention group compared to the control group. These effect sizes ranged from small to large (Table 3).

3.3. Parent and teacher ratings of executive functioning

Significant improvements were reported by parents for all the BRIEF subscales, with the exception of Inhibit (trend for improvement) and Emotion Regulation for the intervention group compared to the waitlist control group (Table 4). In contrast, no significant effect of treatment was observed for any subscale on the teacher BRIEF ratings.

Examination of effect sizes for the teacher ratings revealed improvements for the intervention group compared to the waitlist control group on the teacher rated BRIEF Shift, Emotion Regulation, Initiate, Working Memory subscales, and Behavioral Regulation Index, and a decrement (moderate effect size) for Organization (Table 4), though none of these effects reached statistical significance.

3.4. Child measures of neuropsychological functioning

Children in the intervention group performed significantly better than the waitlist control group on the DKEFS Tower Time per Move Ratio (Table 5). No other significant differences between the intervention and waitlist control group were observed on the other neuropsychological measures.

Table 5
Group differences at outcome: child performance on executive functioning measures.

Examination of effect sizes revealed a small effect favoring the intervention group on the DKEFS Inhibition, DKEFS Inhibition/Switching Errors, DKEFS Tower Mean First Move Time, DKEFS Tower Move Accuracy Ratio, WISC-IV Matrix Reasoning, Quotient Latency, Quotient Variability, and WJ-III Understanding Directions measures, but these did not reach statistical significance. The intervention group also made more errors of omission on the Quotient compared to the waitlist control group.

3.5. Potential treatment moderators

There were no significant interactions between group and IQ, ADHD subtype, medication status, or gender. However, we observed an interaction between age and group on the BRIEF Shift subscale (b = 3.33, Wald Z = 3.26, p < .01), BRIEF Behavioral Regulation Index (b = 1.95, Wald Z = 2.67, p < .01), and the CGI Severity rating (b = .23, Wald Z = 2.56, p < .01) at outcome. For all three interactions, older members of the treatment group had lower scores on those response variables.

4. Discussion

The results of this pilot randomized clinical trial of Pay Attention! in children with ADHD suggest that the intervention was successful in impacting attentional sub-processes targeted by training (i.e., we observed improvement on tasks assessing aspects of sustained, selective, divided and alternating attention like visual detection, response speed, etc.). Further, our data show that the intervention improved ADHD symptomatology by parent and clinician report, reduced impairment by clinician report, improved executive functioning including inhibition, shifting, planning, and self-monitoring by parent report, improved child self-report of ability to focus and shift attention, and improved child performance on a task measuring planning efficiency. Although non-significant, examination of effect sizes revealed small to moderate improvements for the intervention group on teacher ADHD and executive function (Shift, Emotion Regulation, Initiate, and Working Memory) ratings, and child performance on executive function measures of inhibition, planning, comprehension and memory of verbal instructions, cognitive flexibility, and response latency. As with other studies, these promising findings are tempered by the fact that we did not have blinded evaluators, parents may have been affected by expectancy bias or Hawthorne Effect, and the vast majority of the most objective measures administered to both groups (child neuropsychological tests) were not statistically significant despite having small to moderate effect sizes. Further, teacher ratings were not significant.

Although not statistically significant in most cases, the direction of group means were, however, generally consistent with those of the previous small randomized clinical trial of Pay Attention! (Kerns et al., 1999) which reported gains on measures of planning, sustained auditory attention, sustained visual attention, behavioral inhibition/executive function/selective attention, and academic efficiency. There was also a trend observed for improved teacher ADHD ratings. In this trial, which utilized different assessment instruments, we also showed significant improvements on planning efficiency, and non-significant gains in behavioral inhibition/executive function/selective attention, and auditory attention/memory. It should be noted that Kerns (Kerns et al., 1999) utilized raw scores in her analyses to address issues related to not being able to detect treatment effects due to a child remaining in the same “normative” group and to detect a child's own gains compared to their own performance. We opted to use scaled scores allowing us better ability to compare the current results to those on the open trial which utilized the same measures.

The findings from our RCT are also essentially consistent with attention, behavior, and executive functioning improvements we observed in our open trial (Tamm et al., 2010). While we did not replicate our previous findings of significantly improved cognitive flexibility on the DKEFS Inhibition/Switching subtest with the Pay Attention! intervention (Tamm et al., 2010), we did observe greater improvement on the WISC Matrix Reasoning and DKEFS Inhibition subscales for the intervention group compared to the waitlist group (small effects). Thus, we still see evidence of enhanced fluid reasoning or cognitive flexibility with Pay Attention! which may not have achieved significance due to the relatively small sample size. The improvement in these domains, if replicated in future studies, is clinically relevant for individuals diagnosed with ADHD who have neuropsychological deficits in these constructs which are associated with impaired brain functioning (Tamm and Juranek, 2012).

Despite significant improvements on parent ratings of working memory, improvements on child working memory subtests were not observed. There was, however, a small, non-significant effect of treatment on the WJ-III Understanding Directions subtest which does involve a working memory component in that it requires listening and mapping a series of sequential directions onto the mental structure under construction and maintaining the sequence in immediate awareness until a new directive changes the sequence (Gernsbacher, 1991). It may be that Pay Attention! improves listening abilities and selective attention more than the construct of working memory.

Analysis of processing variables necessary for DKEFS Tower subtest completion revealed a significant effect on the Time Per Move Ratio. The Time Per Move Ratio represents the mean amount of time that was spent on each move – it is a ratio of the total number of seconds spent solving a problem relative to the total number of moves used for that problem (Yochim et al., 2009). Interestingly, the intervention group also significantly improved on the TEA-Ch subtests involving timing (Sky Search, Creature Counting), which, taken together with small non-significant improvements on Mean First Move Time (speed) and increased latency (i.e., slowing) on the Quotient continuous performance subtest, suggests that one impact of Pay Attention! training is on the child's efficiency and timing. Certainly, the intervention itself included a significant focus on timing, with children encouraged to improve speed and efficiency for each subsequent administration of the various tasks. This finding, if replicated, could have significant implications for reducing impairment in ADHD, since recent research suggests that individuals with ADHD have specific deficits in timing functions which are postulated to be closely associated with impulsiveness, defined as premature, impatient and delay averse with associated lack of temporal foresight (Hart et al., 2012; Zelaznik et al., 2012).

Analyses investigating potential treatment moderators were not significant for gender, IQ, medication status, or ADHD subtype. Age interacted with group on two parent-rated variables (BRIEF Shift and Behavioral Regulation Index) and one clinician-rated variable (CGI) but not with any of the child neuropsychological variables. We controlled for age in the primary analyses for the CGI but not for the BRIEF which is an age-normed measure. It is tempting to conclude that older children may have benefited more from the intervention, but given that this effect was observed on so few variables, it may be that these few variables are more sensitive to the effects of maturation.

One of the issues with psychosocial interventions which has not received much attention in the literature is whether treatment can have emanative (Whalen et al., 1985) or unintended negative effects. It is often assumed that behavioral or cognitive interventions are either benign or have no impact, versus potentially negatively impacting outcomes. Although the findings were not statistically significant, there was a small effect for the intervention group to make more errors of omission on the Quotient continuous performance task and a moderate non-significant effect for the intervention group to be rated as less organized by teachers than the waitlist control group. Thus, it appears that despite potentially improving inhibition and planning efficiency, the Pay Attention! intervention might be having an unintended negative effect in that the child might be slowing too much and therefore failing to respond quickly enough resulting in somewhat increased errors of omission – again, this effect was not statistically significant but warrants attention. Replication is necessary, given the large number of statistical tests and possibility of chance findings.

There are a number of strengths to this study including that it is amongst the largest randomized clinical trials investigating attention training in ADHD. However, the sample size was still relatively limited, at least for detecting effects on the neuropsychological variables. It should be noted that when we did a post hoc power analysis to examine what sample size would be needed to reject the null hypothesis for the neuropsychological measures, necessary sample sizes were much larger than our sample size (e.g., DKEFS Inhibition, N = 314; WISC-IV Matrix Reasoning, N = 178). The use of a waitlist control group and the resulting lack of blinding to condition could have resulted in a Hawthorne effect or inflated ratings by parent, child, and clinician; replication is necessary with an attention control group. Another limitation was the relative lack of teacher data (approximately 50% of the sample had teacher data available for outcome). Finally, we observed significant variability on some outcome variables suggesting the possibility that participants differed in their response to treatment. Further work is necessary to investigate individual differences in response to treatment and to identify potential moderators.

Overall, our study adds to the growing body of literature suggesting cognitive training may improve executive function and behavior in children with ADHD. However, before Pay Attention! can be considered for use in clinical settings additional research is needed. The optimal design would be an RCT which includes a larger sample of, ideally un-medicated, children with ADHD, blinded evaluators, a control group that is designed to reduce expectancy biases, more teacher ratings, and tests that are selected to measure both proximal (areas trained) and distal (generalization to functional impairment) gains. Nonetheless, our results suggest that Pay Attention! did improve sustained, selective, and alternating attention directly, and specifically timing efficiency. Further, it may be effective in reducing ADHD symptoms as well as improving parental ratings of executive functioning. Effect size examination reveals small although non-significant improvements on teacher ratings (small n) and child neuropsychological measures of executive functioning including planning, cognitive flexibility, inhibition, auditory working memory, and response latency/variability. Caution is warranted, however, given the large number of statistical comparisons conducted.

Acknowledgements

This research project was supported in part by the National Center for Research Resources, National Institutes of Health Grant Number UL1 RR024982. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. Additional funding for the project was provided by a gift from the Sparrow Foundation for the Center for Advanced ADHD Research, Treatment, and Education (CAARTE). We gratefully acknowledge the Pay Attention! interventionists, Aleksandra Foxwell, Jarrette Moore, Lauren Smith, Jeanne Rintelmann, Amanda Gray, Laure Ames, Maryanne Hetrick, Amy Rollo, Cathy Bass, Ana Arenivas, Deidre Edwards, Sarah Swart, Shelley Williamson, Gina Bolanos, and Kyle Clayton, and Conrad Barnes for his data management. We also thank Joyce Pickering, Hum.D. for her support and the use of space at the Shelton School. We appreciate the families who participated in the evaluations and intervention.

Footnotes

Conflict of interest All authors declare that there are no actual or potential conflicts of interest including any financial, personal or other relationships with other people or organizations within 3 years of beginning the submitted work that could inappropriately influence, or be perceived to influence, their work. The funding sources did not have a role in study design, collection, analysis and interpretation of data, writing of the manuscript, nor the decision to submit the article for publication.

References

  • Beauregard M, Levesque J. Functional magnetic resonance imaging investigation of the effects of neurofeedback training on the neural bases of selective attention and response inhibition in children with attention-deficit/hyperactivity disorder. Applied Psychophysiology and Biofeedback. 2006;31:3–20. [PubMed]
  • Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. (Series B).Journal of the Royal Statistical Society. 1995;57:289–300.
  • Brefczynski-Lewis JA, Lutz A, Schaefer HS, Levinson DB, Davidson RJ. Neural correlates of attentional expertise in long-term meditation practitioners. Proceedings of the National Academy of Sciences of the United States of America. 2007;104:11483–11488. [PubMed]
  • Cohen J. A power primer. Psychological Bulletin. 1992;112:155–159. [PubMed]
  • Conners K, Jeff JL. Compact Clinicals. Kansas City: 1999. ADHD in Adults and Children: The Latest Assessment and Treatment Strategies.
  • Coull JT. Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology. Progress in Neurobiology. 1998;55:343–361. [PubMed]
  • Delis DC, Kaplan E, Kramer JH. Delis Kaplan Executive Function System: Examiner's Manual. Psychological Corporation; San Antonio: 2001.
  • Delis DC, Kramer JH, Kaplan E, Holdnack J. Reliability and validity of the Delis–Kaplan Executive Function System: an update. Journal of the International Neuropsychological Society. 2004;10:301–303. [PubMed]
  • Derryberry D, Reed MA. Anxiety-related attentional biases and their regulation by attentional control. Journal of Abnormal Psychology. 2002;111:225–236. [PubMed]
  • DuPaul GJ, Power TJ, Anastopoulos AD, Reid R. ADHD Rating Scale-IV: Checklists, Norms, and Clinical Interpretation. Guilford Press; New York: 1998.
  • Enders CK. Applied Missing Data Analysis. Guilford; New York: 2010.
  • Epstein JN, Tsal Y. Evidence for cognitive training as a treatment strategy for children with attention-deficit/hyperactivity disorder. Journal of ADHD and Related Disorders. 2010;1:49–64.
  • Erickson KI, Colcombe SJ, Wadhwa R, Bherer L, Peterson MS, Scalf PE, Kramer AF. Training-induced functional activation changes in dual-task processing: an FMRI study. Cerebral Cortex. 2007;17:192–204. [PubMed]
  • Ethier M, Braun CMJ, Baribeau JMC. Computer-dispensed cogntive-perceptual training of closed head injury patients after spontenous recovery, Study 1: speeded tasks. Canadian Journal of Rehabilitation. 1989;2:223–233.
  • Fajkowska M, Derryberry D. Psychometric properties of Attentional Control Scale: the preliminary study on a Polish sample. Polish Psychological Bulletin. 2010;41:1–7.
  • Feldon DF, Peugh J, Timmerman BE, Maher MA, Hurst M, Strickland D, Stiegelmeyer C. Graduate students' teaching experiences improve their methodological research skills. Science. 2011;333:1037–1039. [PubMed]
  • Finlayson MAJ, Alfano DP, Sullivan JF. A neuropsychological approach to cognitive remediation: microcomputer applications. Canadian Psychology. 1987;28:180–190.
  • Gernsbacher MA. Cognitive processes and mechanisms in language comprehension: the structure building framework. In: Bower GH, editor. The Psychology of Learning and Motivation. vol. 27. Academic Press; New York: 1991. pp. 217–263.
  • Gioia GA, Isquith PK, Guy SC, Kenworthy L. Behavior Rating Inventory of Executive Function Professional Manual. Psychological Assessment Resources, Inc.; Lutz: 2000.
  • Gray JM, Robertson IH. Remediation of attentional difficulties following brian injury: three experimental single case studies. Brain Injury. 1989;3:163–170. [PubMed]
  • Hart H, Radua J, Mataix-Cols D, Rubia K. Meta-analysis of fMRI studies of timing in attention-deficit hyperactivity disorder (ADHD) Neuroscience and Biobehavioral Reviews. 2012;36:2248–2256. [PubMed]
  • Hoekzema E, Carmona S, Tremols V, Gispert JD, Guitart M, Fauquet J, Vilarroya O. Enhanced neural activity in frontal and cerebellar circuits after cognitive training in children with attention-deficit/hyperactivity disorder. Human Brain Mapping. 2010;31:1942–1950. [PubMed]
  • Hoekzema E, Carmona S, Ramos-Quiroga JA, Barba E, Bielsa A, Tremols V, Vilarroya O. Training-induced neuroanatomical plasticity in ADHD: a tensor-based morphometric study. Human Brain Mapping. 2011;32:1741–1749. [PubMed]
  • Kaufman J, Birmaher B, Brent D, Rao U, Flynn C, Moreci P. Schedule for affective disorders and schizophrenia for school-age children-present and lifetime version (K-SADS-PL): initial reliability and validity data. Journal of the American Academy of Child and Adolescent Psychiatry. 1997;36:980–987. [PubMed]
  • Kerns KA, Eso K, Thompson J. Investigation of a direct intervention for improving attention in young children with ADHD. Developmental Neuropsychology. 1999;16:273–295.
  • Kim YH, Yoo WK, Ko MH, Park CH, Kim ST, Na DL. Plasticity of the attentional network after brain injury and cognitive rehabilitation. Neurorehabilitation and Neural Repair. 2008;23:468–477. [PubMed]
  • Klingberg T, Forssberg H, Westerberg H. Training of working memory in children with ADHD. Journal of Clinical and Experimental Neuropsychology. 2002;24:781–791. [PubMed]
  • Klingberg T, Fernell E, Olesen PJ, Johnson M, Gustafsson P, Dahlstrom K, Westerberg H. Computerized training of working memory in children with ADHD – a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry. 2005;44:177–186. [PubMed]
  • Leon AC, Shear MK, Klerman GL, et al. A comparison of symptom determinants of patient and clinician global ratings in patients with panic disorder and depression. Journal of Clinical Psychopharmacology. 1993;13:327–331. [PubMed]
  • Manly T, Anderson V, Nimmo-Smith I, Turner A, Watson P, Robertson IH. The differential assessment of children's attention: the Test of Everyday Attention for Children (TEA-Ch), normative sample and ADHD performance. Journal of Child Psychology and Psychiatry and Allied Disciplines. 2001;42:1065–1081. [PubMed]
  • Mateer CA, Mapou R. Understanding, evaluating, and managing attention disorders following traumatic brain injury. Journal of Head Trauma Rehabilitation. 1996;11:1–16.
  • Maxwell SE, Delaney HD. Designing Experiments and Analyzing Data: A Model Comparison Perspective. Lawrence Erlbaum Associates; Mahwah: 2004.
  • Muthén LK, Muthén BO. Mplus User's Guide. 6th ed Muthén & Muthén; Los Angeles: 1998–2010.
  • O'Connell RG, Bellgrove MA, Dockree PM, Robertson IH. Cognitive remediation in ADHD: effects of periodic non-contingent alerts on sustained attention to response. Neuropsychological Rehabilitation. 2006;16:653–665. [PubMed]
  • Paelecke-Habermann Y, Pohl J, Leplow B. Attention and executive functions in remitted major depression patients. Journal of Affective Disorders. 2005;89:125–135. [PubMed]
  • Penkman L. Remediation of attention deficits in children: a focus on childhood cancer, traumatic brain injury and attention deficit disorder. Pediatric Rehabilitation. 2004;7:111–123. [PubMed]
  • Posner MI, Raichle ME. Images of Mind. Scientific American Books; NewYork: 1994.
  • Reynolds CR, Kamphaus RW. Behavior Assessment System for Children, Second Edition Manual. AGS Publishing; Circle Pines: 2004.
  • Sattler JM, Dumont R. Assessment of Children: WISC-IV and WPPSI-III Supplement. Jerome R. Sattler, Publisher, Inc.; La Mesa: 2004.
  • Semrud-Clikeman M, Nielsen KH, Clinton A, Sylvester L, Parle N, Connor RT. An intervention approach for children with teacher-and parent-identified attentional difficulties. Journal of Learning Disabilities. 1999;32:581–595. [PubMed]
  • Shalev L, Tsal Y, Mevorach C. Computerized progressive attentional training (CPAT) program: effective direct intervention for children with ADHD. Journal of Cognitive Neuroscience. 2004;(Suppl.) CNS Abstracts. [PubMed]
  • Sohlberg MM, Mateer CA. Effectiveness of an attention-training program. Journal of Clinical and Experimental Neuropsychology. 1987;9:117–130. [PubMed]
  • Strauss E, Sherman EMS, Spreen O. A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary. Oxford University Press; Oxford: 2006.
  • Sumner CR. New tool for objective assessment of ADHD: the Quotient® ADHD system. The ADHD Report. 2010;18:6–9.
  • Swanson JM. School Based Assessments and Interventions for ADHD Students. KC Publishing; Irvine, CA: 1992.
  • Tamm L, Juranek J. Fluid reasoning deficits in children with ADHD: evidence from fMRI. Brain Research. 2012;1465:48–56. [PMC free article] [PubMed]
  • Tamm L, McCandliss BD, Liang A, Wigal TL, Posner MI, Swanson JM. Attention Deficit/Hyperactivity Disorder: Concepts, Controversies, New Directions. vol. 37. Informa Healthcare; New York: 2007. Can attention itself be trained? Attention training for children at risk for ADHD; pp. 399–411. (Medical Psychiatry Series).
  • Tamm L, Hughes C, Ames L, Pickering J, Silver CH, Stavinoha P, Emslie G. Attention training for school-aged children with ADHD: results of an open trial. Journal of Attention Disorders. 2010;14:86–94. [PubMed]
  • Thimm M, Fink GR, Kust J, Karbe H, Sturm W. Impact of alertness training on spatial neglect: a behavioural and fMRI study. Neuropsychologia. 2006;44:1230–1246. [PubMed]
  • Thompson JB, Kerns KA. Cognitive rehabilitation of the child with mild traumatic brain injury. In: Press OU, editor. Neuropsychological Management of Mild Traumatic Brain Injury. Oxford University Press; New York: 1995.
  • Thomson JB, Seidenstrang L, Kerns KA, Sohlberg MM, Mateer CA. Association for Neuropyschological Research and Development. Puyallup; WA: 1984. Pay Attention!
  • Wechsler D. Wechsler Abbreviated Intelligence Scale Fourth Edition: Administration Manual. Harcourt Assessment, Inc.; San Antonio: 2004.
  • Whalen CK, Henker B, Hinshaw SP. Cognitive-behavioral therapies for hyperactive children: premises, problems, and prospects. Journal of Abnormal Child Psychology. 1985;13:391–409. [PubMed]
  • Williams DJ. Doctoral dissertation. University of Washington; Seattle: 1989. A process-specific training program in the treatment of attention deficits in children.
  • Woodcock RW, McGrew KS, Mather N. Woodcock-Johnson III. Riverside; Itasca: 2001.
  • Yochim BP, Baldo JV, Kane KD, Delis DC. D-KEFS Tower Test performance in patients with lateral prefrontal cortex lesions: the importance of error monitoring. Journal of Clinical and Experimental Neuropsychology. 2009;31:658–663. [PubMed]
  • Zelaznik HN, Vaughn AJ, Green JT, Smith AL, Hoza B, Linnea K. Motor timing deficits in children with Attention-Deficit/Hyperactivity disorder. Human Movement Science. 2012;31:255–265. [PMC free article] [PubMed]