|Home | About | Journals | Submit | Contact Us | Français|
A pilot randomized clinical trial was conducted to examine the initial efficacy of Pay Attention!, an intervention training sustained, selective, alternating, and divided attention, in children diagnosed with Attention-Deficit/Hyperactivity Disorder (ADHD). After a diagnostic and baseline evaluation, school-aged children with ADHD were randomized to receive 16 bi-weekly sessions of Pay Attention! (n = 54) or to a waitlist control group (n = 51). Participants completed an outcome evaluation approximately 12 weeks after their baseline evaluation. Results showed significant treatment effects for parent and clinician ratings of ADHD symptoms, child self-report of ability to focus, and parent ratings of executive functioning. Child performance on neuropsychological tests showed significant treatment-related improvement on strategic planning efficiency, but no treatment effects were observed on other neuropsychological outcomes. Treatment effects were also not observed for teacher ratings of ADHD. These data add to a growing body of literature supporting effects of cognitive training on attention and behavior, however, additional research is warranted.
Attention training, also known as cognitive training, is an intervention in which the various components of attention are viewed as skills that can be enhanced by training. Attention in this conceptualization is defined as the appropriate allocation of processing resources to relevant stimuli and is thought to comprise several sub-processes (Coull, 1998). Specifically, attention can be fractionated into (1) attention orientation (direction of attention to a stimulus), (2) selective attention (prioritizing one stimulus in favor of another), (3) sustained attention (attending to one stimulus over time), and (4) divided attention (dividing attention between two plus stimuli) (Coull, 1998). These sub-processes are thought to be the building blocks underlying executive functioning and to have unique neuronal correlates (Coull, 1998). Theoretically, more efficient attentional sub-processes should give rise to enhanced executive processes (Paelecke-Habermann et al., 2005), such as inhibition, task management, planning, monitoring/working memory, and coding. Further, more efficient executive functioning should translate into improvements in behavior – for example, improved inhibition might lead to better self-regulation.
The rationale for attention training is based on the concept that efficiency increases after repetitive practice of specific cognitive operations of attention (Posner and Raichle, 1994), theoretically because practice produces adaptations in the underlying neuroanatomical networks linked to these processes (Kerns et al., 1999). Initial neuroimaging research comparing brain activation before and after different forms of cognitive training supports this contention (Beauregard and Levesque, 2006; Brefczynski-Lewis et al., 2007; Erickson et al., 2007; Kim et al., 2008; Thimm et al., 2006). Furthermore, controlled trials investigating the efficacy of attention training programs report gains in both trained and untrained assays of attention components such as sustained attention and executive function (Ethier et al., 1989; Finlayson et al., 1987; Gray and Robertson, 1989; Mateer and Mapou, 1996; Penkman, 2004; Sohlberg and Mateer, 1987; Thompson and Kerns, 1995).
In the last decade, researchers have turned to investigate attention training as an intervention for Attention-Deficit/Hyperactivity Disorder (ADHD) since ADHD involves impairment in attention and related executive functions. There is a growing literature providing support for attention training in individuals with ADHD (Epstein and Tsal, 2010; Kerns et al., 1999; Klingberg et al., 2002, 2005; O'Connell et al., 2006; Semrud-Clikeman et al., 1999; Shalev et al., 2004; Tamm et al., 2007, 2010; Williams, 1989). These studies report improvements in the cognitive skill being trained directly but also improvements in untrained cognitive skills (Epstein and Tsal, 2010; Tamm et al., 2010; Williams, 1989), although effect sizes are typically small to moderate (Epstein and Tsal, 2010). Furthermore, studies of attention training in ADHD also provide initial support for improvements on untrained measures of attention and academic efficiency (Semrud-Clikeman et al., 1999; Williams, 1989), as well as reductions (although not consistently) in teacher and parent ratings of ADHD symptoms (Kerns et al., 1999; Klingberg et al., 2005; Semrud-Clikeman et al., 1999; Shalev et al., 2004; Tamm et al., 2010; Thomson et al., 1984) and restlessness/head movements (Klingberg et al., 2002). There is also preliminary neuroimaging data to suggest that cognitive training impacts brain function in ADHD (Hoekzema et al., 2010, 2011). After 10 days of cognitive training, children with ADHD showed increases in orbitofrontal, superior frontal, middle temporal, and inferior frontal cortex on an inhibition task and increased cerebellar activity on an attentional task, while the control group showed no such activation differences (Hoekzema et al., 2010). Furthermore, these activation differences were complemented by focal volumetric gray matter increases in bilateral middle frontal cortex and right inferior–posterior cerebellum (Hoekzema et al., 2011). Taken together, these studies suggest that attention training is a promising treatment approach for ADHD. However, to date there have been few randomized clinical trials investigating attention training in ADHD. This is critical given that replication studies have not always been successful and, more critically, studies utilizing blinded evaluators do not always find a positive effect for attention training.
One potentially promising intervention is Pay Attention! (www.lapublishing.com), an intervention targeting sustained, selective, divided, and alternating attention, constructs which have been shown to be compromised in individuals with ADHD (Manly et al., 2001). A small randomized trial of the effects of Pay Attention! in seven children with ADHD reported significant benefits of attention training in measures of sustained attention, executive functioning, and effortful processing compared to children in a video game control group (Kerns et al., 1999). However, the sample size for this study was small and participant diagnosis was not verified by the research team. We conducted an open trial of Pay Attention! with 23 children diagnosed with ADHD and showed effects on executive functioning measures (cognitive flexibility, working memory, parent ratings), as well as improvements in ADHD symptomatology (Tamm et al., 2010). However, we did not utilize a control group as the primary emphasis was to determine feasibility of the intervention in a clinically diagnosed group and in our outpatient setting.
The current trial was initiated to investigate if the Pay Attention! intervention was effective in improving trained and untrained executive functions as well as parent, teacher, and clinician ratings of attention and behavior when compared to a waitlist control group that did not receive the intervention. Consistent with the theoretical underpinnings of attention training, we hypothesized that by training attention (sustained, selective, divided, and alternating), we would see proximal treatment-related improvements in executive functioning and distal treatment-related improvements in attention and behavior (see Fig. 1).
The study was approved by the University of Texas, Southwestern Medical Center at Dallas Institutional Review Board and informed parental consent and participant assent were obtained from all participants prior to initiating any procedures.
Participants (n = 132) were recruited from outpatient clinics at Children's Medical Center at Dallas, the community, and Shelton School, a private school for learning differences; 105 participants were randomized (see Fig. 2, CONSORT diagram). Participants ranged in age from 7 to 15 years (M = 9.3, SD = 1.4) and were predominantly Caucasian. Demographic and baseline characteristics are presented in Table 1. Randomization was stratified by gender, ADHD subtype, and medication status. Participants were asked to maintain a stable medication status (i.e., continue taking medication as directed and not initiate new medication during the study). Exclusion criteria included the following: estimated full scale IQ < 85, history of head injury, history of prenatal drug exposure, diagnosis with other congenital or acquired neurological conditions, and participating in other non-pharmacological treatment interventions for ADHD (e.g., neurofeedback, cognitive-behavioral therapy, etc.).
Families meeting eligibility criteria by phone screen were invited to participate in a baseline evaluation. A semi-structured clinical interview, the Kiddie-SADS-Present and Lifetime Version or K-SADS-PL (Kaufman et al., 1997), was conducted with the primary caregiver, and with the child/adolescent separately, to determine ADHD diagnosis. Information regarding family functioning, family psychiatric history, and developmental and medical histories was also collected. In addition, the clinical interviewers provided ratings of ADHD on the Swanson, Nolan, and Pelham DSM-IV ADHD rating scale (SNAP-IV) based on the interviews, and impairment/functioning using the Clinical Global Impressions (CGI) rating scale. Parents completed several rating scales assessing attention, executive function, and behavior, and participants were administered several tests assessing a variety of executive functions including planning, behavioral inhibition, visual–spatial abilities, and working memory.
Following the baseline evaluation, participants were randomized to either receive the intervention or to a waitlist control group. Participants randomized to the intervention attended twice-weekly 30-min sessions for 8 consecutive weeks (for a total of 16 sessions). Individuals randomized to the waitlist control condition were asked to not begin any new treatment for ADHD during the wait period and were offered the opportunity to receive the intervention at the end of the wait period. All participants and their parents were invited to attend an outcome evaluation. There was variability between subjects as to how long it took them to complete the 16 treatment sessions due to parent and interventionist schedules; thus we opted to re-test all participants on the same schedule (i.e., 12 weeks after baseline). During the outcome evaluation, parents and children were re-interviewed using the K-SADS-PL, clinicians completed SNAP-IV and CGI improvement ratings, and participants completed the same battery of executive functioning measures as at baseline. Parents were also given a packet of ratings to give to teachers to complete at baseline and outcome, although few teachers returned the forms (n = 33 at baseline and n = 27 at outcome for the intervention group; n = 25 at baseline and n = 27 at outcome for the waitlist control group).
The Pay Attention! materials are designed to train sustained, selective, alternating, and divided attention using visual and auditory stimuli. The visual stimuli include a set of cards depicting drawings of children and adults that can be distinguished by various features including age, gender, hair color, and other physical qualities; and a set of home layouts that include several rooms with objects that can be sorted by color, shape, and other characteristics. The auditory stimuli include lists of words played on a CD with the participant required to press a buzzer whenever a specific word is heard. The tasks become progressively more difficult (e.g., a distracting overlay is placed over the visual stimuli, a distracting sound is played during task completion, or participants are asked to complete two tasks simultaneously). All participants completed an initial session to establish performance levels and orient them to the materials. Participants then progressed through the four modules, beginning with the simplest sustained attention tasks. After criterion was reached (e.g., gains in speed while maintaining overall accuracy), the next module was started. Not all participants completed all four modules since they progressed at different rates, although the majority completed at least the sustained and selective attention modules. Participants were given immediate feedback regarding their performance and interventionists spent time each session discussing how the targeted attentional skill could be applied in a home or school setting. Pay Attention! is designed to be flexibly delivered so that the interventionist can tailor the treatment to the specific needs of the participant. Parents were also provided with reading materials about the attention skills being trained and met with the child and interventionist for a few minutes after each session to discuss the activities practiced at each session, which skill was being trained, and how parents could support the child's implementation of the skill in home and school activities.
Demographic and baseline clinical characteristics of the two groups were reported using mean (SD) for continuous variables and frequency (percentage) for categorical variables. To identify any differences between the characteristics of the 2 groups, we used the 2-independent sample t test for continuous variables and the χ2 test for categorical variables.
As a manipulation check of whether the Pay Attention! intervention impacted sustained, selective, divided, and/or alternating attention, we conducted repeated measures ANOVAs in SPSS for each of the 6 subtests administered of the TEA-Ch (note, alternate forms were used for this measure) for the intervention group only; this measure was not available for the waitlist control group.
To address whether the intervention improved behavioral and executive functioning ratings, and child performance on executive function tasks, multivariate and univariate generalized estimating equation (GEE) analyses were performed to test for significant mean differences between the treatment intervention and waitlist control groups at outcome using baseline scores and participant age (for non-normed measures, e.g., SNAP-IV, CGI) as control covariates (e.g., Feldon et al., 2011). The baseline scores were included as covariates to address potential group differences at baseline contributing to the pattern of findings at outcome. Multivariate GEE analyses were used to test for group mean differences among correlated subtests from the same response variable measure (e.g., BRIEF subscales), while univariate GEE analyses were used to test for mean differences among summative subscales or total scores from a response variable measure separately to avoid collinearity (e.g., BRIEF Index scores). These GEE analyses were conducted using Mplus version 6.12 (Muthén and Muthén, 1998–2010) to: (a) handle missing data via maximum likelihood estimation assuming missing at random (MAR) (e.g., Enders, 2010), (b) test for significant response variable mean differences directly using the `Model Constraint' and `New' commands, and (c) ensure that analysis assumptions of homogeneity of covariance matrices and homogeneity of covariate regression slopes were met by imposing the necessary parameter estimate constraints prior to analysis. Testing for group differences on the ATTC attention focusing and attention shifting scales was done via separate analyses of covariance in SPSS (baseline scores and age were entered as covariates) due to the reduced sample size (N = 20) available for those analyses. Given the number of response variable mean difference computations involved, the false discovery rate procedure (Benjamini and Hochberg, 1995) was used to ensure that no more than 5% of the total number of significant findings reported represent Type-I errors. The false discovery rate has been shown to produce fewer Type-I errors than per-comparison critical alpha corrections, and has been shown to better maintain statistical power compared to experiment-wise critical alpha corrections (Benjamini and Hochberg, 1995; Maxwell and Delaney, 2004). The p-values listed in Tables 2–4 are those generated from the false discovery rate procedure and are thus labeled “corrected p-value”. We also report effect sizes which can be particularly illuminating when there is limited power (e.g., teacher ratings, child self-report ratings), and where one wants to compare them with other research in the literature. Cohen's d effect size estimates were computed for all group mean differences.
Finally, additional analyses were conducted to investigate if there were any moderators of treatment response after controlling for baseline score variation. Specifically, for each response variable, separate treatment group by (1) age, (2) gender, (3) IQ, (4) ADHD subtype, and (5) medication status interaction variables were computed and tested for significance. Results of these analyses were also corrected for Type-1 error inflation via the false discovery rate procedure.
Compared to their pre-intervention performance, the group that received intervention improved significantly on the TEA-Ch Score!, Sky Search Completion Time and Attention (combined accuracy and timing score), and Creature Counting Completion Time after the intervention (Table 2).
Parents and clinicians reported significantly fewer ADHD symptoms on the SNAP-IV Inattention and Hyperactivity/Impulsivity ratings for the intervention group compared to the waitlist control group (Table 3). On the BASC-II, parents rated children in the intervention group as having fewer Attention Problems compared to the waitlist control group. Teacher ratings were not significant for the SNAP-IV or BASC-II. Clinician ratings on the CGI indicated lower severity and greater improvement for the intervention group than the waitlist control group. Participants rated themselves as having significantly improved ability to focus their attention [F(3, 19) = 6.6, p < .05, Cohen's d = 1.11] and shift their attention [F(3, 19) = 5.5, p < .05, Cohen's d = .51] on the ATTC.
Examination of effect sizes revealed that teachers were also reporting reductions in Hyperactivity/Impulsivity ratings (medium effect) for the intervention group compared to the control group. On the BASC-II, both parents and teachers reported improvements on the Externalizing, Hyperactivity, and Behavioral Symptoms Index, and teachers reported improvements on the Attention Problems subscale, for the intervention group compared to the control group. These effect sizes ranged from small to large (Table 3).
Significant improvements were reported by parents for all the BRIEF subscales, with the exception of Inhibit (trend for improvement) and Emotion Regulation for the intervention group compared to the waitlist control group (Table 4). In contrast, no significant effect of treatment was observed for any subscale on the teacher BRIEF ratings.
Examination of effect sizes for the teacher ratings revealed improvements for the intervention group compared to the waitlist control group on the teacher rated BRIEF Shift, Emotion Regulation, Initiate, Working Memory subscales, and Behavioral Regulation Index, and a decrement (moderate effect size) for Organization (Table 4), though none of these effects reached statistical significance.
Children in the intervention group performed significantly better than the waitlist control group on the DKEFS Tower Time per Move Ratio (Table 5). No other significant differences between the intervention and waitlist control group were observed on the other neuropsychological measures.
Examination of effect sizes revealed a small effect favoring the intervention group on the DKEFS Inhibition, DKEFS Inhibition/Switching Errors, DKEFS Tower Mean First Move Time, DKEFS Tower Move Accuracy Ratio, WISC-IV Matrix Reasoning, Quotient Latency, Quotient Variability, and WJ-III Understanding Directions measures, but these did not reach statistical significance. The intervention group also made more errors of omission on the Quotient compared to the waitlist control group.
There were no significant interactions between group and IQ, ADHD subtype, medication status, or gender. However, we observed an interaction between age and group on the BRIEF Shift subscale (b = 3.33, Wald Z = 3.26, p < .01), BRIEF Behavioral Regulation Index (b = 1.95, Wald Z = 2.67, p < .01), and the CGI Severity rating (b = .23, Wald Z = 2.56, p < .01) at outcome. For all three interactions, older members of the treatment group had lower scores on those response variables.
The results of this pilot randomized clinical trial of Pay Attention! in children with ADHD suggest that the intervention was successful in impacting attentional sub-processes targeted by training (i.e., we observed improvement on tasks assessing aspects of sustained, selective, divided and alternating attention like visual detection, response speed, etc.). Further, our data show that the intervention improved ADHD symptomatology by parent and clinician report, reduced impairment by clinician report, improved executive functioning including inhibition, shifting, planning, and self-monitoring by parent report, improved child self-report of ability to focus and shift attention, and improved child performance on a task measuring planning efficiency. Although non-significant, examination of effect sizes revealed small to moderate improvements for the intervention group on teacher ADHD and executive function (Shift, Emotion Regulation, Initiate, and Working Memory) ratings, and child performance on executive function measures of inhibition, planning, comprehension and memory of verbal instructions, cognitive flexibility, and response latency. As with other studies, these promising findings are tempered by the fact that we did not have blinded evaluators, parents may have been affected by expectancy bias or Hawthorne Effect, and the vast majority of the most objective measures administered to both groups (child neuropsychological tests) were not statistically significant despite having small to moderate effect sizes. Further, teacher ratings were not significant.
Although not statistically significant in most cases, the direction of group means were, however, generally consistent with those of the previous small randomized clinical trial of Pay Attention! (Kerns et al., 1999) which reported gains on measures of planning, sustained auditory attention, sustained visual attention, behavioral inhibition/executive function/selective attention, and academic efficiency. There was also a trend observed for improved teacher ADHD ratings. In this trial, which utilized different assessment instruments, we also showed significant improvements on planning efficiency, and non-significant gains in behavioral inhibition/executive function/selective attention, and auditory attention/memory. It should be noted that Kerns (Kerns et al., 1999) utilized raw scores in her analyses to address issues related to not being able to detect treatment effects due to a child remaining in the same “normative” group and to detect a child's own gains compared to their own performance. We opted to use scaled scores allowing us better ability to compare the current results to those on the open trial which utilized the same measures.
The findings from our RCT are also essentially consistent with attention, behavior, and executive functioning improvements we observed in our open trial (Tamm et al., 2010). While we did not replicate our previous findings of significantly improved cognitive flexibility on the DKEFS Inhibition/Switching subtest with the Pay Attention! intervention (Tamm et al., 2010), we did observe greater improvement on the WISC Matrix Reasoning and DKEFS Inhibition subscales for the intervention group compared to the waitlist group (small effects). Thus, we still see evidence of enhanced fluid reasoning or cognitive flexibility with Pay Attention! which may not have achieved significance due to the relatively small sample size. The improvement in these domains, if replicated in future studies, is clinically relevant for individuals diagnosed with ADHD who have neuropsychological deficits in these constructs which are associated with impaired brain functioning (Tamm and Juranek, 2012).
Despite significant improvements on parent ratings of working memory, improvements on child working memory subtests were not observed. There was, however, a small, non-significant effect of treatment on the WJ-III Understanding Directions subtest which does involve a working memory component in that it requires listening and mapping a series of sequential directions onto the mental structure under construction and maintaining the sequence in immediate awareness until a new directive changes the sequence (Gernsbacher, 1991). It may be that Pay Attention! improves listening abilities and selective attention more than the construct of working memory.
Analysis of processing variables necessary for DKEFS Tower subtest completion revealed a significant effect on the Time Per Move Ratio. The Time Per Move Ratio represents the mean amount of time that was spent on each move – it is a ratio of the total number of seconds spent solving a problem relative to the total number of moves used for that problem (Yochim et al., 2009). Interestingly, the intervention group also significantly improved on the TEA-Ch subtests involving timing (Sky Search, Creature Counting), which, taken together with small non-significant improvements on Mean First Move Time (speed) and increased latency (i.e., slowing) on the Quotient continuous performance subtest, suggests that one impact of Pay Attention! training is on the child's efficiency and timing. Certainly, the intervention itself included a significant focus on timing, with children encouraged to improve speed and efficiency for each subsequent administration of the various tasks. This finding, if replicated, could have significant implications for reducing impairment in ADHD, since recent research suggests that individuals with ADHD have specific deficits in timing functions which are postulated to be closely associated with impulsiveness, defined as premature, impatient and delay averse with associated lack of temporal foresight (Hart et al., 2012; Zelaznik et al., 2012).
Analyses investigating potential treatment moderators were not significant for gender, IQ, medication status, or ADHD subtype. Age interacted with group on two parent-rated variables (BRIEF Shift and Behavioral Regulation Index) and one clinician-rated variable (CGI) but not with any of the child neuropsychological variables. We controlled for age in the primary analyses for the CGI but not for the BRIEF which is an age-normed measure. It is tempting to conclude that older children may have benefited more from the intervention, but given that this effect was observed on so few variables, it may be that these few variables are more sensitive to the effects of maturation.
One of the issues with psychosocial interventions which has not received much attention in the literature is whether treatment can have emanative (Whalen et al., 1985) or unintended negative effects. It is often assumed that behavioral or cognitive interventions are either benign or have no impact, versus potentially negatively impacting outcomes. Although the findings were not statistically significant, there was a small effect for the intervention group to make more errors of omission on the Quotient continuous performance task and a moderate non-significant effect for the intervention group to be rated as less organized by teachers than the waitlist control group. Thus, it appears that despite potentially improving inhibition and planning efficiency, the Pay Attention! intervention might be having an unintended negative effect in that the child might be slowing too much and therefore failing to respond quickly enough resulting in somewhat increased errors of omission – again, this effect was not statistically significant but warrants attention. Replication is necessary, given the large number of statistical tests and possibility of chance findings.
There are a number of strengths to this study including that it is amongst the largest randomized clinical trials investigating attention training in ADHD. However, the sample size was still relatively limited, at least for detecting effects on the neuropsychological variables. It should be noted that when we did a post hoc power analysis to examine what sample size would be needed to reject the null hypothesis for the neuropsychological measures, necessary sample sizes were much larger than our sample size (e.g., DKEFS Inhibition, N = 314; WISC-IV Matrix Reasoning, N = 178). The use of a waitlist control group and the resulting lack of blinding to condition could have resulted in a Hawthorne effect or inflated ratings by parent, child, and clinician; replication is necessary with an attention control group. Another limitation was the relative lack of teacher data (approximately 50% of the sample had teacher data available for outcome). Finally, we observed significant variability on some outcome variables suggesting the possibility that participants differed in their response to treatment. Further work is necessary to investigate individual differences in response to treatment and to identify potential moderators.
Overall, our study adds to the growing body of literature suggesting cognitive training may improve executive function and behavior in children with ADHD. However, before Pay Attention! can be considered for use in clinical settings additional research is needed. The optimal design would be an RCT which includes a larger sample of, ideally un-medicated, children with ADHD, blinded evaluators, a control group that is designed to reduce expectancy biases, more teacher ratings, and tests that are selected to measure both proximal (areas trained) and distal (generalization to functional impairment) gains. Nonetheless, our results suggest that Pay Attention! did improve sustained, selective, and alternating attention directly, and specifically timing efficiency. Further, it may be effective in reducing ADHD symptoms as well as improving parental ratings of executive functioning. Effect size examination reveals small although non-significant improvements on teacher ratings (small n) and child neuropsychological measures of executive functioning including planning, cognitive flexibility, inhibition, auditory working memory, and response latency/variability. Caution is warranted, however, given the large number of statistical comparisons conducted.
This research project was supported in part by the National Center for Research Resources, National Institutes of Health Grant Number UL1 RR024982. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. Additional funding for the project was provided by a gift from the Sparrow Foundation for the Center for Advanced ADHD Research, Treatment, and Education (CAARTE). We gratefully acknowledge the Pay Attention! interventionists, Aleksandra Foxwell, Jarrette Moore, Lauren Smith, Jeanne Rintelmann, Amanda Gray, Laure Ames, Maryanne Hetrick, Amy Rollo, Cathy Bass, Ana Arenivas, Deidre Edwards, Sarah Swart, Shelley Williamson, Gina Bolanos, and Kyle Clayton, and Conrad Barnes for his data management. We also thank Joyce Pickering, Hum.D. for her support and the use of space at the Shelton School. We appreciate the families who participated in the evaluations and intervention.
Conflict of interest All authors declare that there are no actual or potential conflicts of interest including any financial, personal or other relationships with other people or organizations within 3 years of beginning the submitted work that could inappropriately influence, or be perceived to influence, their work. The funding sources did not have a role in study design, collection, analysis and interpretation of data, writing of the manuscript, nor the decision to submit the article for publication.