|Home | About | Journals | Submit | Contact Us | Français|
The goal was to examine the clinical utility of using the Vanderbilt Attention-Deficit/Hyperactivity Disorder (ADHD) Rating Scale (VARS) to determine when to refer children with ADHD for learningdisorder (LD) evaluations.
A total of 128 stimulant-naive children with ADHD, 7 to 11 years of age, were included in the study. The parents and teachers of 128 children with diagnosed ADHD completed the VARS. The reading, numerical operations, and spelling subtests from the Wechsler Individual Achievement Test, Second Edition, were used to identify children with a comorbid LD. We examined the predictive validity and clinical utility of the VARS performance items for ruling in/out the presence of a comorbid LD.
Thirty-eight percent of the samples met the criteria for a comorbid LD. A cutoff score of 7.5 for the sum of the VARS parent and teacher reading items had excellent clinical utility for ruling out both reading and spelling LDs. Cutoff scores of 4 for the VARS teacher reading and writing items had excellent utility for ruling out comorbid reading and spelling LDs, respectively. None of the VARS performance items effectively identified or ruled out math LDs.
The VARS performance items should be used with an interview about school functioning and a review of school records to rule out the presence of a comorbid reading or spelling LD for children with diagnosed ADHD.
The vast majority of children with attention-deficit/hyperactivity disorder (ADHD) are evaluated by and receive diagnoses from primary care physicians.1,2 In 2000, the American Academy of Pediatrics (AAP) published guidelines providing primary care physicians with evidence-based recommendations for the assessment and diagnosis of children with ADHD.3 These guidelines emphasize that ADHD evaluations should include assessment of commonly occurring comorbid conditions.3 To aid physicians in making ADHD diagnoses on the basis of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), and in assessing comorbid conditions, the AAP published and widely distributed an ADHD toolkit. This toolkit includes parent and teacher versions of standardized measure of ADHD symptoms, the Vanderbilt ADHD Rating Scale (VARS). The VARS contains much of the information required to make a DSM-IV– based diagnosis of ADHD and to screen for common comorbidities.4,5
The AAP guidelines note the importance of screening for comorbid learning disorders (LDs) as part of ADHD evaluations,3 because ~30% of children with ADHD also meet the criteria for a LD.6 Furthermore, children with comorbid ADHD and LD experience more-severe impairments than children with ADHD alone7 and may need adjunctive interventions (eg, direct instruction).8 However, minimal guidance is provided regarding how physicians might use parent and teacher VARS data to screen for LDs and when they should make referrals. It is important that physicians do not over-refer for LD evaluations, because not all children with ADHD require psychoeducational evaluation and the capacity of schools and psychologists to complete psychoeducational testing is limited.
The VARS includes performance items that ask about children’s reading, math, and writing abilities. To date, there has been no research on the predictive validity or clinical utility of these items in identifying children with potential LDs. Bennett et al9 examined the utility of a teacher rating scale, the Academic Performance Rating Scale, in identifying children who should be referred by physicians for a psychoeducational evaluation. Bennet et al9 reported that 2 items had adequate clinical utility and could be used by physicians to screen for LD, that is, teacher’s rating of children’s reading ability and math ability. Unfortunately, the Academic Performance Rating Scale is not a commonly used clinical instrument among physicians. Given that the VARS has items that ask about reading and math ability and already is widely used by physicians, it is important to determine whether these items can be used to screen for potential LDs.
The purpose of this study was to determine whether the VARS performance items can be used reliably to screen for LDs and, if so, to identify cutoff scores that physicians can use as a basis for making referrals. Given that it is not the role of physicians to diagnose LDs, we were interested primarily in establishing guidelines indicating when physicians did not need to refer patients for psychoeducational testing. To accomplish this aim, we examined the predictive validity and clinical utility of the VARS performance items for identifying children without LDs. We also examined group differences for the items, comparing children with ADHD alone and children with ADHD and LD. On the basis of these data, we provide recommendations for when physicians should refer patients for LD evaluations.
A total of 128 stimulant-naive children with ADHD, 7 to 11 years of age (mean ± SD age: 8.13 ± 1.20 years), were recruited to participate in a study focused on the neuropsychological outcomes of children with ADHD. Participants were recruited through schools and local practitioners. Diagnoses were determined by using the Diagnostic Interview Schedule for Children (DISC), Parent Report,10 supplemented with up to 2 symptoms identified by children’s teachers with the VARS for cases falling just below the DISC diagnostic threshold. Children with estimated full-scale IQ values of <80, as assessed with the Wechsler Abbreviated Scale of Intelligence,11 were excluded from participation. Sixty-four children met the criteria for ADHD, predominantly inattentive type, and 64 met the criteria for ADHD, combined type. The sample was 72% male, 72% white, 17% black, and 11% other. According to parent responses on the DISC, 37% of the sample met the criteria for oppositional-defiant disorder, 4% for conduct disorder, 37% for an anxiety disorder, and <1% for a mood disorder.
The VARS includes DSM-IV– based scales with teacher report and parent report forms.4,5 The VARS includes the 18 ADHD symptoms described in the DSM-IV, which are rated on a 4-point Likert scale (from 0 to 3) that indicates whether each ADHD symptom occurs never (score of 0), occasionally (score of 1), often (score of 2), or very often (score of 3). In addition to assessing ADHD symptoms, the VARS includes a set of performance items that assess functional impairment. Parents and teachers rate 8 partially overlapping, functional impairment items (Table 1) on a 5-point Likert scale (from 1 to 5) that indicates whether performance is excellent (score of 1), above average (score of 2), average (score of 3), somewhat problematic (score of 4), or problematic (score of 5) in each domain.
The Wechsler Individual Achievement Test, Second Edition (WIAT-II),12 provides standardized, normative value-referenced, academic achievement scores across a variety of subjects for individuals between the ages of 4 and 85. The reading, numerical operations, and spelling subtests were administered in the present study, to provide brief estimates of academic achievement.
Best-practice recommendations for diagnosing LDs include documentation of an academic skills deficit, defined as a score >1 SD below the mean on a normative value-referenced academic achievement test.13 Accordingly, we choose a definition of LD as a score of ≤85 on any subtest of the WIAT-II. In our sample, when LD was diagnosed on the basis of scores of ≤85 on ≥1 of these subtests, approximately one-third of the children were identified, which is consistent with prevalence rates of ADHD/LD comorbidity.6 With the use of this definition, 38% of the ADHD sample met the study criteria for ≥1 of the 3 types of LD. Eleven percent met the criteria for a reading LD, 30% for a math LD, and 19% for a spelling LD. The majority (59%) of children with comorbid LDs met the criteria in only 1 domain, with 27% meeting criteria in 2 domains and 14% meeting criteria in all 3 domains (reading, writing, and math). No demographic differences were found between children who met the criteria for LDs and children who did not.
The first set of analyses examined group differences on all of the VARS performance items between children with ADHD alone and children with comorbid ADHD and LD. Group differences were examined separately, by using analyses of variance, for parent and teacher ratings. Next, we assessed correlations between each VARS impairment item and each WIAT-II subtest score. Items that were significantly correlated with WIAT-II scores were used in the predictive validity and clinical utility analyses.
To examine the predictive validity of each of the VARS performance items, we performed a series of univariate logistic regression analyses. Specifically, we examined the ability of each item to predict the presence of comorbid reading, math, and spelling LDs. A dichotomous indicator of LD status was used as the criterion variable. We then plotted receiver operating characteristic (ROC) curves for each item that was a significant predictor of a comorbid LD. ROC analyses generate an area under the curve (AUC), which is the probability that the result for a randomly chosen positive case would be higher than the result for a randomly chosen negative case. ROC analyses also produce optimal cutoff scores for discriminating between children with and without a comorbid LD.
The last step was to examine the clinical utility of each of the ROC curve-generated optimal cutoff scores. Previous research supports the examination of clinical utility by calculating positive predictive power (PPP) and negative predictive power (NPP), as well as sensitivity and specificity.14,15 In this study, we applied κ corrections, as described by Frick et al,15 to the PPP and NPP values to correct for base rates of diagnoses in the sample, which can generate inflated values16; this process yielded corrected PPP (cPPP) and corrected NPP (cNPP) values.
We also completed a set of analyses to evaluate the predictive validity and clinical utility of the VARS performance items in combination. Specifically, the performance items were examined to determine whether a strategy of combining items and raters within a single model would have better predictive validity and clinical utility than use of each impairment item individually. To evaluate this possibility, we performed amultivariate logistic regression analysis predicting each type of LD with the domain-specific parent and teacher items entered in the model simultaneously. Subsequent analyses were conducted by using the same procedures as outlined above.
Group differences between children with ADHD alone and children with ADHD and LD are presented in Table 1. Results revealed that children with ADHD and LD, compared with children with ADHD alone, were rated as significantly more impaired on the reading, math, and writing items by parents and teachers and on the overall school functioning item by parents.
Correlational analyses revealed that only the VARS items related to academic ability were significantly correlated with the corresponding WIAT-II score (eg, rating of reading impairment related to WIAT-II reading score). All correlations were in the expected, negative direction (ie, higher VARS ratings were associated with lower WIAT-II scores). Correlations of parent-and teacher-rated reading impairment on the VARS with the WIAT-II reading subtest scores were moderate to high (parent-rated, −0.61; teacher-rated, −0.74; P < .001 for both). Correlations of parent- and teacher-rated math impairment with the WIAT-II math subtest scores were moderate (parent-rated, −0.60; teacher-rated, −0.59; P < .001 for both). Correlations of parent- and teacher-rated writing impairment with the WIAT-II spelling subtest scores also were moderate (parent-rated, −0.51; teacher-rated, −0.59; P < .001 for both).
Univariate logistic regression analyses revealed that teacher ratings of reading, writing, and math impairment were uniquely predictive of domain-specific LDs (reading, R2 = 0.17; math, R2 = 0.11; writing/spelling, R2 = 0.12; P < .01). Parent ratings of impairment in reading and math also were predictive of LDs in their respective domains (reading, R2 = 0.16; math, R2 = 0.11; P < .01), but parent-rated writing impairment did not predict a spelling LD significantly (R2 = 0.03; P > .05).
ROC analyses were conducted for each performance item in relation to the presence of its respective LD (eg, math item and math LD). Analyses using teacher ratings revealed AUCs for reading of 0.75 (reading LD), for math of 0.67 (math LD), and for writing of 0.67 (spelling LD). Use of parent ratings of impairment in relation to the presence of a specific LD resulted in AUCs for reading of 0.74 (reading LD), for math of 0.67 (math LD), and for writing of 0.58 (spelling LD).
By using the ROC analyses, optimal cutoff values for ruling in/out the presence of a LD were examined. Consistent with previous research,17,18 optimal cutoff values were considered to have acceptable clinical utility for ruling in the presence of a LD if the cPPP was ≥0.65 and for ruling out the presence of an LD if the cNPP was ≥0.65. None of the items had acceptable cPPP values for ruling in a LD. A number of the items were clinically useful for ruling out a LD (ie, cNPP) (Table 2). A cutoff score of ≥4 for the VARS reading item, rated by either the parent (cNPP = 0.69) or a teacher (cNPP = 0.80), met the criterion for ruling out a reading LD. A cutoff score of ≥4 for the VARS writing item rated by a teacher had excellent cNPP (cNPP = 1.00) for ruling out a spelling LD. Parent-rated writing did not meet the criterion for ruling out a spelling LD. None of the parent- or teacher-rated items met the criterion for ruling out a math LD.
Multivariate models predicting reading, math, and spelling LDs were calculated with the domain-specific parent-and teacher-rated items entered simultaneously. The parent/teacher combination models produced significant R2 values (reading, R2 = 0.18; math, R2 = 0.15; spelling, R2 = 0.20; P < .01). ROC analyses with the domain-specific performance items in combination (eg, parent reading plus teacher reading items) resulted in AUCs of 0.68 for reading, 0.68 for math, and 0.64 for spelling. The optimal ROC analysis– generated cutoff point was a combined summed score of >7.5 in all cases. Clinical utility analyses using the ROC analysis– generated cutoff scores revealed that parent plus teacher reading items met the criterion for ruling out a reading LD (cNPP = 1.0) and also met the criterion for ruling out a spelling LD (cNPP = 0.89) (Table 2). No combination of items met the criterion for a math LD.
This is the first study to examine the predictive validity and clinical utility of the VARS performance items for predicting LDs. Our results show that the VARS performance items can be used reliably by physicians to determine which children do not need to be referred for evaluation of a comorbid reading or spelling LD. This is important clinical information, given the high rates of ADHD/LD comorbidity6 and the fact that community physicians are relied on to determine whether children with ADHD should be referred for psychoeducational testing. Furthermore, the capacity of schools and psychologists to complete psychoeducational testing is limited, and use of the VARS performance items should increase physicians’ ability to make appropriate referrals.
By using ROC analyses, optimal cutoff values for ruling in/out the presence of a LD were examined. None of the items had acceptable cPPPs for ruling in or confirming a LD. This is not overly problematic, because it is not the role of physicians to diagnose LDs. Furthermore, the VARS items have good clinical utility in identifying children who likely do not meet the criteria for a LD and therefore do not need to be referred for evaluation. In this study, the combination of parent and teacher ratings or teacher ratings alone provided the best clinical utility for ruling out comorbid LDs and the need for referral. As shown in Table 2, a cutoff point of 7.5 for the sum of the VARS parent and teacher reading items had excellent cNPP for ruling out both reading (cNPP = 1.0) and spelling (cNPP = 0.89) LDs. This means that, in this sample, children with parent plus teacher reading item scores of <8 were highly unlikely to meet the criteria for a comorbid reading or spelling LD.
Teacher ratings alone also could be used to rule out the presence of reading and spelling LDs. Specifically, a cutoff score of 4 for the VARS reading item rated by a teacher had excellent cNPP (cNPP = 0.80) for ruling out a reading LD, and a cutoff score of 4 for the VARS writing item had excellent cNPP (cNPP = 1.0) for ruling out a spelling LD. The cNPP of 1.0 for the writing item means that no child with a teacher-rated score of <4 for the VARS writing item met our criterion for a spelling LD. This cutoff point makes sense, given that a rating of 3 on the VARS indicates that the child’s ability in the area is in the average range and scores of 2 and 1 indicate above-average ability.
Only 1 parent-rated item had adequate clinical utility. A cutoff score of 4 for the VARS reading item rated by the parent met the NPP criterion for ruling out a reading LD (cNPP = 0.69). The greater utility of teacher ratings is not surprising, given that teachers observe children’s academic skills directly on a daily basis and in comparison with those of other children. Furthermore, teachers often are aware of children’s reading, writing, and math proficiency on the basis of their scores on school-administered standardized tests. This finding highlights the importance of physicians collecting VARS ratings from the school, as recommended by the AAP guidelines.3
None of the parent- or teacher-rated items met the criterion for ruling in or out a math LD. Physicians need to use the VARS items in combination with assessment of math functioning at school to determine whether a referral is needed. It is important to note that none of the VARS items should be used alone to determine whether to refer and therefore interviewing further about math abilities is not an added burden. Specifically, referral decisions always should be made on the basis of the combination of data from the VARS items, interviews about school functioning, and examination of school records, such as achievement test scores.
In cases in which it is unclear whether a referral is needed, physicians should initiate treatment for ADHD and monitor the VARS academic impairment items for improvement. Children with academic difficulties primarily attributable to ADHD (eg, difficulties focusing in class) should benefit from ADHD treatment and VARS academic impairment ratings should improve. However, academic problems attributable to a comorbid LD are basic skill deficits (eg, reading, math, and writing skills) and are unlikely to normalize with ADHD treatment.19,20 If symptoms of ADHD improve with treatment but the VARS academic performance ratings do not improve, then referral for psychoeducational evaluation may be warranted.
We classified cases as LDs on the basis of the results of abbreviated achievement assessments, and children in the sample did not receive full psychoeducational evaluations. Therefore, some cases might have been classified incorrectly as LDs, which might have affected the results of the clinical utility analyses. Additional research is needed to replicate these results by using full psychoeducational evaluations to diagnosis LDs. In addition, all of the participants in this study were stimulant naive, and our findings may not be generalizable to groups of children already being treated for ADHD. Future research is needed to determine whether the VARS can be used to rule out LDs among children with diagnosed ADHD who are receiving treatment. Finally, it is important to note that children with above-average cognitive abilities may not exhibit academic problems according to VARS ratings during early childhood. Physicians need to screen consistently for academic problems, to determine whether LD evaluation referral becomes necessary as children progress through school.
Approximately 30% of children with ADHD also meet the criteria for a comorbid learning disorder. Determining when to refer children for LD evaluations can be difficult, and data-driven approaches are needed to facilitate these decisions.
The authors evaluate the utility of the Vanderbilt ADHD Rating Scale performance items for identifying children with ADHD who are unlikely to meet the criteria for a LD. The authors use these data to make recommendations about when children should be referred.
Funding for this study was provided by the National Institutes of Health (grant R01MH074770). This research was also supported by a Mid-Career Investigator Award in Patient Oriented Research (PI: Epstein; K24 MH064478) and two Mentored Patient-Oriented Research Career Development Awards from the National Institute of Mental Health (PIs: Brinkman and Froehlich; K23 MH083881 and K23 MH083027).
Funded by the National Institutes of Health (NIH).
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
Reprints Information about ordering reprints can be found online: http://www.pediatrics.org/misc/reprints.shtml