|Home | About | Journals | Submit | Contact Us | Français|
To evaluate the clinical utility of the cutoff recommendations for the Vanderbilt ADHD Diagnostic Parent Rating Scale (VADPRS) comorbidity screening scales provided by the American Academy of Pediatrics/National Initiative for Children’s Healthcare Quality (AAP/NICHQ) and to examine alternative cutoff strategies for identifying and ruling out disorders commonly comorbid with ADHD.
A sample of 215 children (142 with ADHD), ages 7-11, participated in the study. Parents completed the VADPRS and were administered a diagnostic interview to establish diagnoses of oppositional defiant disorder (ODD), conduct disorder (CD), anxiety, and depression. The clinical utility of the VADPRS comorbidity screening scales were examined.
The recommended AAP/NICHQ cutoff strategies did not have adequate clinical utility for identifying or ruling out comorbidities, with the exception of the VADPRS ODD cutoff strategy which reached adequate levels for ruling out a diagnosis of ODD. An alternative cutoff approach using total sum scores was superior to the recommended cutoff strategies across all diagnoses in terms of ruling out a diagnosis, and this was particularly evident for anxiety/depression. Several individual items on the ODD and CD scales also had acceptable clinical utility for ruling in diagnoses.
The VADPRS comorbidity screening scales may be helpful in determining which children likely do not meet diagnostic criteria for ODD, CD, anxiety, or depression. This study suggests that using a total sum score provides the greatest clinical utility for each of these comorbidities and demonstrates the need for further research examining the use of dimensional assessment strategies in diagnostic decision-making.
Attention-deficit/hyperactivity disorder (ADHD) is the most common neurodevelopmental disorder in childhood,1 and primary care physicians (PCPs) are frequently involved in the evaluation and diagnosis of youth with ADHD.2,3 Coexisting mental health problems in children with ADHD is the norm rather than the exception,4 with approximately three-quarters of children with ADHD meeting criteria for another psychiatric disorder.5 Indeed, the American Academy of Pediatrics (AAP) guidelines which provide primary care physicians with evidence-based recommendations for the assessment and diagnosis of children with ADHD emphasize the importance of simultaneously evaluating common comorbid mental health problems.6 Still, over half of pediatricians remain uncertain as to whether they have the requisite training for assessing ADHD comorbidities.7
In 2002, the AAP and the National Initiative for Children’s Healthcare Quality (NICHQ) jointly published a toolkit to be used in the assessment and treatment of ADHD in primary care settings (available at www.nichq.org). This toolkit includes a standardized measure of ADHD symptoms, the Vanderbilt ADHD Diagnostic Parent Rating Scale (VADPRS).8 In addition to items corresponding to the ADHD diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), the VADPRS includes symptom screens for three common comorbidities: oppositional defiant disorder (ODD), conduct disorder (CD), and anxiety/depression (ANX/DEP).8 Screenings for each domain contain a select group of behaviors for each comorbidity (the ODD and CD screens include DSM-IV-based symptoms, whereas the ANX/DEP screen consists of items modified from another pediatric scale). Further, VADPRS scoring algorithms provide recommended scoring thresholds which may indicate the possible presence of these comorbid disorders. While the VADPRS has strong psychometric properties for the assessment of ADHD8,9 and is frequently used by physicians,10 the comorbidity screening scales have not received adequate attention regarding their clinical utility,11,12 that is, the extent to which the dimensional scales of the VADPRS are useful for detecting the presence or absence of corresponding DSM-IV- based diagnoses. Promising evidence to date shows the VADPRS comorbidity screening scales to be reliable8 and broadly correlated with corresponding diagnoses of a structured diagnostic interview.13 However, the cutoff recommendations for the VADPRS comorbidity screening scales provided in the AAP/NICHQ ADHD toolkit have not been thoroughly evaluated for their clinical utility.
Therefore, the purposes of the present study were to (1) evaluate the clinical utility of the cutoff recommendations for the VADPRS comorbidity screening scales provided in the AAP/NICHQ ADHD toolkit, and (2) establish whether alternative cutoff recommendations may improve clinical utility.
The sample consisted of 215 stimulant naïve children aged 7-11. A community sample of children was recruited using newspaper advertisements and by circulating flyers to local schools and practitioners. Recruitment materials specified that the study was seeking children with and without attention problems. To be included in the study, children were required to be between the ages of 7 and 11 and to have never taken psychiatric medication for ADHD or other mental health problems. Children were excluded from participation if their medical history suggested brain injury such as head trauma with loss of consciousness, seizure disorder, or history of infarction. Sample demographics are displayed in Table 1.
All study measures were completed on the same day during a laboratory visit. In addition to the primary study measures below, estimated full scale IQ was assessed with the Wechsler Abbreviated Scale of Intelligence and academic achievement was assessed with the Wechsler Individual Achievement Test.
The VADPRS is a parent-report scale with good internal consistency, factor structure, and concurrent validity for the assessment of ADHD.8,9 The VADPRS includes the 18 DSM-IV ADHD symptoms rated on a 4-point scale that indicates how frequently each ADHD symptom occurs (0 = never, 1 = occasionally, 2 = often, 3 = very often). In addition, the VADPRS includes ODD (8 items), CD (14 items), and ANX/DEP (7 items) comorbidity screening scales. The ODD and CD items correspond to their respective DSM-IV symptoms, whereas the ANX/DEP scale is not DSM-IV-based. The comorbidity screening scales have adequate reliability, factor structure, and preliminary evidence of concurrent validity.8 In the present study, αs = .94 for ODD, .79 for CD, and .93 for ANX/DEP. Last, the VADPRS includes a set of performance items that assess functional impairment rated on a 5-point scale (1 = excellent performance, 5 = problematic performance) across academic and social domains.
According to NICHQ/AAP scoring algorithms, an item score of 2 or 3 indicates symptom presence and a score of 4 or 5 on one of the performance items indicates impairment presence. According to the NICHQ/AAP cutoff recommendations, a child is considered to screen positive if 4 of 8 ODD symptoms are present, 3 of 14 CD symptoms are present, or 3 of 7 ANX/DEP symptoms are present, and at least one performance item is endorsed at a 4 or 5.
DSM-IV criteria for ADHD, including age of onset, pervasiveness, and impairment, were determined using the Diagnostic Interview Schedule for Children, Version IV (DISC-IV).14 The DISC-IV is a structured diagnostic interview designed for use in epidemiological and clinical studies. It contains algorithms to generate DSM-IV-based diagnoses and has demonstrated substantial reliability and validity.14
After establishing DSM-IV diagnoses with the DISC-IV, we first examined functional impairment and academic achievement scores as a function of ADHD diagnosis and comorbidity in order to examine whether participants with comorbid ADHD were more impaired than noncomorbid ADHD or nondiagnosed participants. Next, the clinical utility of various thresholds were examined for each comorbidity screening scale (i.e., ODD, CD, ANX/DEP). We used a strategy consistent with previous research, including the DSM-IV field trials for the disruptive behavior disorders.15 First, the recommended AAP/NICHQ thresholds were applied. Given the importance of establishing the presence of functional impairment in screening and diagnostic decisions and consistent with the AAP/NICHQ scoring algorithm, all cutoff strategies required at least one performance item to be rated as problematic (ie, score of 4 or 5).
To examine the clinical utility of the AAP/NICHQ thresholds for each of the VADPRS comorbidity scales, we first computed the positive predictive power (PPP), which is the proportion of individuals who meet or exceed a specified symptom threshold and who have the associated diagnosis, and negative predictive power (NPP), which is the proportion of individuals who do not meet the specified symptom threshold and who do not have the associated diagnosis.16 As noted by Frick et al.,15 the use of PPP and NPP are particularly relevant to studies examining the diagnostic process “because diagnostic decision-making bases diagnoses on the presence or absence of symptoms” (p. 530). Also consistent with Frick et al.,15 we applied k corrections to the PPP and NPP values in this study in order to correct for base rates of diagnoses in the sample which can generate inflated values;17 this process yields corrected PPP (cPPP) and corrected NPP (cNPP) values. It is also important to note that the results generated when using cPPP and cNPP values to determine clinical utility are similar to the results when other approaches, such as receiving operating curve (ROC) analyses, are used.15 Finally, we computed sensitivity and specificity for each cutoff threshold.15,18 Sensitivity is the percentage of true positive cases identified whereas specificity is the percentage of true negative cases identified.
Next, we evaluated an alternative set of VADPRS comorbidity scale thresholds. Specifically, we examined the full range of symptom counts and total sum scores (ie, the sum score of all items within the comorbidity scale) for each scale. For example, for ODD we evaluated the psychometric properties of different symptom count thresholds (ie, range from 0 to all 8 symptoms endorsed) and the full range of total sum scores (range = 0-24). We also examined the clinical utility of individual scale items (ie, single item endorsed “2” or “3”) at predicting DISC-IV diagnoses. Also, although the AAP/NICHQ recommendations do not provide cutoff guidelines for anxiety and depression separately, but rather a combined ANX/DEP scale, we examined alternative threshold approaches for anxiety and depression both separately and in combination. For combined analyses, the VADPRS ANX/DEP scale was used in relation to either a DISC-IV diagnosis of depression or anxiety; when examined separately, the four VADPRS depression items (α = .92) were used in relation to a DISC depression diagnosis and the three VADPRS anxiety items (α = .83) were used in relation to a DISC anxiety diagnosis. For each alternative threshold, cPPP, cNPP, sensitivity, and specificity were computed.
In comparing the AAP/NICHQ thresholds to alternative thresholds, we first evaluated cPPP and cNPP. Consistent with previous research examining other ADHD symptom measures,19,20 optimal cutoff values were considered to have acceptable clinical utility for ruling in the presence of a DISC diagnosis if the cPPP was ≥0.65 and for ruling out the presence of a DISC diagnosis if the cNPP was ≥0.65. If either of these cutoffs were met, we then examined sensitivity and specificity. Although an accurate measure must be both highly sensitive and highly specific, cutoff recommendations for clinical use necessitates a trade-off between sensitivity and specificity15 with a general guideline for both sensitivity and specificity to be above 0.50.19 Given the screening purpose of the VADPRS comorbidity scales and the implications and high costs associated with not identifying a child with mental health problems,21,22 we prioritized sensitivity over specificity in our decision-making when multiple strategies produced similar results.
Using the DISC-IV, 142 of the 215 children met DSM-IV criteria for ADHD, 82 with ADHD-combined type, 55 with ADHD-predominantly inattentive type, and 5 with ADHD-predominantly hyperactive-impulsive type. The DISC-IV was also used to establish other diagnoses, which are displayed in Table 1. As expected, participants with ADHD had higher rates of all diagnoses compared to participants without ADHD.
Next, participants were placed into three groups in order to examine impairment and academic achievement as a function of comorbidity. Only two participants without ADHD had a psychiatric diagnosis and were removed from these preliminary analyses. Analyses of covariance (ANCOVAs) and subsequent post-hoc comparisons were used to determine if participants with No Diagnosis (n = 71), ADHD Only (n = 72), or Comorbid ADHD (n = 70) differed on the VADPRS performance items and WIAT academic achievement scales while controlling for participant IQ. As shown in Table 2, significant group differences emerged indicating that, as expected, the No Diagnosis group had lower parent-reported functional impairment than either ADHD Only or Comorbid ADHD groups across items. In addition, children in the Comorbid ADHD group had higher social impairment (ie, relations with parents, siblings, and peers) than children in the ADHD Only group.
Table 3 shows the sensitivity, specificity, cPPP, and cNPP for the AAP/NICHQ cutoff recommendations for each of the VADPRS comorbidity screening scales using the full sample of 215 participants. The VADPRS ANX/DEP comorbidity screen does not differentiate between anxiety and depression, and so children with either a DISC-IV anxiety or depression diagnosis were collapsed into one DISC-IV ANX/DEP group for pertinent analyses. At their recommended cutoffs, each of the comorbidity screening scales were highly specific and moderately sensitive. However, only the ODD cutoff strategy had acceptable clinical utility for ruling out the presence of a DISC ODD diagnosis (cNPP = 0.67), as CD and ANX/DEP cutoff strategies had cNPP values below our cutoff for establishing clinical utility (CD = 0.64, ANX/DEP = 0.36). Further, across all of the comorbidity scales, the AAP/NICHQ thresholds had unacceptable cPPP values for ruling in a diagnosis (ODD = 0.59, CD = 0.37, ANX/DEP = 0.25).
Next, cutoff values for ruling in/out the presence of a DISC-IV diagnosis based on the alternative cutoff strategies were examined. Table 3 displays the results of the alternative cutoff approach analyses that provided acceptable cNPP or cPPP (ie, ≥0.65). For the total sum score and symptom threshold approaches, multiple total sum score or symptom threshold cutoffs had acceptable cNPP or cPPP, and so only the threshold with the highest balance of both sensitivity and specificity is displayed in Table 3; for the individual scale item approach, psychometric properties of all items with acceptable cNPP or cPPP are displayed in Table 3.
For ODD, CD, and depression specifically, multiple cutoff approaches provided a high degree of clinical utility. For ODD, the optimal approach was a scale sum score (ie, sum of the eight ODD items) of 10 or higher. This sum score strategy, as well as a severity cutoff of ≥3 symptoms had higher sensitivity and cNPP than the recommended AAP/NICHQ cutoff strategy requiring ≥4 symptoms while having very little corresponding loss in specificity and cPPP (.03 and .04-.03 reduction, respectively). For CD, the optimal cutoff was a scale sum score of 4 or higher, and this strategy or a severity cutoff of 2 or more symptoms had higher sensitivity and cNPP in comparison to the recommended AAP/NICHQ cutoff strategy requiring 3 symptoms. For ANX/DEP, the optimal cutoff was a scale sum score of 4 or higher; when anxiety and depression were separated, sum scores remained optimal and increased the clinical utility for depression slightly (due to higher specificity and cNPP) but not for anxiety (due to lowered cNPP and sensitivity).
Although several individual ODD, CD, and depression items had acceptable cNPP, none had greater overall clinical utility than the overall sum scores. However, two ODD items (“Actively defies to or refuses to go along with adults’ requests or rules” and “Is angry or resentful”) and one CD item (“Has stolen things of value”) had acceptable cPPP for ruling in a diagnosis of ODD and CD, respectively (see Table 3). For practical purposes, Table 3 also displays the percentage of DISC-IV cases positively identified with each VADPRS cutoff approach, although it is important to note that the purpose of this study was to identify the cutoff approach with the highest overall clinical utility, and as such, our decision-making was based on the full range of cNPP/cPPP and sensitivity/specificity statistics as described above.
The purpose of this study was to examine the clinical utility of the VADPRS comorbidity screening scales, particularly in reference to the AAP/NICHQ cutoff recommendations. First, our results demonstrated that children with comorbid ADHD are generally more impaired than noncomorbid children with ADHD, and this is particularly evident across domains of social functioning. In addition, children with ADHD had higher rates of other psychiatric diagnoses than children without ADHD. These results attest to the importance of assessing additional diagnoses when children are referred for ADHD evaluations. In turn, the primary results of this study show that although none of the optimal VADPRS comorbidity scale thresholds had adequate levels for determining which children likely meet diagnostic criteria for ODD, CD, anxiety, or depression (ie, “ruling in” a diagnosis), alternative thresholds on the existing VADPRS scales can be used to determine which children likely do not meet diagnostic criteria for these disorders, and in turn, which children do not need to be referred for further mental health evaluation. Utilization of a total sum score provided the greatest clinical utility for each of the comorbidities considered. Further, a total sum score strategy provided greater clinical utility than the AAP/NICHQ cutoff recommendations which utilize symptom counts, particularly for anxiety and depression. The optimal cutoff strategies for ruling in and out diagnoses are summarized in Table 4.
Among the mental health comorbidities examined in this study, results were most conclusive in relation to ODD. For ODD, a scale sum of 10 or lower had excellent negative predictive power (cNPP = 0.81) for ruling out an ODD diagnosis, and was superior to the AAP/NICHQ cutoff recommendation (cNPP = 0.67). In addition, the items “Actively defies to go along with adults’ requests or rules” and “Is angry or resentful” had adequate positive predictive power for ruling in an ODD diagnosis (cPPP = 0.65 and 0.67, respectively). These cutoffs also reached high levels of sensitivity and specificity (0.55-0.88 and 0.85-0.94, respectively). However, a single scale item should not drive referral or follow-up decision-making, and so these items are likely to be best used in conjunction with the ODD sum score results. Accordingly, physicians should consider children with parent sum scores below 10 as highly unlikely to meet criteria for ODD, whereas children with parent scores of 2 of 3 on the items of active defiance or anger/resentment are potentially likely to qualify for an ODD diagnosis and in need of further evaluation.
Similar to ODD, a sum score approach offered the most clinical utility in ruling out a CD, anxiety, or depression diagnosis. Specifically, children with parent sum scores of <4 were highly unlikely to meet criteria for CD (cNPP = 0.86), and this approach demonstrated greater utility than the AAP/NICHQ cutoff recommendation which did not demonstrate adequate cPPP or cNPP. It is also important to note that although the AAP/NICHQ cutoff recommendation was close to our criterion for adequately ruling out a diagnosis of CD (cNPP = .64), and may thus have some clinical utility, this approach resulted in much lower cNPP and sensitivity statistics compared to the alternative sum score approach. Also, although children with parent-reported symptom endorsement on the item “Has stolen things that have value” highly likely to meet criteria for CD (cPPP = 0.65), the very low base rate for this item in our sample, in addition to the low associated sensitivity (0.22), precludes placing too much confidence in this individual item for ruling in a CD diagnosis.
For ANX/DEP, as well as anxiety and depression separately, the sum scale has the greatest clinical utility and was notably superior compared to the AAP/NICHQ cutoff recommendation which did not have adequate positive or negative predictive power (ie, cNPPs and cPPPs < 0.65). A sum ANX/DEP score below 4 was most useful to rule out either a diagnosis of anxiety or depression (cNPP = 0.75). When the anxiety and depression scales were examined separately, the optimal threshold for ruling out anxiety and depression was a score below 5 (anxiety cNPP = 0.65, depression cNPP = 0.80). The anxiety and depression scores on the Vanderbilt are particularly important for pediatricians to attend to, as up to 35% of children with ADHD experience clinical levels of anxiety or depression symptoms5 which may in turn affect ADHD treatment decision-making.23 Further, depression and anxiety often co-occur in children,24 and in our sample only one child met DISC-IV criteria for depression without simultaneously meeting criteria for an anxiety disorder. Still, diagnostic specificity between anxiety and depression (and their co-occurrence) has important implications for intervention.25 Therefore, physicians may find using either the ANX/DEP composite scale or the distinct anxiety and depression scales to be most helpful depending on the type of practice they operate and the purpose of their evaluations. The ANX/DEP composite scale may offer the greatest utility among PCPs for whom the diagnosis and treatment of anxiety or depression is beyond the scope of their practice and expertise and are primarily interested in knowing when to refer or not refer a child for a possible internalizing disorder.26,27 For these PCPs, however, it is imperative that follow-up and ongoing consultation with referral sources occurs to inform the ongoing evaluation and treatment for ADHD.
Although the use of a total sum score provided the greatest clinical utility for each of the ADHD comorbidities considered, the stability of these results cannot be determined by a single study. In particular, the rates of depression and CD as determined by the DISC interview were low (3 and 4%, respectively), and as such results related to these comorbidities should be considered preliminary until replicated. At a minimum, our results suggest that ongoing efforts are needed to develop measures that maximally balance the need for administration efficiency and clinical utility. For instance, a total sum score approach substantially increased the sensitivity of various thresholds compared to the AAP/NICHQ cutoff recommendations, but this approach in turn lowered the specificity for CD and the combined ANX/DEP scale (but not the separated anxiety and depression scales). As noted earlier, this trade-off is expected but may be difficult to implement among PCPs given the added burden of a more sensitive (rather than specific) assessment strategy. Of importance are the findings for ODD and separated anxiety and depression, as results using a total sum score approach for these scales showed substantial increases in sensitivity with very little corresponding decrease in specificity. Overall, these results are in line with current DSM-5 efforts to develop dimensional approaches to psychopathology assessment in conjunction with categorical diagnostic decision-making28 (see also Diagnostic Assessment Instruments Study Group, www.dsm5.org).
Several sample characteristics and study limitations should be considered. First, our sample consisted of stimulant-naïve children, and although such children are likely to present to PCPs for their initial evaluation, they may not be representative of all children referred for ADHD evaluations or ongoing treatment. However, the rates of comorbidity among our participants with ADHD were similar to rates found in several other community samples.29 Also, our sample consisted of children aged 7 to 11 and our results may not be generalizable to younger or older children. Replication among samples with older youth where CD and depression are more prevalent will be especially important. Last, the present study examined the VADPRS for its clinical utility, and future research will need to examine the teacher-report Vanderbilt for its unique clinical utility or added clinical utility when used in conjunction with the VADPRS, particularly since our study may be subject to mono-informant biases. In providing best-practice for children with ADHD and other mental health conditions, obtaining reports from multiple informants is essential in conjunction with ongoing monitoring and care.
In conclusion, the VADPRS comorbidity scales are intended to screen for potentially clinically impairing symptoms of ODD, CD, anxiety, and depression. Although not intended to be a diagnostic measure, the VADPRS can be useful for pediatricians and clinicians as a screening tool that can help identify which children are at-risk or not at-risk for diagnoses of ODD, CD, anxiety, and/or depression. Given that most pediatricians consider the treatment of childhood mental health problems, with the exception of ADHD, to be beyond the scope of their practice,26,27 the clinical utility of the VADPRS to identify children who do not need additional evaluation is of clinical and practical importance. Our results can be used to empirically guide physicians in their ability to make appropriate referrals in order to avoid high rates of underidentification. Across the ODD, CD, and ANX/DEP comorbidity screens, our results indicated that a total sum score strategy provides a higher degree of clinical utility than the recommended AAP/NICHQ cutoff strategy, with results related to ODD the most conclusive. Given recent research showing that up to 30% of PCPs do not regularly assess for coexisting conditions within the context of an ADHD evaluation,30, cf. 31 and less than 10% use rating scales to aid in the assessment comorbid problems,31 it is essential to provide physicians with empirically-based recommendations for time- and cost-efficient screening of multiple problems that often co-occur with ADHD.
Support: Funding provided by the National Institutes of Health (R01MH074770).
Financial Disclosure: The authors have no financial relationships relevant to this article to disclose.
Conflict of Interest: None.
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Stephen P. Becker, Department of Psychology, Miami University, Oxford, Ohio.
Joshua M. Langberg, Division of Behavioral Medicine and Clinical Psychology, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio and Department of Psychology, Virginia Commonwealth University, Richmond, Virginia.
Aaron J. Vaughn, Division of Behavioral Medicine and Clinical Psychology, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio.
Jeffery N. Epstein, Division of Behavioral Medicine and Clinical Psychology, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio.