Internal consistency reliability
Tetrachoric factor analyses were calculated separately for inattention (AD) and hyperactivity-impulsivity (HD) criteria using the symptom reports of all adolescents and parents who screened into the AD and HD modules. For AD, results indicated a three-factor solution (unrotated eigenvalues: 8.01, 4.18, 1.23, 0.86), with the three rotated (promax) factors corresponding to parent reports of AD and two factors for youth reports (). The existence of two adolescent factors rather than one might indicate that distractibility is somewhat distinct from executive function problems. However, it is difficult to interpret the two factors in any clear conceptual way. As a result, we collapsed all items from these two factors into a single youth factor for the subsequent IRT analysis. The factor analysis of HD symptoms also indicated a three-factor solution (unrotated eigenvalues: 8.44, 3.33, 1.55, 0.72). The rotated (promax) factor solution corresponded to a parent HD factor and two factors for adolescents that separated hyperactivity from impulsivity, with the exception of one DSM hyperactivity item (often talks excessively) that loaded on the impulsivity factor.
Rotated (promax)1 tetrachoric factor analysis (standardized regression coefficients) of parent and youth symptom reports separately for symptoms of inattention and hyperactivity-impulsivity (n = 8470)
One-parameter (1PL) and two-parameter (2PL) IRT models were estimated for each of the two informants (adolescent and parent) on each of the two dimensions (AD and HD) (). Pearson chi-square statistics were calculated for the 1PL and 2PL models, comparing expected and observed outcomes. For both informants on the AD criteria and parents on the HD criteria, the 2PL model was a significantly better fit than the 1PL model. For adolescents on the HD criteria, the 1PL model was a significantly better fit than the 2PL model. Focusing first on the adolescent data, slopes for both the AD and HD factors are moderate (0.80–1.14 for AD and 0.91 for HD), indicating that none of the items is a strong indicator of the underlying dimension. (A slope of at least 1.0 is usually defined as the lower bound for an item that has good precision at its threshold on the underlying scale.) Thresholds were for the most part within one-third (±) of a standard deviation of the mean, indicating that most of the information in the scales is in a part of the severity distribution that is well below the clinical threshold. The conjunction of low slopes and sub-clinical thresholds indicates that the scale is not highly sensitive or specific in discriminating clinical cases from non-cases.
IRT model item parameters for adolescent and parent CIDI inattention and hyperactivity-impulsivity items1
Slopes were considerably higher in the parent data for both AD and HD factors (1.83–3.33 for AD and 1.34–3.39 for HD), indicating that the items have excellent precision at their thresholds. It is noteworthy that the existence of significant slope differences across items for both AD and HD means that optimal scaling would weight items differentially to arrive at an estimate of underlying scale scores. This is different from the stipulation in the DSM that each Criterion A symptom of AD and of HD contributes equally to a diagnosis. Like the slopes, the thresholds of the parent items were a good deal higher than in the youth data (0.81–1.24 for AD and 0.98–1.41 for HD), indicating that the parent scales have much better precision than the youth scales.
The fact that a high proportion of respondents endorsed none of the ADHD symptom questions raises the possibility that the IRT assumption of a normally distributed latent liability might be violated. Based on this concern, we fitted separate two-class IRT mixture models for the adolescent and parent HD and AD data, where one class was stipulated to consist exclusively of respondents outside of the AD or HD spectrum; that is, to have no risk of reporting any ADHD symptoms. The other class was assumed to consist of respondents in the ADHD spectrum. Respondents in the latter class were assumed to have a normally distributed latent liability (including some proportion that would be expected to endorse none of the CIDI symptom questions). Relatively small proportions of adolescent respondents who completed the symptom questions were estimated in this model to be outside the spectrum for AD (8.3%) or HD (3.7%), while much larger proportions were estimated to be outside the spectrum for the parent AD (50.4%) and HD (54.5%) scales. This substantial difference between adolescents and parents is presumably due to the fact, noted earlier, that screening questions were used in the assessment of adolescents but not parents.
The two-class mixture model was a better fit than the standard IRT model for both adolescent and parent AD and HD dimensions (based on Pearson χ2 tests comparing expected and observed values for 2PL and 2PL-mixture models). Eliminating respondents not in the spectrum from the database, we replicated the factor analyses for AD and HD and again identified a three-factor solution for AD (unrotated eigenvalues: 8.44, 3.33, 1.55, 0.72) and a four-factor solution for HD (unrotated eigenvalues 6.67, 3.44, 1.78, 1.16, 0.77). The rotated (promax) factor solution for AD had a factor pattern very similar to the one in the original factor analysis; one factor included all parent reports and two factors included adolescent reports, where we could find no meaningful interpretation of the distinction between items in the two adolescent factors. (Detailed results are not reported, but are available on request.) The rotated (promax) solution for HD, in comparison, was different from the original solution in that it differentiated symptoms of hyperactivity from symptoms of impulsivity both in parent reports and in adolescent reports (). Adolescent and parent primary factor loadings were very similar, with the exception of the impulsivity item ‘difficulty waiting turn,’ which loaded on the impulsivity factor in the adolescent data, but the hyperactivity factor in the parent data.
Table 3 Rotated (promax)1 tetrachoric factor analysis (standardized regression coefficients) for hyperactivity-impulsivity symptoms based on CIDI symptom reports in adolescent–parent pairs where both respondents were classified as being in the HD spectrum (more ...)
Concordance of CIDI symptom reports with clinician ratings
Adolescent responses to the CIDI questions about Criterion A symptoms of AD generally underestimated K-SADS prevalence (). This was due to sensitivity being uniformly low (16.4–35.1%; i.e. a low proportion of adolescents classified by the K-SADS as having a history of the symptom reporting the symptom in the CIDI). Specificity, in comparison, was generally quite good (94.5– 97.9%; i.e. a very high proportion of adolescents classified by the K-SADS as not having a history of the symptom denying the symptom in the CIDI). Concordance of adolescent symptom reports with K-SADS estimates was higher, in comparison, for Criterion A symptoms of HD, but this was as much because specificity decreased (89.3– 96.5%) as because sensitivity increased (21.4–42.9%) (). Parent reports generally overestimated the prevalence of K-SADS Criterion A symptoms of both AD and HD (). This occurred because of both higher sensitivity and lower specificity than in adolescent reports. Similar patterns were found when we generated criterion-level estimates from the symptom reports, including Criterion A1 (six or more symptoms of inattention), Criterion A2 (six or more symptoms of hyperactivity-impulsivity), Criterion B (some symptoms present before the age of seven), and Criterion C (clinically significant impairment) (). CIDI ratings based on adolescent reports overestimated the prevalence of Criterion A2, but underestimated the other criteria. CIDI ratings based on parent reports, in comparison, overestimated the clinical K-SADS diagnosis for all of these criteria except impairment.
Concordance (sensitivity and specificity) of CIDI/DSM-IV ADHD Criterion A inattention and hyperactivity-impulsivity symptoms with blinded K-SADS ratings in the NCS-A clinical reappraisal sample (n = 321)
Concordance (sensitivity and specificity) of criterion-level assessments of DSM-IV ADHD based on the CIDI with those based on blinded K-SADS interviews in the NCS-A clinical reappraisal sample (n = 321)
The CIDI diagnostic algorithms
Consistent with the results reported in the last section, diagnoses based on adolescent CIDI reports substantially underestimated the prevalence of DSM-IV/K-SADS ADHD (3.0% versus 7.8%, χ2 = 9.4, p < 0.05). Individual-level concordance was also poor (κ = 0.19, AUC = 0.57) (). Diagnoses based on parent reports, in comparison, were much more consistent with DSM-IV/K-SADS prevalence (8.0% versus 7.8%, χ2 = 0.2, p = 0.68), and individual-level concordance was much better than for adolescent reports (κ = 0.41, AUC = 0.71). Diagnostic estimates based on composite (i.e. adolescent–parent) CIDI reports were inflated (11.9% versus 7.8%, χ2 = 5.9, p < 0.05), but had better concordance with K-SADS diagnoses than those based on parent report alone (κ = 0.43, AUC = 0.77).
Diagnostic concordance of the original and modified DSM-IV ADHD diagnoses based on the CIDI with diagnoses based on the composite K-SADS in the NCS-A clinical reappraisal sample (n = 321)
We explored several options for modifying CIDI ratings to improve concordance between diagnoses based on the CIDI and the K-SADS. First, we considered the possibility of modifying the symptom thresholds in the CIDI Criterion A symptom reports, but this did not improve concordance. Second, we considered the possibility of using predictive logistic regression analysis to improve concordance of CIDI symptom reports with K-SADS diagnoses. These analyses showed clearly that CIDI parent reports were significant predictors of K-SADS diagnoses while CIDI adolescent reports were not, after controlling for parent reports. Third, based on this result, we focused subsequent analyses on parent reports and considered ways in which we might improve concordance with K-SADS diagnoses by cross-classifying Criterion A1 and A2 CIDI reports and selecting a higher diagnostic threshold than the DSM-IV six of nine A1 or A2 symptoms to address the fact that CIDI parent reports over-diagnose ADHD. In addition, we explored the effects of eliminating Criterion B from the diagnostic algorithm, as parents significantly overestimated this criterion, and the effects of modifying the Criterion C measure of impairment because of its low sensitivity.
The scoring rule that best predicted K-SADS diagnoses required 10 or more endorsed symptoms out of the 18 in the A1 and A2 series in conjunction with CIDI Criterion C modified to require ‘a lot’ of interference in at least one area of role functioning. We could not improve concordance by setting separate thresholds for CIDI A1 and A2 symptom counts, including Criterion B, or introducing additional information from the CIDI adolescent reports. The new algorithm yielded CIDI prevalence estimates of DSM-IV ADHD that did not differ significantly from diagnostic estimates based on the K-SADS (χ2 < 0.01, p = 1.00). This is a considerable improvement over the original CIDI diagnoses.