This study is a prospective cohort study comparing competence judgment by using observational techniques to outcomes of the MacCAT-CR for Children and Adolescents, while at the same time assessing competency-related patient data.
The validity and reliability of the translated and modified MacCAT-CR for Children and Adolescents will be assessed in a sample of children who are candidates to participate in ongoing medical trials. A reference standard for competence will be established first. The usual informed-consent procedure will be performed by the clinical investigators, and they will record their own intuitive clinical judgment of a child’s consent competence. At this point, parents will also be asked to judge whether their child has understood and is able to make a well-considered decision. This informed consent procedure will be videotaped. The recordings will then be reviewed independently by two experts (child psychiatrists, child psychologists or medical ethicists), who will also record a judgment on the child’s consent competence. This will allow us to estimate the inter-examiner reproducibility of the informed-consent procedure.
To establish a reference standard for consent competency, any discordant decisions for a child’s consent competency of the two experts and the clinician will be examined. If there is any discrepancy between the three evaluators’ judgments, a consensus decision will be reached after discussion. The final decisions will form the reference standard for competence.
After the usual informed-consent procedure, within 48 hours the MacCAT-CR interview will be administered by the researchers, independent of the first clinical judgment; these interviews will also be videotaped. The interviewers consist of specially trained special education or psychology graduates. The interviews will be scored by the administrator and independently by two experts (a multidisciplinary team of child psychiatrists, psychologists and medical ethicists). A yes/no decision will be made following the guidelines in the MacCAT-CR manual.
Demographic patient data will be collected and the Wechsler Nonverbal Scale of Abilities short version will be administered within two months of the interviews.
The final objectives are (1) to assess the reproducibility of MacCAT-CR scores and yes/no judgments of competence to consent, and (2) to establish a reference standard to which MacCAT-CR scores can be compared to evaluate the criterion-related validity of the instrument (in the absence of a criterion test for competence to consent). The reference standard will also be used to (1) estimate optimal cut-off scores on the MacCAT-CR scale that minimizes false positive and false negative decisions, and (2) determine ages for informed consent and compare these to current statutory age limits. The agreement between the reference standard and the judgment of the parent(s) will also be examined.
One hundred and sixty pediatric patients aged 6 to 18 who are involved in the selection stages of ongoing medical trials will be recruited. These will be trials that involve heterogeneous groups of children in terms of age and diagnosis (including childhood oncology, lung disease and ophthalmology). The children will be selected consecutively in the order of recruitment for the trials. The lower age limit of 6 is justifiable as younger children cannot be expected in developmental terms to be capable of meaningfully answering the interview questions. Age distribution will be structured in a way that approximately 70% of the sample (± 1 z-score) will be aged 8 to 14, because the transition point in competence is expected to occur there. The purpose is to avoid overestimating reproducibility and criterion validity as a result of excessive contrast in the age distribution. Grounds for exclusion will be insufficient fluency in Dutch.
The outcome measures of the MacCAT-CR will be a total score, domain scores, and a binary assessment (yes/no) of a child’s competence to consent. Clinical investigators and parents will also be asked to give their prior intuitive yes/no assessments of the child’s competence to decide.
The Wechsler Nonverbal Scale of Ability short version (WVN) will be used to assess children’s intelligence quotient. The WNV is a clinical instrument for examining cognitive capacities of children and adolescents aged 4 to 21. The WVN is suitable for the general population as well as for children with cultural, linguistic, educational or socio-economic varying backgrounds. The subtests do not invoke verbal capacities as instructions are made by pictograms. Different subtests are to be administered in children from 4 to 7 and from 8 to 21. The short version for the first age group consists of matrix reasoning and recognizing, and for the second age group matrix reasoning and spatial orientation [35
]. For practical reasons the short version is chosen: the two subtests take 20 minutes together. The full version gives more rigorous outcomes and has psychometrical advantages, but the validity and reliability of the short version are good. In this study the subjects cannot be burdened by the long version due to possible pain or distress. The WNV is the only IQ test with a standardized short version; this is in contrast to the Wechsler Intelligence Scale for Children. The WNV is approved by the Dutch Committee on Tests and Testing Affairs (COTAN). The WNV will be administered by trained certified professionals (special education or psychology graduates) under supervision of a senior professional.
The highest level of education of the highest-educated parent will be noted as an indicator of socio-economic status. Duration of disease, number of trials previously participated in and ethnicity will be noted.
Prior written consent to take part in this study will be requested from all child participants and their parents, separate from any consent required of them for the drug trials. The pediatric patients sampled for the drug trials will be informed together with their parents by the trained clinical investigator about the competence study at the same time as they are informed about the drug trial. They will also be free to choose participation in one of the two studies with no consequences for the other. The outcome of the MacCAT-CR interview will not affect the conduct of the drug trial being carried out by the investigator. The ethics committee of the Academic Medical Center Amsterdam in the Netherlands confirmed that the Medical Research Involving Human Subjects Act (WMO) does not apply to this study and the committee makes no objections to the implementation.
Descriptive summaries of demographic and assessment outcome measures will be generated with respect to all subject characteristics.
Reliability: Statistical analysis will include exploration of internal consistency, by estimating Cronbach’s alpha for the items of each subscale and for the total scale. We will also calculate adjusted item-to-scale total correlations. Factor analysis and item response theory methods (Rasch analysis) will be used to further test scale unidimensionality. To optimize the MacCAT-CR scoring system and to determine necessary item weights, we will use a specific extension of the Rasch measurement model One Parameter Logistic Model (OPLM) [36
]. Conditional maximum likelihood estimation methods will be used to obtain stable item parameters.
Criterion-related validity: The overall accuracy of the MacCAT-CR score in classifying competence against the reference standard will be quantified using receiver operator characteristic curve (ROC) analysis. The area under the ROC curve (AUC) will serve as the validity coefficient; this may range from 50% (chance determination) to 100% (perfect determination). The optimal MacCAT cut-off score and the accompanying sensitivity and specificity rates will be determined using Youden’s method, the cut-off score corresponding to the fewest false positive and false negative classifications. Against our current expectations, this may give an indication of a single score above which competence is more likely. The validity of the current statutory cut-off ages will be tested by the same method, but using age as the predictor of capacity to provide informed consent.
Inter-rater reproducibility: We will determine inter-rater reproducibility (1) for the clinical judgment on competence to consent by the investigator and the experts, (2) for the MacCAT-CR total and subscale scores, (3) for the yes/no outcome of the reference standard and the yes/no outcome of the MacCAT-CR, and (4) for the yes/no outcome of the reference standard and the yes/no decision of the parents. We will quantify reproducibility with intraclass correlation coefficients for total scores on the MacCAT-CR scales. Inter-rater reproducibility of the item scores will be quantified by calculating weighted Kappa coefficients. To quantify the reproducibility of the yes/no outcome for competence to consent, multi-rater (unweighted) Kappa or simple Cohen’s Kappa will be used in the case of pairwise comparisons. We have a special interest in children between 8 and 14 years, and we will compare the ICCs and Kappa values calculated for children in this age group separately.
Statistical power analysis: There is no general agreement about estimating suitable sample size for the psychometric (factor-analytic, Rasch analytic) evaluation of multi-item scales. Simulation studies for the related techniques of regression analysis indicate that a minimum of 10 to 15 observations per variable (item) are needed to obtain stable estimates. For the 13 items of the MacCAT-CR this would result in 130 to 190 observations. Judging from previous studies on MacCAT-CR and MacCAT-T in adults with compromised decisional capacities [18
], our proposed sample size of N
160 is justifiable. In view of the three raters involved in our assessment of inter-rater reproducibility, the intraclass correlation for the MacCAT-CR score can be estimated with ±5% accuracy around the expected level of 0.80 with 95% certainty. The Kappa for the yes/no capacity to consent decision can be estimated with ±11% accuracy and 95% certainty assuming 60% raw agreement between two raters and an expected Kappa of 0.70.
An ROC-AUC validity statistic of 0.80 (null hypothesis AUC 0.69) can be detected with 80% power assuming a 3:1 ratio of test positive (competent to consent) and test negative children. Sensitivity and specificity rates of the MacCAT-CR cut-offs obtained by Youden’s method can be estimated with ±7% accuracy, assuming 0.75 as the expected value. For the Rasch analysis, the MacCAT-CR outcomes from the raters will be combined. Data dependency will not be an issue in our use of conditional maximum likelihood estimation methods, because the method makes no assumptions about the distribution of data in the population or about ways in which the sample was selected.