PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Am J Geriatr Psychiatry. Author manuscript; available in PMC 2014 February 27.
Published in final edited form as:
PMCID: PMC3936673
NIHMSID: NIHMS551201

Interpreting the Clinical Significance of Capacity Scores for Informed Consent in Alzheimer Disease Clinical Trials

Abstract

Objective

Among Alzheimer disease (AD) patients enrolled in a clinical trial, the authors assessed the ability of a standardized capacity assessment procedure to identify persons who are capable of giving their own informed consent.

Design

Cross-sectional interview.

Setting

Thirteen sites participating in a randomized and placebo controlled study of simvastatin for the treatment of mild to moderate AD.

Participants

Persons with mild to moderate AD and their study partners enrolled in the simvastatin clinical trial.

Measurements

Interviews to assess decision-making capacity using the MacArthur Competency Assessment Tool for Clinical Research (MacCAT-CR).

Results

Judges blinded to the subject’s clinical status had a high rate of agreement on patients capable of giving their own informed consent (κ = 0.73). The understanding subscale had the best receiver operator characteristic and an analysis of positive and negative predictive values over a range of hypothetical prevalences of incapacity to consent demonstrated the value of a range of understanding cut-points.

Conclusion

Among mild to moderate AD patients, enrolled in an actual clinical trial, these results suggest evidence based guidelines for using the MacCAT-CR understanding subscale to help guide judgments about whether a patient has the capacity to consent.

Keywords: Informed consent, decision making capacity, Alzheimer disease

Alzheimer disease (AD) is common, chronic, and a growing cause of substantial morbidity and mortality.13 Extensive research efforts are underway to develop better treatments.46 The success of these efforts rests on enrolling AD patients as research subjects, but the very problems caused by AD that warrant research also present ethical challenges to that research.

The patients’ cognitive impairments may significantly impair their capacity to provide informed consent.79 The best studied instrument to assess research consent capacity is the MacArthur Competency Assessment Tool for Clinical Research (MacCAT-CR).10 Several studies have shown the instrument’s reliability and validity in persons with very mild to moderate AD.79

The next step in this research is to show how well this instrument can be applied to the actual practice of clinical research. Specifically, clinical investigators need to know how performance on the MacCAT-CR instrument can guide categorical determinations of the ability of a person with AD to give his or her own informed consent. These data should come from a population who resembles the population of people who will undergo the test.11 Below, we report the predictive ability of a standardized MacCAT-CR capacity assessment in persons participating in an AD clinical trial.

METHODS

Subjects

Subjects of this study were AD patients and their study partners enrolled at 13 sites participating in the Alzheimers Disease Cooperative Study (ADCS) Cholesterol Lowering Agent to Slow Alzheimers Progression (CLASP) study, an 18 month long, randomized and placebo controlled study of simvastatin for the treatment of mild to moderate AD. Patients had National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer’s Disease and Related Disorders Association (NINCDS-ADRDA) probable AD12 with a mini-mental state examination (MMSE) scores13 between 12 and 26 and were permitted to be on stable doses of current AD therapies. Study partners were the persons, typically a family member such as a spouse or adult child, who served as a knowledgeable informant for the CLASP study.

Data Gathering

Study staff obtained a formal, site-specific informed consent to enroll in the CLASP study. At the month 1.5 CLASP study visit, patients and their study partners were recruited to participate in interviews to assess decision making capacity using the MacCAT-CR. The persons responsible for informed consent at the study site performed face-to-face interviews with patients, and a research assistant at a central study site interviewed study partners over the telephone.

The Assessment of Decision Making Capacity

The MacCAT-CR is a semistructured interview that assesses the four decision making abilities for research consent capacity: choice, understanding, appreciation, and reasoning.10 The assessment of these abilities is in turn used to judge whether a person has adequate capacity to provide his or her own informed consent.

“Understanding” describes the ability to factually comprehend the information disclosed. “Appreciation” describes the ability to apply the facts to oneself. “Reasoning” describes the abilities to compare and to describe the consequences of options. “Choice” describes the ability to consistently select and communicate an option.

The MacCAT-CR was adapted to fit the specific information of the clinical trial. Examiners received training in the administration of the MacCAT-CR at the CLASP investigators’ meeting that included reviewing a sample interview and conducting a follow-up telephone training session with this study’s investigators.

The patient and study partner each had a five page copy of a summary of the CLASP trial written at a 10th grade level and organized according to the sections of an informed consent form. After the interviewer read each section aloud, the interviewer asked if the subject had any questions, answered them and then administered the MacCAT-CR understanding question for that section. We modified the usual MacCAT-CR administration procedures by allowing subjects to retain the summary as they answered these questions and modified the scoring criteria to require subjects to paraphrase the information. After answering the understanding questions, the interviewer then asked the subject the MacCAT-CR appreciation, choice, and reasoning questions. A trained rater scored all of the MacCAT-CR interviews using standard scoring criteria for the four decision making ability measures. This rater was blinded to patient clinical measures and to patient or study partner status.

To assess whether the patient was capable of providing his or her own informed consent, three consultation liaison psychiatrists with at least 5 years of post fellowship experience in capacity assessment independently reviewed the audiotaped patients’ MacCAT-CR interviews and answered the question: “Based on the interview you just heard, do you believe that the subject has sufficient capacity to give his own informed consent to enroll in the CLASP trial?” Because capacity is a dichotomous status assessed from the continuum of abilities, the raters selected one of the following judgments “definitely has sufficient capacity, probably has sufficient capacity, probably does not have sufficient capacity, definitely does not have sufficient capacity,” and these answers were collapsed into “capable of consent” or “not capable of consent” for analysis. We created a criterion standard of capable or not capable using the agreement of at least 2 of 3 judges’ competency judgments.7 For each rating, the judges reported their confidence that the interview gave them sufficient information on a scale from 1, “not at all confident” to 10, “completely confident.” These experts were blinded to the subjects’ scores on measures of decision making ability and cognition. They had no prior contact with the subjects.

Data Analyses

We used the rank sum or Fisher’s exact test to compare patient and study partner MacCAT-CR scores. We used the κ statistic to examine the agreement among the three independent expert raters judgment of whether a patient was capable or not capable of providing an informed consent. We compared this value to the effect size standards developed by Landis: excellent (>0.8), good (0.61–0.8), moderate (0.41–0.6), and fair to poor (<0.4).14

To examine how well MacCAT-CR scores predict expert judgments of patient capacity to provide their own informed consent we performed a receiver operator characteristic (ROC) analysis that used the dichotomous categorization of “capable of informed consent” versus “not capable of informed consent.” The closer the area under the ROC curve is to 1, the better the ability of a test to discriminate between those with and without capacity. Generally, an area of 0.9–1.0 indicates excellent discrimination, and 0.8 – 0.9 good discrimination. To demonstrate how the sensitivity and specificity data generated from the ROC analysis can be applied to potential research scenarios, we calculated the positive and negative predictive values (PPV and NPV) for each ability over a range of hypothetical prior probabilities that a patient is not capable of informed consent for three cut-off points on the subscale with the best ROC performance, using the following formulas:

equation M1

equation M2

Human Subjects Protections

The project was approved by each study site’s institutional review board. All study partners provided informed consent and all patients provided either informed consent or assent to participate in this interview study.

RESULTS

Patients (N = 59) and study partners (N = 60) typically reported at least some years of college education (14.1 ± 2.8 [range: 6–20] and 15.0 ± 2.8 [range: 8–20]). The majority of study partners were spouses (75%). Fifty-four (92%) of the patients and 55 (92%) of the study partners were of white race. All other participants identified as African American. By standard MMSE cut-points, 31% of the patients (N = 18) had very mild, 32% (N = 19) mild, and 37% (N = 22) moderate stage disease (analyses by stage of disease includes the two subjects in the moderate stage group whose MMSE scores fell below 12 at the month 1.5 visit).

Patient Performance on the MacCAT-CR

Table 1 shows there was substantial variability in patient performance and performance was worse than study partners on all MacCAT-CR measures except for performance on the simplest measure, the ability to express a choice.

TABLE 1
Comparison of AD Patient and Caregiver Performance on Measures of Decision Making Ability

Expert Judgments of Patients’ Ability to Provide Informed Consent

The three expert judges individually rated 36 of 59 (61%), 31 of 59 (53%) and 28 of 59 (47%) patients capable of consent. There was good overall agreement in these judgments as shown by group κ = 0.73, (95% confidence interval [CI] 0.57–0.84), and the judges were generally confident that the interview gave them sufficient information as median confidence ratings on a scale from 1, “not at all confident” to 10, “completely confident” were 9.0, 8.0, and 8.0. A consensus of at least two of three judges found 31 of 59 (53%) capable of providing informed consent.

Relationships Between Expert Judgment of Patient Ability to Give Informed Consent and Performance on the MacCAT-CR Abilities

Table 2 shows the relationships between the expert judgment of whether a patient was capable or not capable of consent and patient performance on the abilities to understand, appreciate, and reason. We did not perform the ROC analysis for the ability to make a choice because of the small range of scores and the high performance of both patients and study partners. All three subscales were strongly predictive of expert opinion. The understanding subscale had the highest point estimate for the area under the curve, 0.89 (95% CI 0.81–0.97), followed by reasoning, 0.85 (95% CI 0.75–0.95) and then appreciation, 0.82 (95% CI 0.72–0.93).

TABLE 2
Sensitivity and Specificity of Cutoff Scores on MacCAT-CR Subscales for Capacity to Provide Informed Consent Among AD Subjects (N = 59)

Table 3 shows the PPV and NPV across a range of prevalence of incapacity. We selected these ranges based on the published proportion of persons judged capable of consent using the MacCAT-CR instrument,7,8 and the results of this present study. We present the understanding subscale because it had the greatest point estimate for area under the curve.

TABLE 3
Positive and Negative Predictive Values (PPV and NPV) for Hypothetical Incapacity Probabilities for Three Cutoff Points on the Understanding Subscale

A high PPV means there is a low likelihood of mistakenly excluding a person who is capable of informed consent (the false positive rate is low). In contrast, a high NPV means there is a low likelihood of mistakenly enrolling an incapable person (the false negative rate is low). For example, at an understanding score of 18 and a prevalence of incapacity of 50%, the PPV of 78% means that 22% of persons excluded as not capable of giving their own informed consent will in fact be capable of doing so, and given the NPV of 78%, 22% of persons will be included who are actually not capable of giving their own informed consent.

CONCLUSIONS

Our study has two main findings. First, in a group of patients actually participating in a clinical trial, the MacCAT-CR understanding subscale had the best ability to discriminate which subjects were capable of giving their own informed consent. However, across the range of MacCAT-CR understanding cut scores we examined, no single cut-point emerged as the “best” in terms of both PPV and NPV.

Clearly, the choice of cut-point must be informed by an estimate of the prevalence of incapacity and the kind of error in judgment the research wants to avoid. For example, an understanding cut score of 16 would be useful in studies that want to minimize mistakenly excluding capable subjects. In contrast, an investigator would choose the cut score of 21 to minimize the error of including persons who are actually not capable of giving their own informed consent, such as a high risk or early phase study. In sum, reliance on a MacCAT-CR cut score alone is inappropriate. Instead, an investigator should use performance on the MacCAT-CR as evidence to guide what is ultimately an ethical judgment.

Second, in our study sample, nearly half (47%) of subjects were judged not capable of providing their own consent. Although the proportion compares to the proportion found in a study of mild to moderate patient capacity to consent to a hypothetical early phase clinical trial (40%),8 this proportion could vary depending on the risk of the study. To the extent that clinicians follow a risk-sensitive model of capacity determination,15 an increase in either the complexity of the trial or the risk and benefit profile would likely decrease the number of AD patients who would be deemed to have capacity to provide consent and therefore affect the PPV and NPV. Nonetheless, the CLASP trial these subjects were participating in fits within a broad category of studies that present reasonably low research risks.

The proportion judged competent could also vary based on the way the person obtaining informed consent administers the MacCAT-CR. Our administration procedures allowed subjects to retain the CLASP study description while they answered the understanding questions, and correct answers required the subject to paraphrase the information. As a result, understanding performance did not involve significantly a test of a subject’s short term memory. Impairments in understanding likely reflect the marked losses in attention, concentration, and executive functions seen in persons with moderate AD in addition to problems with short term memory.16

Compared with previous studies, our receiver operator curve areas are lower.17,18 The reason for this finding is likely that 13 different interviewers performed the MacCAT-CR interviews. Although the number of sites was necessary to gather research consent capacity data from a population of people who are enrolled in a real clinical trial, variation across sites may introduce variance in the MacCAT-CR measures. Additional training of the persons administering the MacCAT-CR could reduce this variance. Finally, although we have no expectation that capacity should vary as a function of race or ethnicity, our results come from just one ethnicity. These findings reflect the racial make up of persons enrolled in the parent clinical trial of simvastatin to which this study was attached.

Nonetheless, our results and the results of a similar study of MacCAT-CR thresholds in persons with schizophrenia participating in a clinical trial18 suggest that the MacCAT-CR understanding subscale can assist persons who perform informed consent in making evidence based decisions. Additional studies examining the PPV and NPV of MacCAT-CR cut-points in clinical trials of different degrees of risks and benefit could provide AD researchers guidance on cut-points.

Acknowledgments

The authors thank the patients and families who participated in this study; Mary Sano, Ph.D., the principal investigator of CLASP; Allison Ferguson, Tessy Thomas, and Emily Murphy who assisted with data analyses; the ADCS steering committee and the following ADCS sites who participated in this study: Francine Parfitt M.S., C.C.R.C. (Mayo Clinic Jacksonville, FL), Leon Thal, M.D. (University of California San Diego, CA), Judith Heidebrink, M.D. and Norman Foster, M.D. (University of Michigan), Va. Buckles, Ph.D. and John Morris, M.D. (Washington University), Ruth Mulnard R.N., D.N.Sc. and Michael Grundman, M.D., M.P.H. (University of Califormia Irvine), Martin Farlow, M.D. (University of Indiana), Myron Weiner, M.D. (University of Texas, Southwestern) Neelum Aggarwal, M.D. (Rush University), and Martin MacAvoy, Ph.D. (Yale University).

This work was supported by the ADCS (U01-AG10483).

Dr. van Dyck currently receives research grant support from Eisai/Pfizer, Elan Pharmaceuticals, Eli Lilly, Glaxo-SmithKline, Myriad Pharmaceuticals, Neurochem, and Sanofi-Synthelabo, and serves as an unpaid consultant to Bristol-Myers Squibb and Forest Laboratories. Mr. James is funded through NIH training grant T32 AG00247. Dr. Karlawish serves on a DSMB for Myriad. Dr. Knopman has been a consultant to GE HealthCare, GlaxoSmithKline and Myriad Pharmaceuticals, has served on a Data Safety Monitoring Board for Sanofi-Aventis and Neurochem Pharmaceuticals, and is an investigator in a clinical trial sponsored by Elan Pharmaceuticals.

Footnotes

Dr. Marson and Dr. Kim report no conflicts of interest.

References

1. Wolfson C, Wolfson DB, Asgharian M, et al. A reevaluation of the duration of survival after the onset of dementia. N Engl J Med. 2001;344:1111–1116. [PubMed]
2. Schulz R, Mendelsohn AB, Haley WE, et al. End-of-life care and the effects of bereavement on family caregivers of persons with dementia. N Engl J Med. 2003;349:1936–1942. [PubMed]
3. Schulz R, O’Brien A, Bookwala J, et al. Psychiatric and physical morbidity effects of dementia caregiving: prevalence, correlates, and causes. Gerontologist. 1995;35:771–791. [PubMed]
4. Gauthier S. Do we have a treatment for Alzheimer Disease? Yes Arch Neurol. 1999;56:738–739. [PubMed]
5. Pryse-Phillips W. Do we have drugs for dementia? No Arch Neurol. 1999;56:735–737. [PubMed]
6. Mayeux R, Sano M. Treatment of Alzheimer’s disease. N Engl J Med. 1999;341:1670–1679. [PubMed]
7. Kim SYH, Caine ED, Currier GW, et al. Assessing the competence of persons with Alzheimer’s disease in providing informed consent for participation in research. Am J Psychiatry. 2001;158:710–717. [PubMed]
8. Karlawish JHT, Casarett DJ, James BD. Alzheimers Disease patients’ and caregivers’ capacity, competency and reasons to enroll in an early phase Alzheimers Disease clinical trial. J Am Geriatr Soc. 2002;50:2019–2024. [PubMed]
9. Palmer BW, Dunn LB, Appelbaum PS, et al. Assessment of capacity to consent to research among older persons with schizophrenia, Alzheimer disease, or diabetes mellitus: comparison of a 3-item questionnaire with a comprehensive standardized capacity instrument. Arch Gen Psychiatry. 2005;62:726–733. [PubMed]
10. Appelbaum PS, Grisso T. The MacArthur Competence Assessment Tool—Clinical Research. Sarasota, Fla: Professional Resources Press; 2000.
11. Gordis L. Assessing the validity and reliability of diagnostic and screening tests. Epidemiology. Philadelphia: W.B. Saunders Company; 1996. pp. 58–88.
12. McKhann G, Drachman D, Folstein M, et al. Clinical diagnosis of Alzheimer’s disease. Report of the NINCDS-ADRDA work group under auspices of the Department of Health and Human Services Task Force on Alzheimer’s disease. Neurology. 1984;34:939–944. [PubMed]
13. Folstein M, Folstein S, McHugh P. Mini-mental state: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12:189–198. [PubMed]
14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed]
15. Kim SY, Caine ED, Swan JG, et al. Do clinicians follow a risk-sensitive model of capacity-determination? An experimental video survey. Psychosomatics. 2006;47:325–329. [PubMed]
16. Earnst KS, Marson DC, Harrell LE. Cognitive models of physicians’ legal standard and personal judgments of competency in patients with Alzheimer’s disease. J Am Geriatr Soc. 2000;48:919–927. [PubMed]
17. Kim SY, Cox C, Caine ED. Impaired decision-making ability in subjects with Alzheimer’s disease and willingness to participate in research. Am J Psychiatry. 2002;159:797–802. [PubMed]
18. Kim SYH, Appelbaum PS, Swan J, et al. Determining when impairment constitutes incapacity for informed consent in schizophrenia research. Br J Psychiatry. 2007;191:38–43. [PubMed]