|Home | About | Journals | Submit | Contact Us | Français|
To compare results of a specific capacity assessment administered by the treating clinician, and a Standardized Mini-Mental Status Examination (SMMSE), with the results of expert assessments of patient capacity to consent to treatment.
Cross-sectional study with independent comparison to expert capacity assessments.
Inpatient medical wards at an academic secondary and tertiary referral hospital.
One hundred consecutive inpatients facing a decision about a major medical treatment or an invasive medical procedure. Participants either were refusing treatment, or were accepting treatment but were not clearly capable according to the treating clinician.
The treating clinician (medical resident or student) conducted a specific capacity assessment on each participant, using a decisional aid called the Aid to Capacity Evaluation. A specific capacity assessment is a semistructured evaluation of the participant’s ability to understand relevant information and appreciate reasonably foreseeable consequences with regard to the specific treatment decision. Participants also received a SMMSE administered by a research nurse. Participants then had two independent expert assessments of capacity. If the two expert assessments disagreed, then an independent adjudication panel resolved the disagreement after reviewing videotapes of both expert assessments. Using the two expert assessments and the adjudication panel as the reference standard, we calculated areas under the receiver-operating characteristic curves and likelihood ratios. The areas under the receiver-operating characteristic curves were 0.90 for specific capacity assessment by treating clinician and 0.93 for SMMSE score (2p = .48). For the treating clinician’s specific capacity assessment, likelihood ratios for detecting incapacity were as follows: definitely incapable, 20 (95% confidence interval [CI] 3.6, 120); probably incapable, 6.1 (95% CI 2.6, 15); probably capable, 0.39 (95% CI 0.18, 0.81); and definitely capable, 0.05 (95% CI 0.01, 0.29). For the SMMSE, a score of 0 to 16 had a likelihood ratio of 15 (95% CI 5.3, 44), a score of 17 to 23 had a likelihood ratio of 0.68 (95% CI 0.35, 1.2), and a score of 24 to 30 had a likelihood ratio of 0.05 (95% CI 0.01, 0.26).
Specific capacity assessments by the treating clinician and SMMSE scores agree closely with results of expert assessments of capacity. Clinicians can use these practical, flexible, and evaluated measures as the initial step in the assessment of patient capacity to consent to treatment.
Decision-making capacity is the ability to understand relevant information and to appreciate the reasonably foreseeable consequences of a decision.1 Assessment of patient capacity to consent to treatment is a fundamental component of shared physician-patient decision making. Capable patients have the legal and ethical right to make their own treatment decisions. Incapable patients cannot give a valid consent, so someone else must make the decision on the patient’s behalf. Although capacity assessments have serious implications in clinical practice, these assessments have not been adequately evaluated.
Treating clinicians are responsible for the important initial assessment of their patient’s capacity to consent to treatment. Clinicians may form a general impression of capacity during a typical clinical encounter with a patient. Existing studies indicate that these general impressions are easily biased,2 and do not agree closely with expert assessments except when patients are obviously capable or incapable.3, 4 Better approaches to capacity assessment are needed.
One approach is a specific capacity assessment, which is a semistructured evaluation of the person’s ability to understand relevant information and appreciate reasonably foreseeable consequences with regard to the specific treatment decision.5 Some existing measures of specific capacity have not been evaluated on patients,6 while others have been evaluated by trained researchers and experts on patients making decisions about specific psychiatric treatments.7–9 There are no studies of specific capacity assessment that use the treating clinician as the assessor, and no studies that focus on medical patients who are facing actual decisions about medical treatment.
Another approach to capacity assessment is the use of nonspecific cognitive status screening tests, such as the Mini-Mental Status Examination (MMSE). These cognitive status tests have uncertain value for assessing capacity to consent to treatment,10, 11 although MMSE scores agree closely with expert assessments of capacity to complete an advance directive.12
The primary objective of this study was to compare results of specific capacity assessments by treating clinicians, MMSE scores, and an a priori combination of these two tests with results of expert assessments of patient capacity to consent to treatment. We also obtained specific capacity assessments by a trained research nurse and general impressions of capacity from the staff physicians. We studied medical inpatients facing an actual decision about major medical treatment for whom the issue of capacity was clinically relevant and important.
The study was conducted on the general internal medicine inpatient units of The Toronto Hospital from December 1, 1994 to December 20, 1995.
We enrolled consecutive English-speaking patients who were facing nonemergency decisions regarding major medical treatments or invasive medical procedures. We sought an appropriate spectrum of patients on whom capacity assessments would be conducted in usual clinical practice. We included patients who were refusing treatment and patients who were accepting treatment but were not clearly capable according to their treating clinician.1, 13 We excluded patients who were accepting treatment if the clinician was sure that the patient was capable, because capacity is rarely at issue in these patients, who are almost always found capable by independent psychiatric assessment.14 This group of patients is not clinically challenging, so including them would have falsely elevated the agreement between the various study measures. We also excluded patients who were comatose, unable to communicate, suicidal, violent, unable to hear despite assisting devices, under 16 years of age, or had previously been declared legally incompetent to consent to treatment.
We consulted experts from law, ethics, and medicine to develop a specific capacity assessment method, called the Aid to Capacity Evaluation (ACE). The ACE is a semistructured decisional aid that prompts inquiry into seven relevant areas (Table 1) For each area the ACE provides suggested questions, guidelines for scoring patient responses, and scoring examples. In each area the clinician gives a rating of yes, unsure, or no. After assessing each area as completely as possible, the clinician chooses one of four possible overall assessments: definitely incapable, probably incapable, probably capable, or definitely capable. We did not provide a specific method for arriving at the overall assessment, but we advised clinicians that the patient should show the ability to understand relevant information and appreciate reasonably foreseeable consequences, based on the existing legal standard in Ontario, Canada. This is a higher standard of capacity than in some other jurisdictions.15(The complete ACE, including training materials, is available from the corresponding author on request, or at the University of Toronto Joint Centre for Bioethics Web site: www.utoronto.jcb/)
Medical residents and senior medical students attended a 1-hour ACE training session, then were asked to conduct specific capacity assessments, using the ACE, on their patients who participated in the study. To evaluate the reliability of the ACE assessments, each patient-participant also received an independent ACE assessment by the research nurse, who was trained to administer the ACE.
Each patient-participant also received a Standardized Mini-Mental Status Examination (SMMSE).16 The result is a score from 0 to 30. The SMMSE was administered by the research nurse after she completed her ACE assessment.
We obtained a general impression of capacity from the staff physician who supervised the clinical care of the patient-participant. Possible results of the general impressions were definitely incapable, probably incapable, probably capable, and definitely capable.
Two experts conducted videotaped capacity assessments on each patient-participant. The two experts had not worked together previously. Each expert used an independently derived method for conducting their capacity assessments. We instructed the experts not to discuss their methods or opinions regarding capacity assessment for the entire course of the study.
Expert A was the Baycrest Competency Clinic,17 which uses a decisional aid developed by two consensus multidisciplinary panels.18, 19 The nurse from the Competency Clinic conducted the assessments, which were then reviewed with the Competency Clinic psychiatrist. The nurse and psychiatrist had conducted approximately 2,000 assessments over the previous 8 years. Results of the SMMSE were provided to expert A because cognitive status tests were part of the usual assessment protocol, so the comparison between the SMMSE and expert A is potentially biased. However, expert A had no involvement in the development of the ACE or the ACE training sessions and was unaware of the results of the ACE assessments or the general impressions, so the comparison between the ACE and expert A is not biased.
Expert B was an internist with a special interest in capacity assessment. He used a diagnostic interview and decisional aid that was initially developed by a multidisciplinary panel for the Ontario government capacity assessment office. He had conducted approximately 200 capacity assessments for research and clinical purposes. Expert B did not administer formal cognitive tests during the assessment and was unaware of the SMMSE results, so the comparison between the SMMSE and expert B is not biased. Expert B developed the ACE and conducted the ACE training sessions, so the comparison between expert B and ACE assessments is potentially biased. This bias was minimized by ensuring that expert B had no involvement in the performance of ACE assessments by the treating clinicians or the research nurse and was unaware of the results of the ACE assessments or the general impressions.
If the two experts agreed that the patient was capable or incapable, then no further assessment occurred. Disagreements between the two experts were resolved by an independent adjudication panel composed of a psychiatrist, a geriatrician, and a bioethicist. The psychiatrist has conducted approximately 300 capacity assessments as a liaison consultant to medical and surgical services, teaches capacity assessment to psychiatry trainees, and has a research interest in treatment refusal.20 The geriatrician has extensive clinical and research experience in assessing capacity.6, 18 Panel members reviewed both videotaped assessments, discussed the case with each expert, then reached a consensus decision regarding the participant’s capacity. The panel could also decide that the participant’s status fluctuated between the expert assessments, in which case the participant was excluded from the final analysis.
We minimized the potential for bias during the panel’s deliberations. None of the panel was involved in the development of either expert assessment protocol, and the panel did not know the results of the ACE assessments, the SMMSE, or the general impressions. Expert A was specifically instructed not to discuss the SMMSE score with the adjudication panel.
The research nurse obtained relevant medical information for each participant, then reviewed this information with participants prior to the assessments. Assessors were free to further review the information with participants. In most cases, the order of assessment was (1) ACE by research nurse, (2) SMMSE by research nurse (3) ACE by treating clinician (i.e., the medical resident or student caring for the participant), (4) expert A, and (5) expert B. The research nurse obtained the general impression from the staff physician while the other study assessments were being conducted. Every effort was made to complete all assessments as quickly as possible, usually within 24 hours.
For interobserver reliability, we calculated κ statistics.21 To estimate overall agreement with the expert assessments, we calculated the area under the receiver-operating characteristic (ROC) curves using the ROC Analyzer Software nonparametric option. We compared areas under the ROC curves using the method of Hanley and McNeil.22 We calculated likelihood ratios for the ACE and SMMSE using ROC Analyzer Software, and 95% confidence intervals (CIs) using the maximum likelihood option.23 For clarity, we also calculated the pretest and posttest probabilities for each ACE and SMMSE result. Cutoff values for SMMSE scores and for combinations of ACE and SMMSE scores were based on previous studies and were established in advance.12, 14
The Toronto Hospital Committee for Research on Human Subjects approved the protocol. Proxy consent from family members was obtained for participants who appeared unable to consent but who were willing to participate.
We approached 184 potentially eligible participants. Seventy-five were excluded for the following reasons: patient or family refusal (54), deteriorating medical status or hospital discharge (20), and physician refusal (1). Nonparticipants were more likely to be refusing treatment (74% vs 50%, p= .004) and disagreeing with their physicians regarding the treatment decision (49% vs 28%, p= .006).
Of the 109 participants enrolled, 6 did not complete the second expert assessment, and 3 were excluded by the adjudication panel because of fluctuating status. Characteristics of the 100 remaining participants are listed in Table 2). Results are unchanged if we include the 6 participants who did not complete the second expert assessment.
The agreement between the expert assessments was 82%, and the κ was 0.63 (lower 95% confidence limit 0.47). The 18 disagreements were resolved by the adjudication panel; the adjudication panel concurred with one expert in 7 cases and with the other expert in 11 cases.
The median time for ACE assessments by the treating clinician was 15 minutes (interquartile range 10–20 minutes). The agreement between ACE assessments by the treating clinician and the trained research nurse was 93% (κ 0.79, lower 95% confidence limit 0.63).
We compared the ACE and SMMSE results to each expert’s assessments, as well as to an overall expert opinion, i.e., agreement between the two experts or the decision of the adjudication panel (Table 3 For all comparisons, the area under the ROC curve for the ACE administered by the research nurse was significantly higher than the area under the ROC curve for the general impression of the staff physician. For expert A’s assessments, the area under the curve for the SMMSE was significantly higher than the area under the curve for the general impression by the staff physician. Otherwise, there were no statistically significant differences between the various study measures.
Using the results of the overall expert opinion (agreement between both experts or the decision of the adjudication panel), the prevalence (pretest probability) of incapacity was 37 (37%) of 100 (Table 4) An ACE result of definitely incapable significantly increased the likelihood of incapacity (posttest probability 92%, likelihood ratio 20), while an ACE result of definitely capable significantly reduced the likelihood of incapacity (posttest probability 3%, likelihood ratio 0.05). Overall, 43 participants were correctly classified as incapable or capable by ACE results of definitely incapable or definitely capable, respectively, with two misclassifications.
An ACE result of probably incapable also increased the likelihood of incapacity (posttest probability 78%, likelihood ratio 6.1), but five capable participants had an ACE result of probably incapable. An ACE result of probably capable reduced the likelihood of incapacity (posttest probability 19%, likelihood ratio 0.39), but six incapable participants had an ACE result of probably capable. Overall, 55 participants had intermediate ACE results of probably incapable or probably capable.
We also compared the ACE assessments by the research nurse to the expert assessments of capacity. The likelihood ratios (95% CIs) for the ACE assessments by the research nurse were as follows: definitely incapable, ∞ (95% CI 2.5, ∞); probably incapable, 9.5 (95% CI 4.3, 23); probably capable, 0.09 (95% CI 0.03, 0.32); and definitely capable, 0.0 (95% CI 0.0, 0.34).
Two participants refused SMMSE testing, so the prevalence (pretest probability) of incapacity was 35 (36%) of 98. An SMMSE score of 0 to 16 significantly increased the likelihood of incapacity (posttest probability 89%, likelihood ratio 15), while a score of 24 to 30 significantly reduced the likelihood of incapacity (posttest probability 3%, likelihood ratio 0.05) (Table 5) Overall, 61 participants were correctly classified as incapable or capable by SMMSE scores of 0 to 16 and 24 to 30 respectively, with four misclassifications. An SMMSE score of 17 to 23 had little effect on the likelihood of incapacity (posttest probability 27%, likelihood ratio 0.68). Thirty-three participants had an intermediate SMMSE score of 17 to 23.
The combination of ACE and SMMSE was established a priori. An ACE result of probably or definitely incapable combined with an SMMSE score of 0 to 16 significantly increased the likelihood of incapacity (posttest probability 96%, likelihood ratio 40), while an ACE result of probably or definitely capable combined with an SMMSE score of 24 to 30 significantly reduced the likelihood of incapacity (posttest probability 3%, likelihood ratio 0.05) (Table 6) All other combinations of results had little effect on the likelihood of incapacity (posttest probability 31%, likelihood ratio 0.80). Overall, 57 participants were correctly classified as capable or incapable by the combination of ACE and SMMSE, with two misclassifications.
We found that specific capacity assessments by the treating clinician using the ACE and SMMSE scores both agreed closely with expert assessments of patient capacity to consent to treatment. Our results complement and extend the results of previous studies of specific capacity assessments. The MacCAT-T, a specific capacity assessment, had excellent reliability when administered to schizophrenic and schizoaffective patients and normal volunteers by two trained researchers and an expert assessor.9 The Competency Interview Schedule had good reliability when administered to depressed patients by two trained research assistants.7 We have extended these results by showing that trained medical students and residents can reliably administer a specific capacity assessment to medical patients undergoing a broad range of treatment decisions, and these assessments agree closely with expert assessments.
We also found that general impressions by the supervising staff physicians agreed less closely with the expert assessments than ACE assessments by the research nurse. This supports previous observations that general impressions do not agree closely with expert assessments of capacity,3 except in patients who are obviously capable.14
The ACE and the SMMSE both agreed closely with the expert assessments, so clinicians will have to base their choice of capacity assessment method on other considerations. An important drawback of the ACE was the tendency to yield indeterminate results of probably incapable or probably capable. These results significantly altered the likelihood of incapacity, but there were too many misclassifications to allow a definite conclusion regarding incapacity.
A major strength of the ACE is its contribution to the consent process. During an ACE assessment, the clinician provides relevant treatment information to the patient, and asks open-ended questions to ensure that the information has been understood. The clinician also probes the patient’s reasons for accepting or refusing treatment. If the patient is ultimately found to be capable, this dialogue is a necessary and important component of a valid consent.
Although we only studied a single ACE assessment, we noticed that it often identified particular areas of uncertainty that could be further explored in subsequent assessments. For example, if the clinician was uncertain that the patient was able to appreciate the consequences of refusing treatment, a second interview focusing on those consequences might resolve any uncertainty regarding the patient’s capacity.
The SMMSE has several strengths, including its widespread clinical use and its tendency to yield fewer indeterminate results than the ACE. However, we cannot recommend using the SMMSE alone for capacity assessment because the SMMSE is not a useful component of the consent process. If the SMMSE result indicates that the patient is capable, the clinician will still need to engage the patient in a dialogue in which relevant information is exchanged and understanding is ensured. If the SMMSE result is indeterminate, the clinician will not have gained any useful clues to resolving this uncertainty. Both of these limitations of the SMMSE are strengths of the ACE approach.
Another limitation is that the SMMSE does not assess a person’s decision-making abilities, so it may be a less useful measure of capacity for patients with psychosis, depression, or frontal lobe disease.24 Our study population did not include many patients with psychosis, but one patient with an SMMSE score of 30 was incapable, because his refusal of dialysis was based on a delusion.
On the basis of these considerations, we recommend that clinicians perform both an ACE and an SMMSE as the initial step in assessing patient capacity to consent to treatment. The prospectively evaluated combination of ACE and SMMSE yielded fewer indeterminate results than the ACE alone, and avoids the limitations of using the SMMSE alone.
Our study had several strengths. First, we developed a rigorous reference standard using two independent experts and an independent adjudication panel. Our approach is more rigorous than previous studies of capacity to consent to treatment, which used single experts,7, 10, 25 single panels of experts,10 or statistically derived cutoff scores on questionnaires using hypothetical treatment decisions.3, 26 Clinicians often rely on the opinion of a single expert capacity assessment, so our study also used a more rigorous approach than would occur in usual practice. We recognize that our two experts may occasionally have both been wrong, or that a different adjudication panel may have reached different conclusions. We acknowledge that court hearings would have been the ideal reference standard, but court hearings are not available for research purposes.
A second strength of our study was the challenging spectrum of patients enrolled. We specifically sought patients for whom the issue of capacity was important and uncertain, while we specifically excluded patients for whom the issue of capacity was unimportant or obvious.
There are some potential limitations to our methods. First, ACE assessments were not completely independent of expert B, because expert B developed the ACE and conducted the ACE training sessions, so the close relation between expert B and the ACE assessments could be partially explained by bias. However, expert A was unaware of the content of the ACE and the results of ACE assessments, and we found a close agreement between expert A and the ACE assessments. Similarly, expert A was aware of SMMSE results, so the agreement between expert A and SMMSE results could be explained by bias. However, expert B was unaware of SMMSE results, and the agreement between SMMSE and expert B was the same as that between SMMSE and expert A. Therefore, we do not believe that our results can be explained by bias.
Another limitation relates to our study population. The majority of our study population was of North American or European descent. Nonparticipants were more likely to be refusing treatment and disagreeing with their physicians, suggesting that the therapeutic relationship was problematic.20 Capacity assessments may be less reliable if cultural or linguistic barriers are present, or if the therapeutic relationship is problematic.
Finally, the medical residents and students who conducted the ACE assessments received a 1-hour training session as part of the study. They were encouraged to conduct their ACE assessments to the best of their ability, and most of them were enthusiastic. The training, encouragement, and enthusiasm all made important contributions to the study results. However, we believe that any motivated clinician can learn to do an ACE assessment at the same level as the students and residents in our study. (The content of the training session is available from the corresponding author.)
In summary, our study demonstrates that specific capacity assessments by the treating clinician using the ACE and SMMSE scores agree closely with expert assessments of patient capacity to consent to treatment. Clinicians can use these practical, flexible, and evaluated measures as the initial step in the assessment of patient capacity to consent to treatment.
This project was supported by the physicians of Ontario through a grant (94-28) from Physicians’ Services Incorporated Foundation of Ontario. Dr. Naglie is partially supported by an Arthur Bond Fellowship from the Physicians’ Services Incorporated Foundation.
The authors gratefully acknowledge the support of the study participants and their families, the medical residents and students, the staff physicians, and the nurses. They also thank Linus McKenny for his technical support.