|Home | About | Journals | Submit | Contact Us | Français|
People with mental illness constitute a substantial proportion of smokers and an important population for smoking cessation research. Obtaining informed consent in this population is a critical ethical endeavor. We examined performance on a three-item instrument (3Q) designed to screen for understanding of several key elements of research: study purpose, risks, and benefits. Patients were clinically diagnosed with primary unipolar depression (n = 40), a primary psychotic disorder (n = 32), both mood and psychotic disorders (n = 17), and primary bipolar disorder (n = 14). Among an ethnically diverse sample of 124 psychiatric inpatients approached for a smoking cessation trial, 107 (86%) performed adequately on the 3Q (i.e., obtained a score of at least 3 out of a possible 6). Patients were better able to identify the study risks and benefits than to describe the study purpose. The 3Q appears to be a useful tool for researchers working with vulnerable psychiatric patients.
Inpatient psychiatric settings represent important venues for research examining the effects of psychotropic medications or health risk behaviors such as tobacco use and other substance use. Concerns have been raised, however, regarding patients’ understanding of key information relevant to informed consent for research, as well as the suitability of methods used in the informed consent process to assess understanding (Dunn & Jeste, 2001; Roberts, 1998). Given the potential vulnerability of hospitalized patients with serious mental illness (SMI), who frequently have comorbid substance use disorders and are often socioeconomically marginalized, it is critical to ensure that informed consent procedures for research involving this population meet ethical standards and help support the autonomy of these individuals to the degree possible.
People with SMI, as a group, tend to achieve lower scores on various measures of decisional abilities compared to people without mental illness (Appelbaum, 2006; Jeste et al., 2009; Palmer et al., 2004). However, scores on decisional capacity measures vary among people with SMI, depend on the complexity of the decision to be made, and can be improved with targeted interventions (Appelbaum, 2006; Dunn, Candilis, & Roberts, 2006; Jeste et al., 2009; Moser et al., 2006).
The MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR), a widely used decisional capacity instrument for clinical research (Appelbaum & Grisso, 2001), though well-validated and reliable (Dunn et al., 2006), requires substantial rater training, as well as sustained attention on the part of the potential research participant. It consists of 21 items assessing the four widely accepted domains of capacity: understanding (13 items), appreciation (3 items), reasoning (4 items), and expression of a choice (1 item). Because of its length and training requirements, the MacCAT-CR is unlikely to become a widely used method for research capacity assessment, particularly when research is conducted in an inpatient psychiatric setting.
In contrast, a shorter three-item questionnaire (the 3Q) was developed to serve as a brief screening instrument—i.e., to indicate where there are gaps in understanding of three aspects of research: purpose, risks, and potential benefits (Palmer et al., 2005). As a screening tool, the 3Q can suggest the need for further assessment (e.g., using a longer instrument such as the MacCAT-CR), more education about the study, and/or the inappropriateness for study enrollment at this time. Palmer and colleagues evaluated the 3Q in comparison to the MacCAT-CR and found good sensitivity and specificity in identifying patients in need of further evaluation or intervention. Among outpatients with schizophrenia, 77% scored at or above the cutpoint of ≥ 3 on the 3Q, which was established relative to MacCAT-CR scores (Palmer et al., 2005).
To our knowledge, no prior studies have examined the use of the 3Q or other brief screening instruments during research recruitment of potentially vulnerable, acutely hospitalized, psychiatric patients. Therefore, we examined the utility of the 3Q to screen for problems with understanding of research among psychiatrically hospitalized patients. The study purpose was to examine the level and correlates of performance on the 3Q among a racially and ethnically diverse, lower-income patient population. Based on the work of Palmer et al. (2005), we predicted that a similar proportion (≥ 70%) of patients would score at or above an a priori cutpoint (set at ≥ 3 based on prior work) on the 3Q. We further predicted, based on existing literature (Dunn, Candilis, & Roberts, 2006), that older age and lower educational attainment would be associated with lower 3Q scores.
Participants were recruited from three psychiatric inpatient units (one nonacute and two acute units) at San Francisco General Hospital (SFGH). SFGH is the largest public acute care hospital in San Francisco County, serving primarily the city’s uninsured, indigent, and homeless populations. Among patients treated at SFGH in fiscal year 2009–2010, 68% were on Medicare or Medi-Cal (the California Medicaid program), 21% were uninsured, and only 11% had commercial insurance (San Francisco General Hospital Foundation, 2010).
The “parent” clinical trial, on which this consent study was piggybacked, was a randomized controlled trial with two study arms: (1) enhanced standard care, i.e., nicotine replacement therapy (NRT) offered during hospitalization, plus a pamphlet about quitting smoking and referrals to smoking cessation programs, if interested; compared with (2) enhanced standard care plus a computer-delivered intervention, treatment manual, and individual counseling based on the transtheoretical model for smoking cessation with nicotine replacement therapy (NRT) post-hospitalization, if interested (Prochaska, Hall, & Hall, 2009).
Inclusion criteria were: (1) smoking at least five cigarettes per day (CPD) prior to hospitalization, (2) age 18 years or older, and (3) fluency in written and spoken English. Acutely psychotic, hostile, or hypersomnolent (to the extent of being unable to respond to questions) patients, and those with a contraindication for NRT use, were excluded. Smokers were study eligible even if they had no intention of quitting smoking. Recruitment occurred from April 2009 until April 2010 on three locked psychiatric units at SFGH. Hospitalized smokers were identified by trained research staff through review of medical records. Clinicians of identified smokers asked patients if they would be willing to hear information about a smoking study. Patients who expressed interest were introduced to research staff, who described the study in detail. Those who remained interested proceeded to the consent process, which included the 3Q (described below). The protocol and consent forms for the parent trial, including the use of the 3Q, were approved by the Committee for Human Research at the University of California, San Francisco and by the San Francisco General Hospital administration.
Data from the 3Q were collected from 124 adult smokers during their acute psychiatric hospitalization (76% recruitment rate). Of these 124, 5 patients were not enrolled in the parent trial because they did not attain adequate scores on the 3Q, 10 withdrew during the consent process and prior to randomization, and 9 were recruited for an initial usability study.
Patients who agreed to consider the parent trial were given a consent form to follow while the interviewer read it aloud. The interviewer stopped at appropriate points to respond to questions (if any). Interviews were conducted in the dayroom on the locked units.
The 3Q consists of three open-ended questions assessing understanding of: (1) study purpose, (2) study risks, and (3) potential benefits (Palmer et al., 2005). These items cover three of the most important aspects of research that participants should understand before agreeing to participate, though it should be emphasized that these elements represent only several of the required elements that must be included in consent forms according to the Common Rule (U.S. Department of Health & Human Services, 2009). The 3Q instrument was slightly modified from the original version, as follows: We added the phrase “Can you tell me, in your own words” to the original question “What is the purpose of the study?”; we added the word “potential” to the questions, “What are the [potential] risks of the study?” and “What are the [potential] benefits of the study?” Credits were assigned for study purpose as follows: 2 points for “to evaluate a quit smoking service” or “to evaluate a computer program and counseling service for quitting smoking;” 1 point for “a study about smoking;” and 0 points for “don’t know” or a response that was unrelated to the study purpose. For study risks, credits were assigned as: 2 points for correctly identifying at least two study risks, including any of 15 side effects of using the nicotine patch, potential loss of privacy, stress or inconvenience from answering questions or quitting smoking; 1 point for identifying one of the risks; and 0 points for “don’t know” or a risk not mentioned in the consent form. For study benefits, credits were assigned as: 2 points for “quitting,” “changing smoking habits,” “contributing to research aimed at helping smokers quit,” or “to inform development of a better quit smoking service;” and 1 point for “to get help;” and 0 points for “don’t know” or other benefit not mentioned in the consent form. Because IRBs typically discourage or do not allow investigators to list compensation as a benefit of participation, we coded responses identifying financial incentives as a study benefit as “0” (consistent with Palmer et al.’s  scoring criteria). However, because participants themselves often do view incentives as a benefit of participation, we also coded these responses in a separate analysis as “2” (full credit). The 3Q took approximately 10 minutes to administer. The senior author trained the interviewers in administration and scoring of the 3Q.
The interviewer probed for clarification and provided feedback if a patient’s initial response was vague or suggested misunderstanding of the question. The responses were recorded verbatim, then independently coded by two study authors, who assigned scores of 0 (incorrect response), 1 (partially correct response), or 2 (correct response), following previously developed scoring criteria (Palmer et al., 2005). We summed the three questions to create a 3Q total score with a possible range of 0 to 6 (higher scores suggest a better understanding of the study’s protocol), and used the previously identified cutpoint of 3 that maximized sensitivity and specificity of the 3Q in relation to the MacCAT-CR.
Interrater reliability calculated for 3Q scoring was very good (i.e., the intraclass correlation coefficient was r = .93, p < .001; N = 124); therefore, for statistical analyses we used the mean of the two raters’ scores for each participant, as described by Palmer et al. (2005).
Demographic characteristics collected for each enrolled participant (n = 109) included age, gender, race-ethnicity, education, income, marital and employment status, and living situation (i.e., stable or unstable residence for the past six months). Clinical variables obtained from the medical record were DSM-IV diagnosis (American Psychiatric Association, 2000) and any past month alcohol and illicit drug use.
The study design was descriptive and correlational. Because 15 patients who completed the 3Q did not enroll in the parent trial, demographic and clinical characteristics were unavailable for these 15 patients. Therefore, analyses describing the sample and involving statistical comparisons were limited to the 109 participants for whom these data were available.
Descriptive statistics were used to examine the sample’s demographic and clinical characteristics, and to describe performance on the 3Q. A chi-square test was used to examine differential response patterns to 3Q items. We also examined the content of participants’ 3Q responses to identify the most frequently misunderstood aspects of the study. Associations of 3Q scores with demographic and clinical variables were examined using the following nonparametric statistics: Spearman’s rho correlations were used to test for associations between age and education level and 3Q scores, as studies have shown that fewer years of education and older age are associated with poorer understanding of a consent form (Dunn & Jeste, 2001). Mann–Whitney U tests examined associations of 3Q scores with diagnosed psychotic disorder and a combined variable of past month crack cocaine, methamphetamine, cannabis, heroin, and alcohol use, as these clinical variables were expected to be associated with impaired research understanding. The hypothesis-wise error rate was set at an alpha level of .05 for analyses involving comparisons. A Bonferroni correction was applied to control for the increased risk of a Type I error. A total of seven comparisons were analyzed; therefore, the corrected alpha was .007. Analyses were conducted using SPSS for Windows, release version 17.0.0 (SPSS, Inc., 2008, Chicago, IL).
Table 1 shows the demographic characteristics for 109 study-enrolled participants. Participants had a mean age of 40.0 (SD = 11.2) years, and completed an average of 12.9 (SD = 2.5) years of education. Over half were men (65%, n = 71), racial-ethnic minorities (56%, n = 61), and lacked stable housing (51%, n = 56).
Enrolled participants (n = 109) received a mean of 2.1 (SD = 0.7) clinical disorders at hospital discharge. In terms of DSM-IV diagnoses, 37% (n = 40) were diagnosed with primary unipolar depression, 13% (n = 14) with primary bipolar disorder, 29% (n = 32) with a primary psychotic disorder, and 16% (n = 17) were diagnosed with both mood and psychotic disorders. Substances used in the past month included alcohol, 67% (n = 73); cannabis, 47% (n = 51); crack cocaine, 41% (n = 45); methamphetamine, 31% (n = 34); and heroin, 8.0% (n = 8).
For the entire sample of 124 participants who completed the 3Q, the mean 3Q score was 4.2 (SD = 1.2; range = 1–6); 86% (n = 107) scored at or above the cutpoint of 3. Among participants scoring ≥ 3, 12% (n = 13) misunderstood the study’s purpose, 9% (n = 10) misunderstood the benefits, and 8% (n = 8) misunderstood the study risks. The proportion of the entire sample referring to the consent form before responding was 12%, 23%, and 15% for the study’s purpose, risks, and benefits, respectively. The proportion of individuals for whom interviewers probed for clarification due to an initial vague response was 8%, 19%, and 9% for the study’s purpose, risks, and benefits, respectively. Table 2 shows the distribution of scores for each of the 3Q items. The pattern showed better performance on identification of study risks and benefits than of study purpose, X2 (df = 16) = 39.83, p = .001. Including the financial incentive as a study benefit increased the proportion receiving full credit from 79.8% to 85.5%, but only slightly increased the mean 3Q score from 4.2 (SD = 1.2) to 4.3 (SD = 1.2).
The majority of participants (59%, n = 73) out of the entire sample (N = 124) identified “I will quit smoking” as the study’s purpose, which received a score of 1 (partially correct response). Only 5% (n = 6) recalled the study purpose as evaluating a quit smoking service, which received a score of 2 (correct response). In terms of study risks, 73 (59%) identified at least two side effects of using the nicotine patch and received a score of 2. Participants identified a mean of 1.8 (SD = 1.3) NRT side effects. The possibility of quitting smoking was seen as the primary benefit of the study, reported by 72% (n = 89) of the sample. Fifteen (12%) reported that their participation may inform a future quit smoking service. Twenty-eight (23%) participants reported that the financial incentive was a benefit for the study, a response that did not receive credit.
Participants’ (n = 109) 3Q scores were not associated with age (ρ = .02, p = .83) or education (ρ = .10, p = .30) using Spearmen’s rho correlations. Results from Mann–Whitney U tests showed no differences in the distribution of 3Q scores between patients with and without a primary psychotic disorder (Mann–Whitney U = 1152.5, n with psychosis = 32, n without psychosis = 77, p = .58) or past month substance use (Mann–Whitney U = 890.5, n using substances = 87, n not using substances = 22, p = .59).
Individuals with mental illness deserve a voice in the research enterprise. Given the vulnerable nature of the setting (i.e., locked unit, acute psychopathology), precautions are needed to ensure adequate understanding of a study’s purpose, risks, and benefits. The current findings demonstrate that, overall, hospitalized psychiatric patients with SMI recruited from a large urban public hospital are able to understand several key elements of clinical research as assessed by a brief consent screening instrument. The findings are consistent with prior research with outpatients with SMI (Moser et al., 2006; Palmer et al., 2005) and extend the findings to a racial-ethnically, socioeconomically, and psychiatrically diverse, urban sample of hospitalized adults with SMI.
Demographics and clinical variables were not correlated with performance on the 3Q, consistent with Palmer and colleagues’ finding of no significant associations between age and education with scores on the Understanding or Reasoning subscales of the MacCAT-CR (Palmer et al., 2005). In concordance with our findings, Palmer and colleagues did not find an association of age and education with 3Q scores among outpatient veterans with schizophrenia. Our sample tended to be younger, restricting the ability to detect agerelated differences in performance. Moreover, previous literature that found an association between performance on capacity measures and education used more comprehensive capacity assessment tools and included primarily patients with schizophrenia (see Dunn, Candilis, & Roberts  for a review of this literature). Given that, in the present study, the consent form was read aloud to potential participants, and that the items were meant to be easily understood by individuals of any educational background, this finding highlights the broad utility of this brief measure for use with samples from diverse backgrounds.
The sample’s mean 3Q score of 4.2 is slightly higher than the mean 3Q score of 3.9 found among an outpatient sample of veterans with schizophrenia (Palmer et al., 2005), a diagnosis that is often associated with specific types of cognitive impairment. In our study, eight patients received a diagnosis of schizophrenia, and this subgroup scored a mean (SD) 3Q score of 3.9 (1.8), identical to Palmer’s.
Another concern expressed about potentially vulnerable populations has been their ability to exercise voluntary choice when consenting to research (Appelbaum, Lidz, & Klitzman, 2009a, 2009b). In particular, some worry that financial incentives for research will influence patients with low socioeconomic status to accept risks that they otherwise would not (Grady, 2009). For the parent smoking cessation trial described here, participants were reimbursed for study-related time and travel during follow-up assessments. These financial incentives were designed to be noncoercive (i.e., $10 for the initial assessment, with a maximum of $110 total, distributed in $20 increments at five follow-up visits, over 12 months). Although the University’s IRB guidance specifies that payment be listed in a separate section of the consent form from “benefits,” a substantial subgroup (28 participants; 23%) nevertheless identified it as such. This finding is consistent with Dunn and Gordon’s (2005) argument that payment is often viewed as a benefit among research participants, regardless of how or where payment is described in consent forms. Study participants who identified the financial incentive as a study benefit also identified other benefits; thus, regardless of how we scored responses listing payment as a benefit (Table 2), mean 3Q total score was essentially unchanged. Furthermore, it is unknown how individuals weighed the different perceived benefits when deciding whether to participate. Notably, studies examining voluntariness in vulnerable populations have not found strong evidence that financial incentives serve as an undue influence for research participation (Appelbaum et al., 2009b; Dunn et al., 2009). However, further research is necessary to examine whether incentives such as larger payments or other, nonmonetary incentives (e.g., treatment access) may influence research participation in unexpected ways.
Few participants reported the study’s purpose as informing smoking cessation services for smokers with mental health problems, despite the interviewer explicitly reading this statement. Instead, many participants (59%) reported that the study was designed to help them stop smoking. The responses suggest at least some degree of therapeutic misconception (Lidz et al., 2004)—i.e., believing that the purpose of the study was more personally tailored (i.e., to help the individual patient to stop smoking) than it actually was.
A limitation of the current study was recruitment of participants from a single hospital. The sample may not be representative of inpatients in other geographic regions or hospital settings. On the other hand, the setting was an urban public hospital serving a highly marginalized patient population with high rates of substance abuse and low socioeconomic status. Therefore, this sample represents an understudied population in the research ethics literature. Yet, this seriously psychiatrically ill population, with numerous comorbid conditions, represents the kinds of potentially vulnerable patients that elicit greater concerns about understanding of content in a consent form. Structured clinical diagnostic interviews were not administered and data on cognitive functioning were not available; therefore, we could not ascertain the extent to which cognitive abilities such as working memory related to 3Q scores. Prior research has shown that cognitive abilities assessed using neuropsychological tests correlate with scores from decisional capacity instruments in the expected direction (Carpenter et al., 2000; Cohen et al., 2004; Moser et al., 2002). We did not have access to data regarding medication use specific to the timing of the assessment, which may have affected attention.
Another clear limitation of this study was the lack of a full capacity assessment, as the 3Q was not developed as a comprehensive decisional capacity instrument. Thus, we cannot say with confidence that all participants would be considered capable if they had been administered the full MacCAT-CR. However, there is no consensus on what score, on which subscales or combination of subscales of the MacCAT-CR, should be construed as adequate decisional capacity (Dunn, Nowrangi et al., 2006). This important normative issue of setting cutpoints continues to elicit discussion and debate in the literature. Although we would not argue that a score of 3 out of 6 represents adequate understanding for every type of study, it does provide reassurance that these individuals understood at least two of three key elements of the study being considered. Moreover, the findings were consistent with other studies that used brief screening tools in patients with schizophrenia, in which the majority were able to perform adequately (Moser et al., 2002; Palmer et al., 2005). Nevertheless, a strength of the present study is the examination of research understanding in the context of an actual, rather than hypothetical, clinical trial. While the 3Q does not provide a comprehensive measure of decisional capacity, nonetheless, it has the advantage of being brief, provides clear information about which study components need clarification, and is feasible to administer in an inpatient psychiatric setting.
This study highlights that most participants with SMI—including those with comorbid substance use disorders—performed at or above an a priori cutpoint that was based on prior research (Palmer et al., 2005) which validated this tool against the more comprehensive capacity assessment tool, the MacCAT-CR. The 3Q, which was developed to function as a brief screening tool, should not be considered a comprehensive or formal “capacity assessment” instrument. However, as shown here, it can help investigators efficiently screen for potential problems in the understanding of key elements of research. If the tool had not been used for the parent smoking cessation trial (and indeed, no capacity assessment was required by the IRB), participants who simply endorsed that they understood the trial (by self-report), asked no questions, and signed the consent form would have been included. This is in fact standard procedure in most research studies.
In contrast, the use of a brief instrument such as the 3Q can help investigators achieve a more stringent standard of actually assessing understanding, and providing corrective feedback (and reassessing understanding) when a participant misses a question. Moreover, our scoring criteria were actually rather strict, requiring a fairly sophisticated understanding on each domain to obtain full credit. This was intentional, so that we would be able to distinguish among different levels of understanding of purpose, risks, and benefits.
These findings support the benefits of using the 3Q as a screening tool for clinical research, as it is brief, and provides the research team with an important opportunity to assess and enhance understanding of the research protocol. Our findings suggest several best practices when recruiting people with SMI. First, brief screening tools such as the 3Q and others (Jeste et al., 2007) can be readily incorporated into the consent process for most types of research. Given the high levels of psychiatric comorbidity and socioeconomic adversity, it will be important to develop applied methods for ensuring that patients fully understand the importance of research and its purpose, as well as the potential risks and benefits involved in participating. Second, researchers should be prepared to clarify risks, benefits, and purpose in a manner that can be understood by the individual. Third, researchers should exclude individuals who do not demonstrate sufficient understanding of the key elements necessary for informed consent.
Important gaps remain regarding best practices for informed consent for research participation among people with SMI. The influence of challenging, frequently intersecting life circumstances (e.g., poverty, homelessness, repeated institutionalization—including incarceration) and comorbid conditions (e.g., substance abuse, medical illnesses) on the understanding of research participation warrants in-depth investigation. Further, given that understanding of a research protocol may fluctuate over time, studies are needed that examine the relationship between this construct, symptom severity, and medication side effects over multiple time-points. Finally, research should examine the utility of booster sessions that include reminders of the study purpose, benefits, and risks for people with SMI enrolled in a study for an extended time.
We recommend that researchers become familiar with reliable and valid instruments, as part of their research training, that can serve as both brief screening instruments (such as the 3Q), as well as more comprehensive capacity assessment tools (such as the MacCAT-CR). IRBs should encourage researchers to use reliable and valid instruments that assess and enhance understanding of the research protocol when the study involves mental health populations and other vulnerable groups. Students and research staff should be taught that assessing research understanding and providing feedback is more ethically responsible than reading a consent form to a potentially vulnerable participant or, more commonly, relying on the individual to read it themselves. Papers by the following authors are recommended for readers interested in ethical issues related to research understanding and the assessment of decisional capacity in people with mental illness (Albala, Doyle, & Appelbaum, 2010; Appelbaum, 2006, 2007, 2010; Dunn, 2006; Dunn, Candilis, & Roberts, 2006; Dunn et al., 2009; Dunn, Nowrangi et al., 2006; Jeste et al., 2007; Jeste et al., 2009; Palmer et al., 2008; Roberts, 1998; Roberts, Warner, & Brody, 2000).
This research was supported primarily by grants K23DA018691, P50DA09253, and T32DA007250 from the National Institute on Drug Abuse. Dr. Dunn is supported, in part, by the National Institute on Aging (NIH R01AG027986) and by the Greenwall Foundation. The authors gratefully acknowledge Barton W. Palmer, Ph.D., for providing valuable input on an initial draft of this manuscript.
Norval Hickman is a Postdoctoral Fellow with the Treatment Research Center in the Department of Psychiatry at the University of California, San Francisco (UCSF). His interests are in the ethical conduct of culturally informed tobacco cessation treatments for underserved groups of smokers. Dr. Hickman contributed to the study design and implementation, conducted and interpreted data analyses, and wrote drafts of the manuscript.
Judith J. Prochaska is Associate Professor in the UCSF Department of Psychiatry and a member of the Tobacco Control Program in the UCSF Helen Diller Family Comprehensive Cancer Center. Her research centers on developing effective treatments for tobacco dependence and other leading health risk factors with a special focus on populations with co-occurring psychiatric or addictive disorders. Dr. Prochaska’s expertise includes methodological and ethical considerations when conducting research in inpatient psychiatry settings. She contributed to study design and implementation, analyses, and writing and revisions of the manuscript.
Laura B. Dunn is Associate Professor of Psychiatry at the University of California, San Francisco, where she is the Director of Psycho-Oncology in the Department of Psychiatry and UCSF Helen Diller Family Comprehensive Cancer Center. Dr. Dunn’s research falls into two primary areas: (1) empirical ethics, including informed consent, decision-making capacity, and research participants’ perspectives; and (2) psycho-oncology. She was involved in the conceptualization of the study and interpretation of the data.