|Home | About | Journals | Submit | Contact Us | Français|
The public’s level of mental health literacy remains low, despite growing access to information regarding mental illnesses. Because few measures exist to assess the level of knowledge of mental illnesses in lay samples, the Multiple-Choice Knowledge of Mental Illnesses Test (MC-KOMIT) was developed, initially for use in a study involving police officers, some of whom received 40 hours of training focused on recognizing mental illnesses, using verbal de-escalation techniques, and making mental health referrals when appropriate. This report details the initial development of the 100 questions in the item bank and the eventual selection of the final 33 items that were retained. For these 33 items, internal consistency was demonstrated, and test-retest reliability was examined among officers who completed the test on Monday morning and again on Friday afternoon. Construct validity was examined by three hypothesis tests: the MC-KOMIT was sensitive to change related to the educational intervention, scores were significantly correlated with years of educational attainment, and officers reporting past or current treatment for a mental health problem scored higher than those without such a history. This initial report suggests that the MC-KOMIT may be useful in quantifying knowledge of mental illnesses in police officers and other diverse lay samples.
At a time when there is an unprecedented volume of information in the public domain, the level of knowledge about mental illnesses (sometimes called ‘mental health literacy’) is meager (Crisp et al., 2005). Mental health literacy encompasses an individual’s knowledge and beliefs about mental illnesses, which aid in their recognition, management, or prevention (Jorm et al., 1997a). Mental health literacy consists of several components, including: (a) the ability to recognize specific disorders or different types of psychological distress; (b) knowledge and beliefs about risk factors and causes, (c) knowledge and beliefs about self-help interventions, (d) knowledge and beliefs about professional help available, (e) attitudes that facilitate recognition and appropriate help-seeking, and (f) knowledge of how to seek mental health information (Jorm et al., 2006). It is assumed that people with greater mental health literacy should be able to recognize discrete mental disorders, name their causes and risk factors, know where to seek relevant help and information, and appreciate that they are illnesses that can be treated (Dunn et al., 2009). Discouragingly, however, research has shown that most people in the general public cannot accurately label common mental disorders (Jorm et al., 1997a; Lauber et al., 2003).
Thus, knowledge of mental illnesses is an important variable that may be applicable to many areas of investigation. Examples of the multitude of research questions that might seek to quantify the knowledge of mental illnesses construct include: “Is a primary care provider’s level of knowledge of mental illnesses related to improved quality of care for patients?”, “Is a mental health service user’s willingness to adhere to treatment associated with greater knowledge of mental illnesses?”, and “Does the public’s extent of knowledge of mental illnesses play an influential role in shaping attitudes and perceptions about people diagnosed with a mental illness, contributing to avoidance and social distance?” Additionally, researchers may seek to answer, “Does knowledge of mental illnesses among police officers partly determine the ways they interact with persons with a mental illness during their patrol and response duties?”
Despite the potential relevance of knowledge of mental illnesses as a research construct, remarkably few instruments have been developed to measure this domain of knowledge. Although very few instruments measure lay samples’ knowledge about mental illnesses broadly, several scales have been constructed specifically to assess Alzheimer’s disease knowledge in the general public; knowledge of anxiety and depression among health care providers; and knowledge of schizophrenia among family members, caregivers, and others; several of these are very briefly reviewed here. First, Blay and Peluso (2008) used a face-to-face interview administered by trained professionals to assess knowledge and perceptions of Alzheimer’s disease in the lay public. As part of the assessment, a written vignette was provided, with open-ended questions examining identification of the illness, and follow-up questions assessing causal beliefs of the identified illness using a 5-point Likert scale. Second, to measure clinical knowledge about the recognition, diagnosis, and management of anxiety disorders and depression, McCall and colleagues (2004) developed a 24-item multiple choice and true/false knowledge questionnaire, which reflected the content of a continuing medical education course. Twelve of the items, limited to depression, had previously been developed and used by Blashki (2003). Third, Dunn et al. (2009) created a scoring mechanism to allow an overall, general measure of depression-related mental health literacy to be calculated and examined in terms of validity. Fourth, the Knowledge about Schizophrenia Test (KAST), developed by Compton and colleagues (2007), is a multiple-choice test to measure knowledge about schizophrenia among family members of patients, lay community members, and police officers participating in Crisis Intervention Team (CIT) training (Compton et al., 2006).
Because so few measures exist to assess knowledge of mental illnesses among lay samples in general, the Multiple-Choice Knowledge of Mental Illnesses Test (MC-KOMIT) was constructed. This report describes the development, item analysis, and final item selection of the MC-KOMIT, which was designed for use in a study involving police officers, some of whom received a specialized, 40-hour training on mental illnesses and how best to respond to individuals with mental illnesses during crisis situations. Initial testing of reliability (internal consistency and test-retest stability) and construct validity in a sample of police officers is provided.
as further reduced from 70 to 50. Again, items were deleted in a systematic way, this time informed by item analyses of responses among these participants. Figure 1c shows the coverage matrix for the resultant 50-item bank that would then be administered to all other participants.
Data for further item analysis and psychometric assessment of the MC-KOMIT were collected from officers in non-CIT (n=125) and CIT classes (n=74) from several jurisdictions in Georgia. The non-CIT police officers were in week-long training classes on topics that included sexual assault, domestic violence, and management/supervision at the Georgia Public Safety Training Center. The CIT officers were in week-long classes that provide select officers with specialized training on recognizing mental illnesses, addictive disorders, and developmental disorders; verbally de-escalating crisis situations involving individuals with such disorders; and referring subjects to mental health services rather than arrest/incarceration, when appropriate (Bahora et al., 2008; Compton & Chien, 2008; Oliva & Compton, 2008; Compton et al., 2010). The MC-KOMIT was administered on Monday mornings and Friday afternoons for both types of class participants. For 53 participating non-CIT officers, matching data were available from both the Monday and Friday administration times; and CIT officers completed the test at the beginning and end of their specialized training. All participants provided written informed consent, and all study procedures were approved by the university’s institutional review board.
After the final item bank of 50 questions had been derived, item analyses were conducted in three steps. First, items with a corrected point-biserial correlation of <0.13 were omitted. This value can range from −1 to +1 and indicates the correlation between that item (correct/incorrect) and the total score of the test, when the total score is derived excluding that item (thus, it is “corrected”). Second, items that had more than one distractor breaking the “2% rule” (identifying problematic distractors that were selected by ≤2% of the sample, meaning that very few participants deemed it to be a plausible answer choice) were removed. Third, items with a proportion correct value of <0.20 or >0.80, indicating that the items were too difficulty or too easy, respectively, were removed. The proportion correct is simply the proportion getting the item correct, and is a measure of item difficulty, ranging from 0 to 1.
Internal consistency reliability of the final test was assessed with the Kuder-Richardson formula 20 (KR-20) reliability coefficient for dichotomous items (Streiner & Norman, 2001; Streiner, 2003). As is true for Cronbach’s alpha, values typically range from 0 to 1, and the preferred value for the KR-20 is usually considered to be >0.70. The KR-20 can be thought of as the mean of all possible split-half reliabilities. The internal consistency was computed for all officers on both their Monday morning and Friday afternoon test administrations. Additionally, scores from non-CIT officers and those undergoing CIT training were examined separately for each test administration. Test-retest reliability was assessed by calculating a Pearson correlation coefficient for scores from 50 non-CIT officers who completed the MC-KOMIT on the Monday morning and Friday afternoon of their training week, which did not pertain to mental health.
Construct validity was assessed using three hypothesis tests. First, sensitivity to change through an educational intervention (CIT training) was tested using a paired samples t-test among officers at the beginning and end of their CIT training week. Second, an association between knowledge test scores and years of educational attainment was examined. Third, construct validity was further tested with an independent samples t-test using the “known groups” approach based on the hypothesis that officers reporting a personal history of psychiatric treatment would have higher MC-KOMIT scores than those without a history of treatment. To address this, we asked participants, “Have you ever received, or are you currently receiving, treatment for a mental health problem (such as therapy, counseling, or medicine for mental or emotional problems)?” as part of the demographics questionnaire. The SPSS 16.0 statistical software package was used for all descriptive statistics, reliability assessments, and validity tests.
Basic demographic characteristics of the study sample are shown in Table 1. The mean age of participants was 38.3±8.4 years, and the average years of education completed was 13.9±1.7 years. As expected, the majority of participants were non-Hispanic, White/Caucasian males. Two thirds (131, 66.5%) were married or living with a partner, and over half (113, 57.4%) reported a total annual income for everyone in their household last year of >$60,000.
Analysis of the 50 items completed by the overall sample (n=199) was conducted in three steps as outlined above. Ten items were removed due to a low point-biserial correlation and two items were deleted because more than one distractor broke the 2% rule. A number of other items were de-selected because they met more than one of the criteria for removal; two items were removed because they had a point-biserial correlation of <0.13 and an unacceptable proportion correct value; one was removed because it broke the 2% rule and had an unacceptable proportion correct value; another item was removed because it broke the 2% rule and had a point-biserial correlation of <0.13; and another item was de-selected because it met all three criteria for removal. Thus, the final version of the MC-KOMIT included 33 items (shown in the Appendix) with good item performance based on these criteria. Figure 1d shows the coverage matrix for the final 33 items, which reveals a good range of content areas, content domains, and taxonomy types.
Mean 33-item MC-KOMIT scores are shown in Table 2. These scores were derived by simple summation of the 33 items, where each correct item counts as one point. When respondents skipped one or two items (e.g., 15 of 199 officers (7.5%) in the Monday morning test administrations), those items were considered incorrect in the calculation of the score. In cases in which more than two items were skipped (e.g., 10 of 199 officers (5.0%) in the Monday morning test administrations), a score was not computed as this was often due to the officer not completing the test or skipping an entire page of the test.
Across all four administrations, the test appeared to perform well, with no obvious floor or ceiling effects. KR-20 statistics were 0.72 and 0.73 for all participants completing the test on Mondays and Fridays, respectively. When non-CIT officers and officers in CIT training were examined separately, these values were 0.74 and 0.73, respectively, for the Monday morning administration and 0.68 and 0.75, respectively, for the Friday afternoon administration.
Test-retest reliability was examined among 53 non-CIT officers who completed the MC-KOMIT on the Monday morning and Friday afternoon of their non-mental health training week. The Pearson correlation coefficient was 0.79 (P<0.001). The scatter plot illustrating this correlation is shown in Figure 2.
Construct validity was first tested by examining mean MC-KOMIT scores of police officers at the beginning and the end of their CIT training week, to demonstrate sensitivity to change. Among 68 participants with available data at both time points, the mean MC-KOMIT score among officers at the beginning of the training week was 18.6±5.2, compared to 20.8±5.2 at the end of the training week (paired samples t=4.24, df=67, P<0.001), representing a mean difference of 2.2. Thus, the test is sensitive to change after an intervention designed to convey knowledge of mental illnesses.
Second, Monday morning MC-KOMIT scores were significantly correlated with years of educational attainment (ρ=0.35, P<0.001). This is depicted graphically in Figure 3, which shows mean MC-KOMIT scores by 12–16 years of educational attainment. A third test of construct validity was conducted using the known-groups approach in which officers who reported having received past or current treatment for a mental health problem were compared to those not endorsing this item. The 15 officers with a history of treatment had a mean score of 22.5±3.4, compared to 18.3±5.0 among the 172 officers without a history of treatment (t=3.19, df=185, P=0.002).
Knowledge of mental illnesses can be measured efficiently with a brief, self-administered, multiple-choice test. This study provides initial support for the psychometric properties of the MC-KOMIT, which appears to have utility among law enforcement officers. Such a knowledge test may also be applicable in studies measuring knowledge in other lay samples in which a high level of variability in knowledge is expected. Starting with an initial item bank of 100 questions, the item bank was narrowed to 70 items and then 50 items. In-depth item analysis of these 50 items resulted in the derivation of the final 33-item instrument. Scores from the 33-item MC-KOMIT were demonstrated to have acceptable internal consistency reliability, test-retest stability, and construct validity, the latter being assessed through three hypothesis tests.
The development and initial testing of the MC-KOMIT demonstrates that the knowledge of mental illnesses construct can be efficiently measured with a brief, self-administered, multiple-choice test in settings outside of the formal mental health system. The need for the public to have greater mental health literacy is highlighted by the remarkably high lifetime prevalence of mental disorders (up to 50%, according to Kessler et al., 1994) and the great number of individuals who are indirectly affected. The field of psychiatric research may make use of such a measure in addressing diverse research questions, beyond the initial use of the MC-KOMIT in a study of police officers engaged in specialized mental health training. Several examples can be given to illustrate this potential as a variable of interest. First, knowledge tests may be useful in studies involving caregivers, who oftentimes do not possess the knowledge needed to adequately care for a loved one with a mental illness. Studies show that families routinely request information on basic facts about mental illnesses and their treatments, behavior management skills, and the mental health system in order to better cope with their relatives’ illnesses (Mueser et al., 1992; Ascher-Svanum et al., 1997; Gasque-Carter and Curlee, 1999). Second, among society at large, having basic knowledge of mental illnesses, before any personal or family encounter with a mental illness, would potentially be beneficial. One consequence of poor mental health literacy is that the bulk of the intervening around mental disorders is largely confined to professionals. However, the prevalence of mental disorders is so high that the mental health workforce cannot adequately serve everyone affected, and therefore mental health professionals tend to focus on those with more serious and chronic problems (Jorm, 2000). Third, there is a significant problem of non-detection of prevalent mental illnesses in primary care settings; many individuals with psychiatric disorders are either undiagnosed or misdiagnosed and therefore inadequately treated (Freeman, 1992).
Measuring the knowledge of mental illnesses variable may also be crucial in research on stigma and related attitudes and perceptions. It is thought that an individual’s knowledge of mental illnesses may play a major role in shaping attitudes and perceptions about people diagnosed with such disorders, and that a lack of knowledge could contribute to avoidance and social distance stigma (Esterberg et al., 2008), as well as impede early recognition and appropriate help-seeking behaviors (Jorm, 2000; Jorm et al., 1997a; Jorm et al., 1997b). Somewhat similar to the knowledge of mental illnesses construct, perceptions of illness are highly relevant to psychological and social research. In being faced with a situation such as the experience of an unusual symptom, or the provision of a diagnosis from a doctor, individuals will construct their own representation which, in turn, will determine their behavior and other responses, including help-seeking and medicine-taking (Weinman & Petrie, 1997).
Several methodological limitations of this study should be recognized. First, although the MC-KOMIT is likely to be useful in research involving diverse lay samples (e.g., the general public), the present item analysis and psychometric testing was conducted specifically with police officers by virtue of the investigative team’s ongoing research. When used with other samples, reliability and validity should be re-assessed. Second, in assessing construct validity, one approach was to demonstrate sensitivity to change after an educational intervention; yet, the mean difference in scores before and after CIT training was only 2.2 points, which is arguably a relatively small change. This is likely driven by the fact that the week-long CIT training program for police officers focuses not only on conveying knowledge in a didactic format, but more importantly on shifting attitudes through interactions with individuals with mental illnesses and their family members and also on practical de-escalation training that is not tied to specific knowledge of mental illnesses. A third limitation, pertaining to the instrument rather than the study, is that the MC-KOMIT focuses on knowledge of mental illnesses in a broad sense, but in some instance it may be much more appropriate to assess knowledge in specific areas (e.g., pertaining to specific mental illnesses amongst caregivers or primary care practitioners). Knowledge gains in the specific areas of a curriculum’s objectives—rather than using an all-purpose knowledge test—would also be more appropriate in some instances. In terms of stigma research, although knowledge of mental illnesses in general may be beneficial, it is also conceivable that for some research endeavors the development of instruments focusing on one particular disorder would be more advantageous.
Clearly, there are diverse research questions that may pertain to the construct of knowledge of mental illnesses. Yet, there exists a prominent dearth of established, psychometrically tested instruments to measure this construct. This initial description and analysis of the MC-KOMIT suggests that this and similar carefully constructed multiple-choice knowledge tests may be very useful. Further research is needed in diverse samples.
This study was supported by a grant from the National Institute of Mental Health (R01 MH082813) to the first author. The authors greatly appreciate assistance provided by Ms. Letheshia Husbands, Ms. Shaily Krishan, Ms. Harriett Laurence, Ms. Tarianna Stewart, and Ms. Pat Strode in the recruitment of study participants and the collection of data.
Bold font indicates correct responses, and the original numbering system is retained.
E5a. Some people with obsessive-compulsive disorder may benefit from which of the following?
J5b. A 76-year-old man has mild memory problems, difficulty maintaining his house, and some confusion about what month it is. Which of the following resources may be most beneficial to him?
A1b. A 19-year-old begins to hear voices and act paranoid several months after graduating from high school. Which of the following is the most likely cause of these problems?
J5a. Which of the following community services is especially important for people with dementia and their families?
D5b. A man is evaluated by a psychiatrist, who tells him that psychotherapy is much more important for his problems than any medications. Which of the following disorders is he most likely being seen for?
B3a. People who have had an episode of major depression are at most risk for which of the following?
D5a. Psychotherapy may be the most beneficial resource for which of the following conditions?
F2a. Which of the following is a feature of all developmental disabilities?
H4a. Which of the following is most true of psychiatric medicines for children?
D1b. A woman sees a therapist weekly to work on her maladaptive patterns of behavior which have repeatedly interfered with relationships and work. When stressed out, she becomes very impulsive, feels empty, thinks of suicide, and makes multiple cuts on her forearms. Which of the following most likely contributed to her problems?
E2a. Which of the following is a common symptom of obsessive-compulsive disorder?
F3b. A woman brings her 16-year-old son to a psychologist for an evaluation of failing grades. In order for the psychologist to ensure that this young man does not have mental retardation, which of the following questions is he likely to ask?
I1b. A 35-year-old woman has been court-ordered to see a psychologist for problems stemming from her alcohol and cocaine use. Which of the following is the psychologist likely to be working to change?
D2b. A woman who has borderline personality disorder frequently becomes suicidal due to an ongoing fear. Which of the following is she most likely subconsciously afraid of?
E3a. People who have posttraumatic stress disorder are at highest risk for also having which of the following?
F4b. A 13-year-old girl with severe mental retardation has been followed by a child psychiatrist for ten years. She has never taken medications, but the psychiatrist has now decided that a medication may be beneficial. Which of the following is the most likely reason the medication is being prescribed now?
I2a. Delirium and seizures may occur during withdrawal from heavy, long-term use of which of the following substances?
A4b. Which of the following is a common reason why a woman with schizophrenia might not regularly take her medicine?
B5a. Most people in treatment for depression are treated by which of the following?
F4a. Which of the following best describes the treatment approach for developmental disabilities?
G5b. A mother and father meet with a mental health professional for three sessions over the course of a month after their 25-year-old son committed suicide. Which of the following is the most likely reason for their meetings?
H1a. Which of the following is likely a cause of attention-deficit/hyperactivity disorder in children?
J3a. Which of the following is the most common long-term course of dementia?
B1b. A woman is diagnosed with depression by her doctor during a hospital stay for asthma. Her doctor was most likely prompted to screen for depression because the woman has which of the following?
C2a. Which of the following is a common sign of mania?
F5b. A 25-year-old woman receives a number of services through the local mental health system and social agencies, including someone to visit her at home to help with hygiene and cooking, a job coach, a psychiatrist, and a neurologist. Which of the following conditions does she most likely have?
H2b. An 8-year-old boy is diagnosed as having depression by a psychiatrist. The psychiatrist tells the boy’s parents that sometimes depression in children appears different from depression in adults. Which of the following is an example of what the psychiatrist is describing?
I3a. Which of the following best describes an addiction?
B1a. Which of the following is most likely to increase one’s risk for depression?
C2b. A man at an airport is talking and laughing very loudly, passing out twenty-dollar bills, and telling people he plans to fly around the world to attend sixteen diplomatic meetings. He likely has which of the following?
D3a. People who have personality disorders usually have these problems during which of the following life stages?
E4b. A man has been having episodes of chest pain, hyperventilation, and extreme anxiety during the past few months. He is afraid that he will die of a heart attack during one of these episodes. He has seen several doctors, who assure him that his heart is healthy. He now avoids driving because it seems to trigger these attacks. Which of the following treatments would be most helpful?
H2a. Which of the following is a common sign of oppositional defiant disorder in childhood?
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.