|Home | About | Journals | Submit | Contact Us | Français|
This article describes the development of an instrument to screen male and female offenders for co-occurring substance use and mental disorders. This phase developed and pilot tested (N = 100) the Criminal Justice Drug Abuse Treatment Studies (CJDATS) Co-occurring Disorders Screening Instrument for Mental Disorders (CODSI-MD), a 6-item instrument derived from three standard mental health screeners. The overall accuracy of the CODSI-MD (81%) compared favorably with the three standard instruments. A second 3-item instrument, developed to screen for severe mental disorders (the CODSI-SMD), had an overall accuracy of 82%. The results of this pilot study must be viewed cautiously, pending validation of the findings with a larger sample.
Offenders who have co-occurring substance use and mental disorders (hereafter referred to as co-occurring disorders, or COD) constitute a population of significant concern. In practical terms, the special needs of offenders with COD (e.g., segregated programming, multiagency discharge planning) place exceptional demands on the criminal justice system. Although rapidly growing throughout the criminal justice system, this population remains largely understudied, despite its attendant clinical and administrative challenges. Forensic programs and services must be established for these offenders.
In community-based settings, substance abuse programs have typically reported that 50% to 75% of their clients had some type of COD (although not usually a severe mental disorder), whereas studies in mental health settings reported that between 20% and 50% of their clients had a co-occurring substance use disorder (for a summary of studies, see Sacks, Sacks, De Leon, Bernhardt, & Staines, 1997; for more recent studies, see Compton, Cottler, Phelps, Abdallah, & Spitznagel, 2000; Havassy, Alvidrez, & Owen, 2004).
Studies have found the prevalence of mental disorders to be higher in the prison system than in the general population (Fazel & Danesh, 2002; O’Brien, Mortimer, Singleton, & Meltzer, 2003). For example, the rate of lifetime prevalence for schizophrenia and bipolar disorder in the general population was 0.8% and 1.5%, respectively, whereas the corresponding rates in state prison populations were between 2.3% and 3.9% for schizophrenia and between 2.1% and 4.3% for bipolar disorder (Veysey & Bichler-Robinson, 1999). In a large meta-analysis of 62 studies, Fazel and Danesh (2002) found that prisoners were 2 to 4 times as likely to suffer from psychotic illness or major depression compared to the general population. The U.S. Department of Justice reported that 16% of state prison inmates, jail inmates, and probationers reported either a mental condition or an overnight stay in a mental hospital during their lifetimes (Ditton, 1999). The number of correctional clients with mental disorders also appears to be increasing. As an example, reports from the Colorado Department of Corrections chronicle a steadily rising proportion of inmates with mental illness, from 4% in 1991, to 14% in 2001 (Kleinsasser & Michaud, 2002), to 19% more recently (Stommel, personal communication, 2005); an estimated three fourths of these have a co-occurring substance use disorder.
Among women in jail, nearly three fourths of those with severe mental disorders (schizophrenia or major affective disorder) met the criteria for a substance use disorder, and 15% of women with a substance use disorder had a co-occurring severe mental disorder. Overall, 8% of the women had co-occurring substance use and severe mental disorders (Abram, Teplin, & McClelland, 2003). It would be expected, therefore, that the number of women meeting the criteria for COD based on a more inclusive full range of mental disorders would be considerably higher. These data suggest that COD is common in criminal justice settings and warrants special attention.
The greater risk factors and poorer treatment outcomes for inmates with COD warrant increased attention in the criminal justice system. Studies of nonoffender populations have found poorer outcomes for those with COD, particularly those with severe mental disorders, including higher rates of HIV infection, relapse to substance use, rehospitalization, depression, and suicide risk, as compared to those with a single disorder (Drake et al., 1998; Office of the Surgeon General, 1999). Similar findings occur in criminal justice studies of offenders with COD. In a study of residential substance abuse treatment for offenders as an alternative to prison, those who had a history of mental disorder were twice as likely to drop out of treatment (Lang & Belenko, 2000). In a study of probationers, a prior history of mental disorder was associated with shorter lengths of stay in substance abuse treatment, as were higher levels of depression and anxiety (Hiller, Knight, & Simpson, 1999). In jails, offenders with mental disorders were more likely to fail to complete treatment by ratios of nearly 2 to 1 (National Institute of Justice, 1997) and 3 to 1 (Brady, Krebs, & Laird, 2004) compared to inmates who did not have histories of mental disorder.
Prisoners with mental disorders frequently face stigmatization, isolation by other prisoners, and high rates of victimization (Correctional Association of New York, 2004). They adapt less well to the complexity of prison life (Morgan, Edwards, & Faulkner, 1993), are more likely to have to serve out their full sentences (AISHealth.com, 2001), and are more likely to be written up in an incident report during the first 3 months of incarceration (DiCataldo, Greer, & Profit, 1995).
Also of special concern to the criminal justice system, studies have demonstrated that the combination of any substance use and mental disorder puts individuals at greater risk for committing violent acts (Monahan et al., 2000, 2001, 2005; Steadman et al., 1998) and that the risk for violent behavior is higher for those with higher scores on a measure of substance abuse (Melnick, Sacks, & Banks, 2006b) and for those with both alcohol and drug use problems (Melnick, Sacks, & Banks, 2006a). The relationship between COD and violence is further documented in a 10-year follow-up study of inmates diagnosed with schizophrenia and treated in a medium security prison (Baxter, Rabe-Hesketh, & Parrott, 1999). Of the 63 inmates, 30% were reconvicted for at least one violent offense. The scope of the problem is even more evident considering that episodes of violent behavior outnumbered violent reconvictions by a ratio of 4 to 1. The role of substance abuse was evident in that 71% of the sample had a history of drug abuse and 29% had a history of alcohol abuse and that both poly-drug use and alcohol problems were significant predictors of violent behavior, with odds ratios of 3.18 and 3.0, respectively.
At the same time, recent research shows that offenders with COD can benefit from treatment and that treatment is cost-effective. In a series of studies, Sacks and colleagues (Sacks, Sacks, McKendrick, Banks, & Stommel, 2004; Sullivan, McKendrick, Sacks, & Banks, 2006) demonstrated significantly greater reduction in crime and substance abuse for offenders with COD in a modified therapeutic community program as compared to those in standard mental health treatment. In addition, a 3-year study of persons with COD showed that poor treatment engagement was associated with multiple arrests but that effective treatment of substance use reduced arrests and incarcerations so that engaging COD offenders in treatment can be cost-effective for the criminal justice system (Clark, Ricketts, & McHugo, 1999).
The high prevalence of COD, the negative outcomes of untreated COD, and the recent studies indicating positive outcomes when COD is effectively treated point to the importance of identifying those with COD so that appropriate treatment can follow. This leads to the conclusion that instruments to screen offenders for COD must be developed and tested.
Screening is a formal process of testing to determine whether or not a person potentially has a disorder that merits further assessment (Center for Substance Abuse Treatment [CSAT], 2005). Screening is the first step in a clinical planning process that includes assessment, diagnosis, treatment planning, and referral to, or delivery of, services. Screening is important to program and policy planners in identifying the extent of need in a larger population. To date, both clinical and program planning for offenders have been limited by the lack of an adequate, standardized, and uniformly accepted COD screening instrument that would provide valid and consistent results across studies. Ideally, a screening instrument is brief, taking from 5 to no more than 20 minutes to complete, and can be administered by direct care staff without any additional training.
Although numerous screening instruments have been developed for substance use and mental disorders, at present no single instrument is suitable to screen for the combined disorders within offender populations. Of the numerous discrete instruments that have been developed to screen for substance use (both alcohol and drug use) separately and mental disorders, some have been tested in criminal justice settings. The overall goal of this study was to identify brief screening instruments for both substance use and mental disorders that could then be combined into a screening device for COD. A comprehensive review of screening instruments is beyond the scope of this article, but condensed descriptions of selected instruments are provided below.
Substance use screening instruments have a long history of development in general community settings (Inciardi & Lockwood, 1994). Instruments to screen for alcohol and drug disorders in prison settings have been evaluated (Peters, Greenbaum, Steinberg, & Carter, 2000). Seven instruments, selected on the basis of common usage and good psychometric support, were tested and evaluated for their psychometric properties in prison settings. These included the Alcohol Dependence Scale (ADS; Skinner & Horn, 1984), the Addiction Severity Index (ASI; McLellan et al., 1985, 1992), the Drug Abuse Screening Instrument (DAST-20; Gavin, Ross, & Skinner, 1989; Staley & Guebaly, 1990), the Michigan Alcoholism Screening Test short version (SMAST; Gibbs, 1983; Ross, Gavin, & Skinner, 1990), the Substance Abuse Subtle Screening Inventory-2 (SADDI-2; Miller, 1985; Svanum & McGrew, 1995), the Simple Screening Instrument (SSI; CSAT, 1994), and the Texas Christian University Drug Screen (TCUDS; Broome, Knight, Joe, & Simpson, 1996; Simpson, 1995; Simpson, Joe, Rowan-Szal, & Greener, 1997). Several instruments, including a combination of the ASI Drug Screen and ADS, the TCUDS, and the SSI, were found to have equally met the stated criteria, namely, overall accuracy, brevity, in the public domain, and capable of administration by nonprofessional (mental health) staff.
Numerous screening instruments have been developed to screen for mental disorders in the general population. Treatment Improvement Protocol 42, Substance Abuse Treatment For Persons With Co-Occurring Disorders (CSAT, 2005), provides a list of some recommended screening instruments for mental disorders. Among the most commonly used are the Global Appraisal of Individual Needs (Dennis, 1998), the Mini-International Neuropsychiatric Interview (MINI; Sheehan et al., 1998), and the Mental Health Screening Form (MHSF; Carroll & McGinley, 2001). A more recent addition, the K6 (Kessler et al., 2003), is a very brief screening tool designed to monitor population prevalence and trends that identified 96% of those who did not have a severe mental disorder (e.g., no mental disorder or mild mental disorder) but only 36% of those whose mental disorder was severe.
Two reports released in 1997 (Peters & Bartoi, 1997; Peters & Hills, 1997) reviewed screening and assessment instruments for mental disorders. These reviews included the recommendation of the Referral Decision Scale (Teplin & Schwartz, 1989), and the Brief Symptom Inventory (BSI; Derogatis, 1993). Other instruments in use in criminal justice settings include the Psychiatric Diagnostic Screening Questionnaire (Zimmerman & Mattia, 2001). Several new screening instruments have been released in the recent past, such as the Brief Jail Mental Health Screen, which the authors recommend for men only (Steadman, Scott, Osher, Agnese, & Robbins, 2005), and the Jail Screening Assessment Tool, a semistructured interview that has been used to assess women prisoners in jails (Nicholls, Lee, Corrado, & Ogloff, 2004).
The purpose of the study reported in this article was to identify an instrument, and cutoff scores, that best meets the needs of criminal justice systems to screen for mental disorders in prison substance abuse treatment settings.
The overall design for this study was to select mental health screening instruments for use in prison substance abuse treatment programs and to conduct a pilot study to determine the suitability and precision of the instruments in screening for mental disorders.
A literature search was conducted to identify the best available mental health and substance abuse screening instruments consistent with the initial criteria: good psychometric properties, in the public domain, requiring no more than 20 minutes to complete, and suitable for administration by the participant or by correctional or substance treatment staff who do not have specific mental health training. The plan was to identify instruments with high (.80 or higher) test-retest reliability and with known validity for identifying substance use and mental disorders. This review included prior reviews (e.g., American Psychiatric Association, 2000; Dawe, Loxton, Hides, Kavanagh, & Mattick, 2002; Peters & Bartoi, 1997) and Internet search engines, such as those of the National Institutes of Health (Medscape.com) and of the American Psychological Association (APA.org). In all, more than 150 instruments were considered for inclusion in the study.
Two sources of input were sought in addition to the literature search: (a) a stakeholders panel consisting of 12 criminal justice professionals (e.g., directors of state corrections departments and wardens) and substance abuse treatment program directors and (b) an expert panel of 10 experts and researchers familiar with the screening and treatment of substance abuse, mental disorders, and COD in criminal justice settings. These panels provided additional information about instruments currently in use in criminal justice settings, the type of personnel and maximum training time available, the maximum time available for administration, the ease of use and scoring, acceptability of questions to respondents, type of profile generated, and utility estimates (e.g., the trade-offs between the length of the instrument and the desired level of accuracy).
Considering the advice of the expert panel, a previous review (Peters et al., 2000), and current usage in Criminal Justice Drug Abuse Treatment Studies (CJDATS) studies, the TCUDS (Broome et al., 1996; Simpson, 1995; Simpson et al., 1997) was accepted as the drug component of the overall COD screening instrument for the pilot study. This decision allowed the field test to concentrate on the evaluation of the mental health screeners for use in prison substance abuse treatment programs. Three instruments, which met the initial evaluation criteria and which received the panels’ endorsements, were chosen for the field test study. The instruments selected to form the Co-Occurring Disorders Screening Instrument (CODSI) Screening Battery were the MINI (Sheehan et al., 1998), the MHSF (Carroll & McGinley, 2001), and the Global Appraisal of Individual Needs Short Screener version 1.0 (GSS; Dennis, Chan, & Funk, in press).
The Structured Clinical Interview for DSM-IV (SCID-IV; First, Spitzer, Gibbon, & Williams, 2002) was used as the criterion measure for mental disorders. The SCID-IV is a structured interview that provides Diagnostic and Statistical Manual of Mental Disorders (DSM) diagnoses for AXIS I and AXIS II disorders based on both 30-day and lifetime information. It is generally considered to be a gold standard for assessing substance use and mental disorders. The screening and SCID instruments are considered to be in agreement when the participant’s score is above the mental disorder cutoff on the screener and the participant receives a mental disorder diagnosis on the SCID, or alternatively when the participant’s score does not reach criterion on either.
All participants were given a modified version of the CJDATS Intake Interview (CJDATS, 2005; Fletcher, Lehman, Wexler, & Melnick, 2007 [this issue]), which consists of a structured interview that encompasses the present incarceration and sociodemographic background information including education and employment, criminal history, health and psychological status, and drug history.
Several factors are involved in determining the precision of screening instruments, and the criteria chosen depend, in part, on the purposes of and circumstances in which the screening is administered. The sensitivity of an instrument considers only those individuals who actually have the particular disorder. In the present instance, sensitivity represents the percentage of individuals who have a positive diagnosis on the SCID and who the screening instrument has correctly identified as having a mental disorder. For example, if the SCID finds 100 people with the condition and the screening instrument correctly identifies 80, the sensitivity of the instrument would be 80%. The specificity of an instrument considers only those people who do not have the disorder. In the present instance, it represents the percentage of individuals who do not have a SCID diagnosis who are correctly identified as not having a mental disorder by the screening instrument. If 100 people do not have the condition and 60 are accurately identified, the specificity of the instrument would be 60%. The overall accuracy of the instrument represents the total number of people correctly identified as either having the condition or not having the condition over the total number who were screened. In the example above, this would be 80 + 60 = 140 of 200, or 70%. In instances (such as the current study) where the frequency of mental disorders is expected to be high and where it is critical not to miss cases, sensitivity is generally stressed. In instances where the expected frequency is low and the population to be tested is very large, specificity is more important, largely because too many referrals for assessment could surpass the available resources.
Instruments were administered in two face-to-face sessions conducted within 1 month of each other. Literacy was of concern with this prison population, and both sessions were orally administered to avoid any misunderstandings. The first session consisted of completing the informed consent and administering the initial test battery consisting of the CJDATS Intake Interview and the CODSI Screening Battery, in that order. The order in which each of the three mental health screeners appeared within the Screening Battery was randomized across participants to control possible ordering effects. The SCID (Version IV) was administered in the second session.
The first session was administered by experienced interviewers who received standardized training (independently provided by each of the participating research centers) in obtaining informed consent and administering the Intake Interview and the Screening Battery. Training consisted of reading manuals on proper interviewing techniques, participating in mandatory training on human participants, conducting interviews under the observation of experienced staff, and completing multiple practice interviews with supervisors. The SCID interviews (the second session) were conducted by personnel trained in SCID administration under the oversight of a highly experienced SCID supervisor who reviewed all interviews for completeness and accuracy. To avoid contaminating the SCID interview and diagnosis with the results of the Screening Battery, SCID interviewers had no knowledge of the results of the first session.
The sample consisted of consecutive new admissions1 to prison substance abuse treatment programs across the participating CJDATS research centers (NDRI Rocky Mountain, Brown University, University of California, Los Angeles, and Texas Christian University). The rationale for choosing new admissions to prison substance abuse treatment programs was twofold. First, the presence of COD represents a significant problem for these programs (Sacks et al., 2004). Second, previous research indicated a high prevalence rate for mental disorders among admissions to prison substance abuse treatment programs—59% reported previous psychological treatment, whereas 68% reported serious depression, 61% serious anxiety, 46% trouble controlling violent behavior, and 11% a previous suicide attempt (Prendergast, Hall, Wexler, Melnick, & Cao, 2004). The relatively high percentage of participants with a mental disorder provides a sufficient number of positive cases needed for testing the screening battery on a relatively small sample. Finally, the ability to identify mental disorders among offenders entering a substance abuse program was considered to be an initial step in the development of a COD screening tool for the general prison population.
The planned sample for the entire project will consist of 330 cases, 100 in the pilot study and 230 in the validation study. The 100 pilot cases reported here were randomly chosen from the 182 cases available the time of the analysis. A stratified random sampling procedure was used to ensure an equal distribution across three of the participating CJDATS research centers (NDRI Rocky Mountain, Brown University, and Texas Christian University), with an additional 8 cases from the University of California, Los Angeles, which had fewer cases at the time of the analysis. By design, women represented 25% of the sample (75 male and 25 female), a ratio similar to that of men to women in the prison populations at these sites. To facilitate the interpretation of the findings, the sample of women was further stratified so as to produce the same ratio of positive SCID diagnoses as the men (60%). All testing was completed within 6 weeks of entry into the program.
The four CJDATS research centers obtained data from 13 different prison substance abuse treatment programs. Throughout the study (pilot and validation; only the pilot study is discussed in this article), 29 participants refused to participate either in the full CODSI study or in completing the SCID (the criterion instrument discussed above), representing a 9% refusal rate. Because this rate of refusal is relatively low and not a threat to validity, the authors did not collect any additional information on the participants who refused to participate. The reasons for refusing to participate in the study were consistent across sites and included: not interested in participating in studies, denial of drug history, had ADHD and sitting for long periods was a problem, and did not want to spend the time.
The precision of the instrument was determined by the sensitivity, specificity, and overall accuracy as defined above. These were determined by cutoff scores used to classify individuals as having a potential mental disorder severe enough to warrant further diagnostic assessment and special placement. A range of cutoff scores wide enough to produce a curve of ascending to descending overall accuracy was calculated. When overall accuracy was the same at two cutoff scores, the cutoff score with the highest sensitivity was chosen. This decision emphasizes the need for substance abuse programs to identify individuals in need of special services. The discriminant value of the instruments was determined by calculating receiver operating characteristics (ROC; Metz, 1978) curves for each instrument. These curves measure the extent to which instruments discriminate over chance, where chance is equivalent to a diagonal line with half the area of the graph under the curve. When the actual plotted line lies above the diagonal the additional area under the curve indicates the difference between chance and the actual discriminant value of the test. Generally, this discriminant value should approximate 80% or better. In addition to evaluating the precision and discriminant value of the three instruments (hereafter referred to as the standard instruments), regression analysis was used to determine the predictive value of individual items. Items that demonstrated statistically unique variance were combined to form a new instrument and tested by the same criteria as the standard instruments (see below).
Table 1 shows the sensitivity, specificity, and overall accuracy for the three standardized instruments using a spread of cutoff points. The MHSF demonstrated the highest overall accuracy (73%) and second highest sensitivity (90%), with a cutoff score of 3 or higher; specificity was less satisfactory at 47.5%. A cutoff point of 6 produced a better balance between sensitivity (75%) and specificity (68%), with an equivalent degree of overall accuracy (72%). The GSS produced similar overall accuracy with a cutoff score of 2 or higher (71%), based primarily on the sensitivity score (83%) while achieving relatively low specificity (45%). The MINI, with a cutoff score of 5 or 6, closely approximated the other instruments in overall accuracy (69%). At a cutoff score of 6, the MINI was similar to the MHSF with a cutoff score of 6 in both sensitivity (73%) and specificity (63%). The MINI produced the highest level of specificity (80%) at a cutoff score of 8 with sensitivity of 50% and overall accuracy of 62%.
ROC curves shown in Figure 1 demonstrate a higher discriminant value for the MHSF followed by the MINI and the GSS short screener with areas under the curve of .805, .741, and .731, respectively.
In an attempt to explore the possibility of developing a more powerful screener using selected items from each of the three standard instruments, all items from the three instruments were correlated with a positive diagnosis on the SCID. Items with a statistically significant (p < .05) correlation were then tested in regression analysis against the positive SCID diagnosis to remove items accounting for the same variance. The regression results are shown in Table 2.
Six items with standardized coefficients ranging from 0.194 to 0.389 (disregarding the signs) and p values ranging from p < .000 to p < .02 were significant in the regressions. These items and the instruments from which they were drawn included (a) “Have you ever felt you needed help with your emotional problems or have other people told you that you should get help for your emotional problems?” (MHSF), (b) “Have you ever talked to a psychiatrist, psychologist, therapist, social worker, or counselor about an emotional problem?” (MHSF), (c) “Have you been told by teachers, guidance counselors, or others that you have a special learning problem?” (MHSF, reverse scored), (d) “Have you ever been advised to take medication for anxiety, depression, hearing voices, or for any other emotional problem?” (MHSF, reverse scored), (e) “Have you had one or more occasions when you felt intensely anxious, frightened, uncomfortable, or uneasy even when most people would not feel this way?” “Do you feel anxious or uneasy in a place or situations where you might have the panic like symptoms we just spoke about? Or do you feel anxious or uneasy in a situations where help might not be available or escape might be difficult?” (MINI), (f) “During the past 12 months, have you been a bully or threatened other people two or more times?” (GSS). These items formed a 6-item screener, the Co-occurring Disorder Screening Instrument for Mental Disorders (CODSI-MD). Although the reverse scoring for 2 of these items appears unusual, the reverse scores compensate for an overestimation of mental disorder by the 4 positively scored items. unusual, the reverse scores compensate for an overestimation of mental disorder by the 4 positively scored items.
The area under the ROC curve for the 6-item CODSI-MD was .808, indicating good discriminant value. Table 3 shows sensitivity and specificity at several potential cutoff scores. At a score of 3 or higher, the CODSI-MD achieved an overall accuracy of 81.0%, with a sensitivity of 86.7% and a specificity of 72.5%.
Table 4 shows the results by gender. The CODSI-MD performed better for men than for women, but the overall accuracy was acceptable for both genders. Disaggregating the results by gender gave ROC curve values of .833 for men and .733 for women. With a cutoff score of 3 or higher, sensitivity for men was 84.4%, specificity was 80.0%, and overall accuracy was 82.7%. The same cutoff score of 3 also produced the highest overall accuracy for women with sensitivity of 93.3%, specificity of 50.0% and overall accuracy of 76.0%.
A secondary aim that emerged in the course of conducting this study was to determine the precision of instruments for detecting severe mental disorders (major depression, schizophrenia, and bipolar disorders) and risk for suicide. Along with the three standard instruments, the study tested a 3-item screener for severe mental disorders (the CODSI-SMD), which was developed using a regression analysis to determine the items in the standard instruments that were most associated with severe disorders. The first 3 items selected in the regression, the instrument in which they appeared, and the standardized coefficients were (a) “Have you felt sad, low, or depressed most of the time for the past two years?” (MINI, β = .266, t = 3.053, p < .01), (b) “Did you ever attempt to kill yourself?” (MHSF, β = .515, t = 3.921, p < .01), and (c) “Have you ever had a period of time when you were so full of energy and your ideas came very rapidly, when you talked nearly non-stop, when you moved quickly from one activity to another, when you needed little sleep, and believed you could do almost anything?” (MHSF, β = .205, t = 2.668, p < .01).
Table 5 shows the overall sensitivity, specificity and overall accuracy for CODSI-SMD, the MHSF, MINI and GSS Internal Disorder Screener scale (GSS-IDS; the subsection of the GSS that pertains to severe mental disorder) using the cutoff score reflecting the highest overall accuracy. The CODSI-SMD with a cutoff score of 2, and the MHSF with a cutoff score of 11, showed the best scores for overall accuracy (82.0% and 76.0%, respectively), specificity (90.3% and 88.9%, respectively), and sensitivity (60.7% and 42.9%, respectively). Although these two instruments were similar in specificity, the CODSI-SMD showed higher sensitivity (the ability to detect severe mental illness when it is present). Table 5 shows the results of the ROC curves for the four instruments with the cutoff scores specified in the table. Areas under the curve, indicative of the relative discriminant value over chance of each instrument, were .755 for the CODSI-SMD, .659 for the MHSF, .642 for the MINI, and .639 for the GSS-IDS.
Of the three standard instruments tested, the MHSF produced the highest overall accuracy (73.0%) and the best sensitivity (90.0%). In other words, the MHSF was the most consistently accurate instrument under the circumstances of the study and was best able to identify any mental disorder when present. The ROC curve also indicated that the MHSF was the best predictor of the presence of a mental disorder. The MINI produced the highest specificity score (52.5%), indicating that it was best able to identify the absence of any mental disorder. Nevertheless, at the suggested cutoff scores, the three instruments were within 4 percentage points of each other in overall accuracy. In a sample of this size, the shift of only a few individuals would have altered the relative position of the instruments. Furthermore, these overall accuracy scores, and the selection of the best cutoff points, depend on the prevalence of mental disorder in the population to be screened. Specifically, because the pilot study sample had a high prevalence of mental disorders (60% had an identifiable mental disorder), selection would favor an instrument and cutoff score with a high sensitivity rating and only a moderate specificity rating. If the population were to have had a low prevalence of mental disorders (10% to 20%), then the MINI with a cutoff score of 8 or higher would have produced the greatest overall accuracy with a specificity score of 80% and a sensitivity score of 50%. Alternatively, in populations with prevalence rates above 60%, the MHSF with a cutoff score of 2, or the GSS with a cutoff score of 1, would have produced the highest overall accuracy with their specificity scores of 95%. Thus, in the pilot phase of the present study, the performance of the three standard instruments was approximately equal, and the choice of an instrument would depend on the expected prevalence of mental disorders in the population to be screened.
The 6-item CODSI-MD, developed from items selected from the standard screeners, performed approximately as well as the standard instruments, demonstrating promise as a potential brief screening instrument. The CODSI-MD had an overall accuracy of 81%, successfully identified 83% of the men and 76% of the women with mental disorders among admissions to prison substance abuse treatment programs, and correctly identified 80% of the men and 50% of the women who did not have a mental disorder. Although given the high prevalence of mental disorder among female inmates receiving substance abuse treatment in prison it would be preferable to have an instrument that screened out a greater proportion of women, the overall accuracy of the CODSI-MD was high and the instrument could be a useful tool to distinguish both men and women in need of further psychiatric assessment and specialized psychiatric services. The need for reverse scoring of 2 of the items in the CODSI-MD, as indicated by the negative β values in the regression analysis, can likely be attributed to the need to compensate for the overinclusiveness of the remaining 4 items.
The development of a screening device to identify severe mental disorders, the CODSI-SMD was a secondary goal that emerged in the course of conducting the study. All of the standard instruments were better at screening out those whose mental disorder was not severe (specificity) than in identifying those whose mental disorder was severe (sensitivity); compared to the standard instruments, the 3-item CODSI-SMD proved to be highly proficient in identifying severe mental disorders. The CODSI-SMD surpassed the other instruments, producing an overall accuracy of 82.0% and sensitivity of 60.7% while maintaining about the same level of specificity (90.3%) as the best standard instrument, the MHSF. The MHSF also performed well, with an overall accuracy of 76.0%, specificity of 88.9%, and sensitivity of 42.9% at a cutoff score of 11. The MINI followed, with an overall accuracy of 72.0%, specificity of 81.9%, and sensitivity of 46.4% at a cutoff score of 10. The GSS-IDS, at a cutoff score of 5, had an overall accuracy of 68.0%, with specificity of 84.7% and sensitivity of 25.0%. Thus, the CODSI-SMD showed promise as a screening instrument that can eliminate individuals without a major mental disorder and assist in identifying those individuals with a severe disorder.
Although the study included all inmates entering substance abuse treatment programs in prison, the initial interview may have been conducted up to 2 weeks into the treatment program, followed within a month by the SCID interview. Some dropout would be expected during this time, and inmates with severe mental disorders would likely be at high risk for such dropout. As a result, some severe mental disorders may be underrepresented in the sample. Also, as encouraging as the present results appear, they must be cautiously viewed. No firm conclusions can be drawn about the 6-item CODSI-MD or the 3-item CODSI-SMD screeners compared to the three other measures tested in this pilot study until the validation study is completed. Because the items for the 3- and 6-item screeners were selected for their discriminant utility within this particular sample, one would expect the screeners to perform well compared to other instruments developed on other samples. A better test of the value of the CODSI-MD and CODSI-SMD compared to the other instruments will be determined when the validation phase of this project is completed and all of the instruments are compared on a new sample.
It should be noted that the CODSI mental health screeners have the same limitation as all screening instruments, namely, that they are designed only to identify people who have a disorder. General screening instruments, such as the CODSI, are not designed to be a comprehensive diagnostic interviews nor to provide the basis for treatment planning. General screeners of this type are solely intended to indicate the potential existence of a mental disorder that warrants a more complete mental status workup and special services. It remains the function of the program intake protocol to conduct a more comprehensive clinical evaluation, including diagnostic assessment, for purposes of treatment planning.
As indicated, a main future direction will be to validate the two CODSI instruments and to evaluate the standardized instruments on a larger sample. Further development of the CODSI will focus on screening for both substance use and mental disorders at entry into prisons and jails. Adapting the CODSI for use at entry into prisons and jails will include modifying the instrument to suit a population that will have a prevalence of mental disorders considerably lower than that of the present sample. The lower prevalence rate will mean that the overall accuracy will be determined more by the instrument’s specificity, or its ability to identify individuals who do not have a mental disorder. In large populations with relatively low prevalence, specificity is critical so as not to overtax the ability of the criminal justice system to conduct assessments. We expect that, under these circumstances, further modification of the CODSI for determining mental disorders may well be necessary.
This study was supported by Grant 1 U01 DA16200, Rocky Mountain Research Center for CJDATS, funded under a cooperative agreement, National Criminal Justice Drug Abuse Treatment Studies (CJDATS), from the U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health (NIH), National Institute on Drug Abuse (NIDA), with additional support from the National Institute on Alcohol Abuse and Alcoholism (NIAAA), Centers for Disease Control and Prevention (CDC), and the Substance Abuse and Mental Health Services Administration (SAMHSA), and the Center for Substance Abuse Treatment (CSAT). Views and opinions are those of the authors and do not necessarily reflect those of Department of Health and Human Services, the NIH, NIDA, or of other participants in CJDATS. The authors gratefully acknowledge the collaborative contributions by federal staff from NIDA, members of the Coordinating Center (University of Maryland at College Park, Bureau of Governmental Research and Virginia Commonwealth University), and the other research center grantees of the NIH/NIDA CJDATS Cooperative that did not participate in the CODSI project: Connecticut Department of Mental Health and Addiction Services; National Development and Research Institutes, Inc., Center for Therapeutic Community Research; University of Delaware, Center for Drug and Alcohol Studies; University of Kentucky, Center on Drug and Alcohol Research; and University of Miami, Center for Treatment Research on Adolescent Drug Abuse.
Stanley Sacks, PhD, is the director of CIRP (Center for the Integration of Research and Practice at NDRI) and principal investigator for several major government grants, including current projects within the Colorado prison system. During the past 15 years, he has concentrated his research efforts in the area of co-occurring disorders, particularly within special populations, such as the homeless and the criminally involved. His expertise in co-occurring disorders, recognized nationally and internationally, resulted in his appointment as chair of CSAT’s TIP 42, Substance Abuse Treatment for Persons with Co-Occurring Disorders, and his selection as expert leader for SAMHSA’s national initiative, the Co-Occurring Center for Excellence (COCE), to improve treatment services for co-occurring disorders nationwide.
Gerald Melnick, PhD, is a senior principal investigator at CIRP (at NDRI). His current research activities include surveying the responsiveness of state criminal justice systems to the treatment of co-occurring substance use and mental disorder (COD), the development of screening instruments for COD in criminal justice settings, the effect of stigma on recidivism in first-time, nonviolent felons, and the effect of organizational culture on clients’ engagement in substance abuse treatment, substance abuse treatment outcomes, and staff attitudes toward correctional institutions. His previous studies included the development of several instruments, including the Survey of Essential Elements Questionnaire (SEEQ), the Multimodality Quality Assurance Instrument (MQA), a Client Matching Protocol (CMP), and the Circumstance, Motivation and Readiness scales (CMR).
Carrie Coen, MA, is an assistant project director at the Center for the Integration of Research and Practice (CIRP) at NDRI who manages several CJ-DATS research projects for CIRP investigators. She has contributed to other completed projects with addicted populations (with and without co-occurring mental disorders).
Steven Banks, PhD, is a mathematical statistician who provides statistical consulting to CIRP and who has appointments at the University of Massachusetts Medical School, the National Institutes of Health, and the Florida Mental Health Institute. His research interests include large-scale observational studies using administrative databases and methodologies for analyzing randomized clinical trials. He is a statistical consultant to a number of organizations and agencies, including NDRI.
Peter D. Friedmann, MD, MPH, is an associate professor of medicine and community health at Brown Medical School, an established substance abuse health services researcher and internal medicine physician, and an ASAM-certified addictionist. He has published extensively on the organization and integration of substance abuse treatment services, treatment process and outcomes, and relapse prevention. He is principal investigator of the Rhode Island Research Center of NIDA’s Criminal Justice Drug Abuse Treatment Studies (CJ-DATS) and directs the Program to Integrate Psychosocial, & Health Services in Chronic Disease, & Disability, a center for health services and translational research at the Providence VA Medical Center.
Christine Grella, PhD, is a research psychologist at the Integrated Substance Abuse Programs (ISAP) at the University of California, Los Angeles. Her research focuses on the intersection of multiple service delivery systems, including substance abuse treatment, mental health, child welfare, health services, HIV services, criminal justice, and the relationship of service delivery to treatment outcomes. She has published her work widely in the areas of addiction, mental health, health services, and evaluation research.
Kevin Knight, PhD, is a research scientist at the Institute of Behavioral Research (IBR) at Texas Christian University. His work, reported in several publications, including a book titled Treating Addicted Offenders: A Continuum of Effective Practices, centers on evaluating substance abuse treatment process and outcomes and on the development of evaluation systems for correctional settings. He is serving as the Southwest Research Center principal investigator on the Criminal Justice Drug Abuse Treatment Studies (CJDATS) project, a large NIDA-funded cooperative agreement designed to improve correctional treatment. He has worked closely with criminal justice agencies and data systems at national, state, and regional levels in the United States. He also serves on journal editorial boards, including serving as coeditor of Offender Substance Abuse Report, and he participates in advisory activities for a variety of organizations that address substance abuse and related policy issues.
1Inmates were to receive the initial test battery within 2 weeks of entry to the program. Exceptions were made under special circumstances, such as missing potential participants because of lockdowns, which meant that interviews could not be scheduled within the 2-week interval.
Stanley Sacks, National Development and Research Institutes, Inc., New York, NY.
Gerald Melnick, National Development and Research Institutes, Inc., New York, NY.
Carrie Coen, National Development and Research Institutes, Inc., New York, NY.
Steven Banks, University of Massachusetts Medical School, Worcester.
Peter D. Friedmann, Brown University, Providence, RI.
Christine Grella, University of California, Los Angeles.
Kevin Knight, Texas Christian University, Ft. Worth.