|Home | About | Journals | Submit | Contact Us | Français|
To date, there has not been a time-efficient and resource-conscious way to identify cognitive impairment in patients with substance use disorders (SUD). The present study assesses the validity, accuracy, and clinical utility of a brief (10 min) screening instrument, the Montreal Cognitive Assessment (MoCA), in identifying cognitive impairment among SUD patients. The Neuropsychological Assessment Battery-Screening Module (NAB-SM), a 45-minute battery with known sensitivity to the mild-to-moderate deficits observed in SUD patients, was used as the reference criterion for determining agreement, rates of correct and incorrect decision classifications, and criterion-related validity for the MoCA. Classification accuracy of the MoCA, based on receiver-operating characteristic (ROC) analysis, was strong, with an area under the ROC curve = 0.86 [95% CI: 0.75-0.97]. The MoCA also showed acceptable sensitivity (83.3%) and specificity (72.9%) for the identification of cognitive impairment. Using a cut-off of 25 on the MoCA, the overall agreement was 75.0%; chance-corrected agreement (kappa) was 41.9%. These findings indicate that the MoCA provides a time-efficient and resource-conscious way to identify SUD patients with neuropsychological impairment, thus addressing a critical need in the addiction treatment research community.
Cognitive impairment in patients with substance use disorders (SUDs) contributes to poorer treatment outcomes, including decreased treatment retention (Aharonovich, et al., 2006; Aharonovich, Nunes, & Hasin, 2003; Donovan, Kivlahan, Kadden, & Hill, 2001; Fals-Stewart, 1993; Fals-Stewart & Schafer, 1992) and less abstinence from substances of abuse (Aharonovich, et al., 2006). Cognitive dysfunction has also been shown to have a negative impact on “therapeutic mechanisms of change” (Bates, Pawlak, Tonigan, & Buckman, 2006). For example, it is associated with less treatment adherence (Bates, et al., 2006), less treatment engagement (Katz, et al., 2005), less readiness to change (Blume, Schmaling, & Marlatt, 2005), lower self-efficacy (Bates, et al., 2006), decreased insight (Horner, Harvey, & Denier, 1999; Shelton & Parsons, 1987), increased denial of addiction (Rinn, Desai, Rosenblatt, & Gastfriend, 2002), and greater reflection impulsivity (Clark, Robbins, Ersche, & Sahakian, 2006). In addition, cognitive impairment among alcoholics has been shown to have a negative impact on drink refusal skill acquisition and aftercare treatment attendance (Smith & McCrady, 1991).
Estimates regarding the prevalence of cognitive impairment in SUD patients vary widely and range from about 30-80% (Bates & Convit, 1999; Grant, Adams, Carlin, & Rennick, 1977; Meek, Clark, & Solana, 1989; NIDA, 2003; O’Malley, Adamse, Heaton, & Gawin, 1992; Parsons & Nixon, 1993; Rourke & Loberg, 1996). These deficits may range from the relatively subtle temporary effects of cannabis use (Bolla, Brown, Eldreth, Tate, & Cadet, 2002; Hart, van Gorp, Haney, Foltin, & Fischman, 2001; Pope, 2002; Pope, et al., 2003; Pope, Gruber, & Yurgelun-Todd, 2001; Solowij, et al., 2002), to the moderate executive control deficits observed in chronic cocaine users even following several months of abstinence (Di Sclafani, Tolou-Shams, Price, & Fein, 2002; O’Malley, et al., 1992; Strickland, et al., 1993; Woicik, et al., 2009). Although these estimates do not include alcoholics who develop permanent cognitive deficits such as Wernicke-Korsakoff Syndrome, they do include the enduring visuospatial information processing deficits observed in non-demented persons with alcohol use disorders (Schandler, Clegg, Thomas, & Cohen, 1996).
Specialized treatment enhancements aimed at cognitively impaired SUD patients, such as cognitive rehabilitation (Fals-Stewart & Lucente, 1994; Goldstein, Haas, Shemansky, Barnett, & Salmon-Cox, 2005; Grohman & Fals-Stewart, 2003; Grohman, Fals-Stewart, & Donnelly, 2006) and accommodation (Czuchry & Dansereau, 2003; Czuchry, Dansereau, Dees, & Simpson, 1995; Dansereau, Dees, Chatham, Boatler, & Simpson, 1993; Dansereau, Joe, & Simpson, 1995; Newbern, Dansereau, Czuchry, & Simpson, 2005) have shown some success, but this is still an area in its infancy and further research is needed. One of the primary challenges in developing treatments and enhancements for cognitively impaired SUD patients is a lack of knowledge about which patients should be targeted for specialized interventions (NIDA, 2003).
Unfortunately, neuropsychological assessment is typically not an aspect of patient evaluation in substance abuse treatment programs because it is prohibitively time and resource consuming. Further, studies show that cognitively impaired SUD patients cannot be adequately identified by drug counselors via clinical impression (Fals-Stewart, 1997) or through self-report (Horner, et al., 1999; Shelton & Parsons, 1987). If accounting for and addressing the presence of cognitive deficits among substance abusing patients involves, as a first step, identifying those with neuropsychological impairment, treatment providers and researchers alike need a practical neurocognitive assessment approach for patients with SUDs that is both accurate and comparatively less labor intensive.
The Montreal Cognitive Assessment (MoCA) (Nasreddine, et al., 2005) is a brief (10-minute) cognitive screening instrument that was developed on a geriatric population to be sensitive to mild cognitive impairment (MCI). Though not a clearly-defined syndrome, MCI is regarded as cognitive dysfunction in excess of normal age-related decline that does not interfere notably with activities of daily living and is often undetectable via standard mental status examination. The MoCA has been shown to be sensitive to subtle cognitive deficits in a variety of populations. In comparison to the Mini-Mental State Examination (MMSE) (Folstein, Folstein, & McHugh, 1975), the MoCA was shown to be more sensitive to early detection of cognitive decline in persons with asymptomatic cerebrovascular (CV) disease ( i.e., without signs of CV disease, but having one or more risk factors (Popovic, Seric, & Demarin, 2007). In part due to the findings of Popovic and colleagues, the National Institute of Neurological Disorders and Stroke (NINDS) recommends use of the MoCA over the MMSE as part of a brief, minimal dataset for identifying persons in the early stages of cognitive impairment related to vascular factors (Hachinski, et al., 2006). For example, the MoCA is recommended for identifying subtle changes in cognitive performance due to “silent stroke.” The MoCA has also been found to be more sensitive than the MMSE in detecting cognitive impairment in Parkinson’s patients (Nazem, et al., 2009; Zadikoff, et al., 2008). Further, based on its agreement with a lengthier neuropsychological battery, data support that the MoCA is reliable and valid as a screening test for detection of early or mild cognitive impairment in Parkinson’s disease (Gill, Freshman, Blender, & Ravina, 2008).
The purpose of the present study was to assess the validity, accuracy, and clinical utility of the Montreal Cognitive Assessment in identifying cognitive impairment among SUD patients in a clinical research setting. We accomplished this through the following steps: (1) assessment of the validity of the screener through its strength of agreement with an accepted standard criterion measure; (2) assessment of its classification accuracy by generating a confusion matrix and deriving measures of sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve; and (3) assessment of its clinical utility via a combination of qualitative assessment of patient acceptability and practical considerations.
Participants were 60 adult patients receiving treatment at the McLean Hospital Alcohol and Drug Abuse Treatment Program (ADATP), in Belmont, Massachusetts with at least one current substance use dependence diagnosis (DSM-IV criteria). Study participants were recruited through the ADATP Partial Hospital and Residential Programs. To be eligible for the study, participants had to meet the following inclusion criteria: (a) recent admission to either the Partial Hospital or Residential Program at the McLean Hospital Alcohol and Drug Abuse Treatment Program, (b) any non-nicotine DSM-IV substance dependence disorder, (c) abstinence from all drugs of abuse other than nicotine for at least 7 days, and (d) age 18-65 years. The inclusion criteria were designed to be as broad as possible, to maintain ecological validity of the study findings without compromising internal validity of the neuropsychological measures. Exclusion criteria included acute intoxication or withdrawal, and any medical illness or psychiatric condition (including dementia) that in the view of the investigators would interfere with provision of consent or valid self-report or otherwise compromise participation in research.
On average, participants were [mean (standard deviation)] 38.3 (13.2) years of age and completed 15.0 (2.4) years of education. Participants were predominantly Caucasian (95%; n=57) and unemployed during the past month (60%; n=42). About half of the participants were male (52%; n=31), and half were never married (52%; n=31).
After providing written informed consent, participants completed all study measures at a single time point requiring approximately 2.5 hours. Cognitive measures included the Montreal Cognitive Assessment (MoCA) (Nasreddine, et al., 2005), Neuropsychological Assessment Battery-Screening Module (NAB-SM) (Stern & White, 2003), and the National Adult Reading Test-Revised (Blair & Spreen, 1989). Administration of the MoCA and NAB-SM were counterbalanced to preclude order effects. Substance use disorder diagnoses were made using the DSM-IV checklist for SUD (Wu, et al., 2009); all other Axis I diagnoses were made using the Structured Clinical Interview (SCID-I/P) for DSM-IV-TR - Patient Edition (First, Spitzer, Gibbon, & Williams, 2002). Quantity and frequency of drug and alcohol use during past 30 days, 90 days, and one year were measured using the Timeline Followback (TLFB) Method (Sobell & Sobell, 1992). Anxiety was assessed using the Hamilton Anxiety Scale (Hamilton, 1959, 1960), and depression was assessed using the Quick Inventory of Depressive Symptomatology (Rush et al., 2003). Participants were paid $50 (in the form of gift cards) for completing all study assessments.
The Montreal Cognitive Assessment (MoCA) samples behavior across fourteen performance tasks that engage multiple cognitive domains including attention, language, visuospatial, executive, and memory (Table 1). The time to administer the MoCA is approximately 10 minutes. The total possible score is 30 points (31 if the patient is aged 12 years or younger); and a score of 26 or greater is classified as “normal,” i.e., without evidence of cognitive impairment (Nasreddine, et al., 2005).
The Neuropsychological Assessment Battery-Screening Module (NAB-SM) (Stern & White, 2003) assesses cognitive functioning across five domains: attention, language, memory, visuospatial, and executive (Table 1). Administration time is approximately 45 minutes. The NAB-SM has been recommended for use with SUD patients because of its sensitivity (.81) and specificity (.92) in classifying patients with present or absent cognitive impairment in this population (Grohman & Fals-Stewart, 2004). The NAB-SM was used as the reference criterion for determining agreement and rates of correct and incorrect decision classifications for the MoCA. Decision classifications using the NAB-SM were based on the dichotomous Total Screening Index (TSI). The TSI is a composite measure of the five cognitive domain scores, and is a standardized score representing the examinee’s overall test performance. Total Screening Index values less than or equal to 84 (corresponding to a score of more than 1 standard deviation, or 15 points, below the mean of 100) are indicative of cognitive impairment. Test norms were demographically corrected relative to a standardization sample (N = 1,448) of neurologically healthy individuals of the same age, sex, and educational level (Stern & White, 2003).
The present study assessed the accuracy, validity, and clinical utility of the Montreal Cognitive Assessment in identifying cognitively impaired SUD patients in a clinical research setting. Accuracy and validity of the MoCA were evaluated statistically; and clinical utility was assessed via practical considerations and through qualitative assessment of patient acceptability. Clinical accuracy was evaluated using a decision theory approach. Based on the NAB-SM as the reference criterion, analyses were performed to assess sensitivity and specificity of the MoCA in accurately detecting cognitive impairment. Receiver-operating characteristic (ROC) analysis was conducted to assess the quality of the screener at a range of possible cut-off values. Criterion-related validity of the MoCA was evaluated through its overall agreement with the NAB-SM. The kappa statistic (k) for dichotomous data (presence/absence of cognitive impairment) was used to measure chance-corrected agreement. Patient acceptability was assessed through qualitative assessment of two Likert-type questions (“How demanding was this test?” and “How unpleasant was this test?”) on a scale from one to five, anchored with “not at all” (value of one) and “extremely” (value of five).
On average, each participant met criteria for 1.4 (SD=0.70; range of 1-3) substance dependence diagnoses. The most common diagnosis was for alcohol dependence (65%; n=39), followed by dependence on opioids (32%; n=19), cocaine (17%; n=10), cannabis (12%; n=7), benzodiazepine (10%; n=6 ), and amphetamine (8%; n=5). Primacy of SUD diagnoses was not determined. Forty-one percent of participants met criteria for any co-occurring DSM-IV Axis I disorder. The most common co-occurring disorder was Bipolar Affective Disorder (17%; n=10), followed by Post Traumatic Stress Disorder (13%; n=8), Generalized Anxiety Disorder (12%; n=7), Panic Disorder (8%; n=5), and Major Depressive Disorder (5%; n=3) and Social Phobia (5%; n=3).
Thirty-eight percent of patients were classified as impaired based on the MoCA, and 20% were classified as impaired based on the NAB-SM (Table 2). Across the five cognitive domain scores that comprise the NAB-SM total composite score, the proportion of participants classified as “impaired” ranged from a high of 37% for the attentional domain to a low of 12% for the language domain. Classification accuracy of the MoCA was based on receiver-operating characteristic (ROC) analysis, which showed an area under the ROC curve equal to 0.86 [95% CI: 0.75-0.97] (Figure 1). Areas under the ROC curve for the five cognitive domain scores comprising the NAB-SM total composite ranged from 0.73 to 0.92 and included the following scores: Attention = 0.73 [95% CI: 0.60-0.86], Language 0.9205 [95% CI: 0.80-1.00], Memory 0.76 [95% CI: 0.61-0.90], Spatial 0.77 [95% CI: 0.62-0.930], and Executive 0.80 [95% CI: 0.64-0.96]. The MoCA also showed acceptable sensitivity (83.3%) and specificity (72.9%) to identify cognitive impairment. Using a cut-off of 25 or below on the MoCA, the overall agreement was 75.0%; chance-corrected agreement (kappa) was 41.9%. The quality of the screener across a range of other possible cut-off values was also assessed via ROC analysis (Table 3). A visual inspection of the trade-off between sensitivity and specificity across cut-point values shows that 25 is the optimal cut-point for a sample of SUD patients.
Assessment of patient acceptability yielded the following results. Twenty-seven percent of participants found the MoCA to be “not at all demanding,” 61% of participants found it “somewhat” or “fairly” demanding, and 12% found it “rather” or “very” demanding. Fifty-five percent of participants found the MoCA “not at all unpleasant,” 40% found it “somewhat” or “fairly” unpleasant, and 5% found it “rather” or “very” unpleasant.
The purpose of this study was to examine the accuracy, validity, and clinical utility of a brief cognitive screening instrument in identifying cognitive impairment in SUD patients in a clinical research setting. Results generally support its appropriate and practical use in this population. Based on its agreement with a reference criterion, the MoCA showed evidence of criterion-related validity and good accuracy in correctly classifying cognitive impairment cases and non-cases.
The Neuropsychological Assessment Battery-Screening Module (NAB-SM) served as the reference criterion, and was used for determining agreement and rates of correct and incorrect decision classifications. The NAB-SM and MoCA similarly sample a broad range of cognitive domains. Results of the present study showed good agreement between the MoCA and the five NAB-SM cognitive sub-domains, including attention, language, memory, spatial, and executive. Thus, among the processes sampled across the NAB-SM cognitive sub-domains, none are disproportionately weighted in the MoCA. However, unlike the NAB-SM, the MoCA does not include performance tasks related to psychomotor speed or visual learning and delayed recognition. Also, the NAB-SM assesses verbal memory through learning and delayed recall of verbally-presented narrative, whereas the MoCA assesses verbal memory through immediate and delayed recall of five unrelated words. How these differences might affect overall test performance is not apparent.
Based on patient acceptability and other practical considerations the MoCA has good clinical utility. Assessment of patient acceptability indicated that patients in general did not find the MoCA particularly unpleasant or demanding. It also provides an accurate and valid screening measure that is easy to use, time-efficient, and resource-conscious. This makes conducting cognitive assessment with SUD patients more practical for treatment settings and providers, such that patients who screen positive may be referred to more comprehensive evaluation. Further, the MoCA, including protocol sheet, instructions for administration and scoring criteria are available at no cost by the test developer (http://www.mocatest.org/). The MoCA may be used, reproduced, and distributed without permission for clinical and educational non-commercial purposes. For non-commercially funded research, it may be used with prior written permission. If used for commercially funded research, prior written permission and a licensing agreement are required. In contrast, the list price of the NAB-SM is $825, plus the cost of additional screening module record forms ($94) and response booklets ($52) per every 25 administrations. Purchase of the NAB-SM is restricted to professionals who meet competency-based qualification guidelines; and who have completed the registration and qualification process attesting to their eligibility on the basis of training, education, and experience.
In comparison to the 10-minute administration time required for the MoCA, the NAB-SM takes approximately 45 minutes to complete. Further, hand-scoring the NAB-SM can take 30 minutes or longer. Scoring software is available (i.e., NAB Software Portfolio; NAB-SP) that can reduce the time needed to score the NAB-SM, but requires a PC-based computer with CD-ROM drive for installation, and Internet connection or telephone for software activation. In comparison to the MoCA, administration of the NAB-SM also requires significantly more space - i.e., a larger working surface with sufficient space to spread-out testing materials including puzzle pieces and the stimulus book.
Finally, in addition to English, the MoCA has been translated into 22 languages. Multiple language versions of the MoCA have shown high sensitivity for screening patients with mild cognitive impairment, including the Korean (Lee, et al., 2008), Arabic (Rahman & El Gaafary, 2009), and Chinese (Wen, Zhang, Niu, & Li, 2008) language versions.
There are several strengths of this investigation. The natural heterogeneity of the sample is a strength of the present study because it demonstrates the validity of the MoCA for standard clinical practice. In other words, the ecological validity of the study findings is maximized without compromising the internal validity of the neuropsychological measures. Another strength is that the criterion measure has specifically been recommended for use with SUD patients because of its strong sensitivity and specificity in classifying patients with present or absent cognitive impairment in this population. The most common neuropsychological batteries typically require several hours of administration time, scoring, and interpretation (Rabin, Barr, & Burton, 2005). The 45-minute administration time and computerized scoring of the NAB-SM enabled a more comprehensive evaluation across cognitive domains while conferring two additional benefits: (a) enabled testing in a single day, and (b) avoided test fatigue that almost invariably results from lengthy neuropsychological batteries. Another strength of the present study is the counter-balanced presentation of the MoCA and NAB-SM, which avoids possible test order effects.
A limitation of this study is the unknown influence of abstinence duration on overall prevalence of cognitive impairment. An exclusion criterion for the present study was known substance use within seven days prior to study participation. This is consistent with the recommendations made by some investigators who suggest a duration of at least one week between admission to treatment and testing (Miller, 1985; Parsons & Farr, 1981). The rationale for this recommendation is that some cognitive recovery generally follows abatement of intoxication and acute abstinence effects. As a result, rates of detection among newly admitted patients may be artifactually inflated due to the effects of residual intoxication or withdrawal. However, because the goal of the study was not to study prevalence of cognitive impairment, but rather the concordance between the MoCA and NAB-SM, the latter should be relatively insensitive to any potential inflation.
The relationship between duration of abstinence and cognitive impairment, furthermore, is unclear. Paradoxically, cognitive performance may actually deteriorate slightly over the first few weeks of abstinence before gradually improving. For example, a recent study showed a gradual worsening in most neuropsychological categories, such that cocaine dependent persons with a positive urine drug screen for cocaine (typically indicating use within the past 72 hours) perform better on a broad range of neuropsychological measures in comparison to cocaine dependent individuals with a negative screen (Woicik, et al., 2009). These findings are consistent with a previous study showing that the scope of neuropsychological deficit among currently abstinent cocaine dependent persons actually increased from 72 hours to 14 days (Berry, et al., 1993). These findings suggest that the influence of abstinence duration on cognitive performance may not be linear. However, it is unknown how exclusive such “non-linear” effects of abstinence duration may be to predominantly heavy psycho-stimulants users. Therefore, there may be little relevance to the present study given that only 17% of patients met criteria for cocaine dependence and 8% met criteria for amphetamine or methamphetamine dependence.
A body of evidence is emerging showing that cognitive impairment in patients with substance use disorders (SUD) has a significant and negative impact on treatment outcomes and therapeutic mechanisms of change. Specialized treatments and enhancements aimed at improving outcomes for SUD patients have shown some success, but this is still an area in its infancy and further research is needed. One of the main challenges associated with developing treatments for cognitively-impaired SUD patients is uncertainty about which patients to target for specialized interventions. To date, the search for a brief cognitive screening instrument sensitive to the mild-to-moderate impairment observed in SUD patients has been unsuccessful. The present findings show the Montreal Cognitive Assessment addresses a critical need in the addiction treatment research community by providing a quick and accurate screening instrument that can expedite the progression of research in this area.