|Home | About | Journals | Submit | Contact Us | Français|
The MATRICS Psychometric and Standardization Study was conducted as a final stage in the development of the MATRICS Consensus Cognitive Battery (MCCB). The study included 176 persons with schizophrenia or schizoaffective disorder and 300 community residents. Data were analyzed to examine the cognitive profile of clinically stable schizophrenia patients on the MCCB. Secondarily, the data were analyzed to identify which combination of cognitive domains and corresponding cut-off scores best discriminated patients from community residents, and patients competitively employed vs. those not. Raw scores on the ten MCCB tests were entered into the MCCB scoring program which provided age-and gender-corrected T-scores on seven cognitive domains. To test for between-group differences, we conducted a 2 (group) × 7 (cognitive domain) MANOVA with follow-up independent t – tests on the individual domains. Classification and regression trees (CART) were used for the discrimination analyses. Examination of patient T-scores across the seven cognitive domains revealed a relatively compact profile with T-scores ranging from 33.4 for speed of processing to 39.3 for reasoning and problem-solving. Speed of processing and social cognition best distinguished individuals with schizophrenia from community residents; speed of processing along with visual learning and attention/vigilance optimally distinguished patients competitively employed from those who were not. The cognitive profile findings provide a standard to which future studies can compare results from other schizophrenia samples and related disorders; the classification results point to specific areas and levels of cognitive impairment that may advance work rehabilitation efforts.
The MATRICS Consensus Cognitive Battery (MCCB) was developed to address the absence of a uniform, standardized method by which to measure cognition in clinical trials of cognition-enhancing drugs (Marder & Fenton, 2004). The MCCB, now accepted as a standard by the U.S. Food and Drug Administration, is comprised of ten tests that assess seven cognitive domains (speed of processing, attention/vigilance, working memory, verbal learning, visual learning, reasoning and problem solving, and social cognition) (Nuechterlein et al., 2004). Previous articles have detailed the process of MCCB development, co-norming, and evaluation of co-primary measures for clinical trials (Green et al., 2008; Kern et al., 2008; Nuechterlein et al., 2008). In this paper, we extend previous findings by examining the cognitive profile of chronic, clinically stable schizophrenia outpatients on the MCCB using data from the two studies involved in its development (Psychometric and Standardization Study, PASS Phase I and II). Secondarily, we address two discrimination questions relevant to cognitive dysfunction in schizophrenia. Specifically, which combination of MCCB cognitive domains and corresponding cut-off scores best discriminate: a) schizophrenia individuals from community residents, and b) vocational status within the schizophrenia group.
PASS Phase I was conducted to evaluate the psychometric properties of tests in the beta version of the battery (20 tests) and included 176 schizophrenia and schizoaffective disorder outpatients from five academic sites (Duke University, Harvard University, University of Kansas, Maryland Psychiatric Research Center, and UCLA) (Nuechterlein et al., 2008). PASS Phase II gathered normative data on the MCCB and included 300 community residents aged 20-59 from the same sites (Kern et al., 2008). Inclusion criteria for schizophrenia participants and community residents are described in earlier publications (Kern et al., 2008; Nuechterlein et al., 2008) and summarized below:
For schizophrenia participants, inclusion criteria included: (a) a DSM-IV diagnosis of schizophrenia or schizoaffective disorder, depressed subtype, based on SCID interview, (b) no medication changes in the previous month, (c) clinical stability, (d) age 18–65 years, (e) no substance dependence in the past 6 months, (f) no substance abuse in the past month, (g) no neurological disease or head injury, and (h) no substance use or excessive alcohol consumption in the days prior to testing or no excessive lifetime alcohol or substance use.
The community sample included representative numbers of persons according to gender, race/ethnicity, and educational attainment based on the 2000 U.S. Census. Inclusion criteria were: (a) no history of diagnosis of schizophrenia or other psychotic disorder, (b) no neurological disease or head injury, (c) no mental retardation or pervasive developmental disorder, (d) not currently taking any medications that may interfere with test performance (e.g., narcotics for pain), and (e) no recent alcohol or substance use or excessive lifetime alcohol consumption or substance use. Table 1 presents the demographic characteristics of the schizophrenia and community resident samples.
All study participants were administered the ten tests that comprise the MCCB. Persons in the schizophrenia sample were administered a beta version that included the ten MCCB tests that were interspersed with ten other cognitive tests that were candidates for inclusion in the final battery. Community residents received only the ten tests that made up the final MCCB. Hence, the groups differed by the number of tests they were administered and test order. Table 2 includes the ten MCCB tests, according to domain, and their corresponding dependent measures. Schizophrenia participants were also administered the Brief Psychiatric Rating Scale (BPRS) (Lukoff et al., 1986) for assessment of psychiatric symptoms and the Birchwood Social Functioning Scale (SFS) (Birchwood et al., 1990) supplemented by sections from the Social Adjustment Scale (SAS) for assessment of community and work functioning (Weissman and Paykel, 1974). Employment status was determined from the SAS which distinguishes regular paid work from assisted work (e.g., job coach), supported work (e.g., sheltered workshop), volunteer work, and no work related activities based on activity over the past three months. Our analyses focused on distinguishing regular paid work from the other categories.
Statistical analyses were performed using SPSS 17.0. For the MCCB impairment profile, patients' raw scores from each of the ten MCCB tests were entered into the MCCB scoring program to produce age- and gender-corrected T-scores for the seven cognitive domains (normative mean = 50; standard deviation = 10) 1. These descriptive data allow for a straightforward interpretation of severity of cognitive impairment against norms representative of the demographic make-up of persons in the U.S. as described in the 2000 Census Report. To test whether the patient profile was significantly different from that of community residents, the data were analyzed using a one-way MANOVA with group (schizophrenia participants vs. community residents) as the between-subjects variable and age- and gender-corrected T-scores from the seven cognitive domains as the dependent variables. Follow-up contrasts (parallel to two sample t - tests) were conducted on each of the respective cognitive domains. Within the schizophrenia group, additional follow-up contrasts were conducted to determine whether there were any domains of relative strength or weakness by comparing the average performance level on each cognitive factor with the mean of the remaining six.
To examine which combination of cognitive domains and corresponding cut-off scores best distinguished persons with schizophrenia from community residents, data were analyzed using classification and regression trees (CART) (Breiman et al., 1998). CART is a data mining technique that derives decision trees to predict values of continuous (regression) or categorical (classification) outcomes from a set of predictor variables. In contrast to traditional regression or discriminant function analyses, CART is non-parametric and does not assume a monotonic relationship between predictors and outcomes. Trees branch and grow iteratively by identifying optimal cut-points for key discriminating variables in the predictor set. For classification problems, the tree grows until a stopping criterion is met or no further improvement in correct classification of study participants is possible. With this distribution-free and flexible approach, CART can often represent a complex set of overlapping predictor variables and achieve good classification with a few simple “if-then” rules. CART was also applied to address classification of persons with schizophrenia on vocational status. For both analyses, we used uncorrected T-scores to allow examination of age and gender as separate predictors. The predictor variables for these analyses included T-scores for the seven MCCB domains plus the demographic variables age and gender. We also examined each MCCB domain for their overall importance/centrality to the prediction model. This measure provides information on the robustness of selected variables to serve as a proxy for other measures within the tree.
The age- and gender-corrected MCCB cognitive profile of the schizophrenia group is illustrated in Figure 1. T-score means ranged from 33.4 for processing speed (greatest impairment) to 39.3 for reasoning and problem-solving (least impairment). The MANOVA results revealed an overall effect of diagnostic status with the schizophrenia group significantly impaired relative to the community resident group (F(7,448) = 56.04; p < .001). Schizophrenia participants showed significant impairment relative to community residents on each of the seven MCCB cognitive domains (all ps < .001). Within the schizophrenia group, the outlier analyses that compared each cognitive domain with the mean of the remaining six revealed speed of processing and working memory to be most impaired (t(167) = 6.241; p < .001; t(167) = 2.302; p = .023, respectively) and reasoning and problem-solving to be least impaired (t(167) = 4.384; p < .001).
The CART results for discrimination of persons with schizophrenia from community residents are presented in Figure 2 along with the independent variable importance for each of seven MCCB domains. A cut-off T-score of 43.9 (27th percentile) on speed of processing splits the sample into two groups, one of which is 87.6% community residents and the other of which is 66.4% persons with schizophrenia. Both these groups are better discriminated than the original sample which was 63% (n=300) controls and 37% (n=176) patients. Viewed as a classification rule, the new model correctly identifies 81.8% (144/176) patients and 75.7% (227/300) controls or 77.9% of participants overall. Discrimination was further improved with the additional consideration of social cognition performance which resulted in one node that was 92.4% community residents; another node that was 79.4% schizophrenia patients and two nodes that were relatively evenly split. Group discrimination was not further improved by other MATRICS scores or the demographic variables of age and gender. In sum, persons with schizophrenia were best distinguished from community residents based on their processing speed which was further improved with consideration of level of social cognition performance.
Figure 3 presents the CART results for discriminating patients who were in competitive employment from those who were not as well as the independent variable importance for each of the seven MCCB domain predictor variables included in the model. A cut-off T-score of 30.5 (2nd percentile) on speed of processing splits the sample into two groups, one of which is 89.4% non-workers and 34.5% workers. Both these groups are better discriminated than the original sample of patients which included 74.4% (n=131) non-workers and 25.6% (n=45) workers. Viewed as a classification rule, the new model correctly identifies 45.0% (59/131) of non-workers and 84.4% (38/45) of workers or 55.1% of patients overall. Discrimination was further improved with the additional consideration of visual learning and attention/vigilance. On the left hand side of the tree, a cut-off T-score of 43.0 (24th percentile) on the BVMT-R (visual learning) resulted in one node that was 94.6% non-workers. On the right hand side, a cut-off T-score of 47.9 (42nd percentile) on the CPT-IP (attention/vigilance) resulted in one node that was 50.0% workers. Discrimination was not further improved by other MATRICS scores or the demographic variables of age and gender. In sum, these findings indicate that marked impairments in processing speed best distinguished non-workers from workers, and the additional consideration of visual learning ability further discriminated the two. In contrast, workers were less clearly distinguished from non-workers by level of cognition.
These findings are the first reported on the MCCB impairment profile of schizophrenia outpatients from the sample used in developing this test battery. As such, they provide a standard of comparison for future studies examining the profile of cognitive impairment in other schizophrenia samples and comparison groups as well as clinical trials testing the efficacy of cognition-enhancing drugs. In interpreting these data, it is important to note that the patient and community resident samples included in MCCB development were designed to be broadly representative of adults in the U.S. and included men and women from a wide range of age and education, differing racial and ethnic backgrounds, and urban and rural settings. Also, a key feature in developing the MCCB was the co-norming of tests in the final battery. Although a number of the individual tests that comprise the MCCB are widely used neuropsychological tests with normative data, this combination of tests had never been administered together as a single unit and co-normed in a representative community sample. It should also be noted that the component tests of the MCCB were selected with a focus on their use within the context of clinical trials and were evaluated based on test-retest reliability, utility as a repeated measure, relationship to functional outcome, practicality and tolerability, and more broadly sensitivity to change. Hence, other measures might be more ideal to characterize the cognitive impairments of schizophrenia outside of this context.
Results from the cognitive profile analyses revealed schizophrenia patients to be impaired relative to community residents on each of the seven MCCB cognitive domains. The range of impairment across domains was relatively compact with the breadth of impairment severity covered by six-tenths of a standard deviation. Of the seven MCCB cognitive domains, speed of processing and working memory were most impaired (4th and 7th percentiles, respectively). Greater impairment observed in these domains may be associated with the fact that they were measured using multiple tests and may have greater reliability relative to other MCCB domains assessed by a single test. In terms of generalizability of these findings to other schizophrenia samples, a Norwegian study (Holmen et al., 2010) of early onset schizophrenia spectrum disorder patients found a z-score impairment range of .8 to 1.8 on the MCCB with the curious exception of social cognition that was found to be equivalent with the study's normal comparison group. It should be noted that z-scores were derived relative to the study's normal control group not the MCCB normative sample. At the other end of the age spectrum, it is not known if these profile findings apply to older patient samples.
The findings for speed of processing are in keeping with independent investigations and meta-analyses examining a commonly used measure of processing speed, digit symbol type tests. In a study of 127 schizophrenia patients and 127 demographically matched controls (Palmer et al., 2010), processing speed measured by digit symbol and symbol search tests was found to show greater levels of impairment than cognitive measures of verbal comprehension, perceptual organization, working memory, and auditory memory. Similarly, in a meta-analysis of 37 studies, the digit symbol test was found to yield a large mean effect size (g = 1.57) and the magnitude was greater than that found for measures of episodic memory, working memory, and executive functioning (Dickinson et al., 2007). In a separate meta-analysis of 43 first episode schizophrenia samples, Cohen's d was 1.59 for the digit symbol test, remarkably consistent with that found in more chronic samples (Mesholam-Gately et al., 2009). In the present analyses, Cohen's d for the BACS symbol coding test was 1.35. For working memory, meta-analyses and reviews (Aleman et al., 1999; Heinrichs and Zakzanis, 1998; Lee and Park, 2005) indicate a mean effect size ranging between .45 to .82 which are considerably lower than we found. However, a number of studies in the meta-analyses examined experimental tasks of working memory that may have had lower task difficulty levels than the ones included in the MCCB. A recent memory study of chronic schizophrenia and schizoaffective disorder outpatients that included the MCCB measure of working memory, Letter-Number Span, reported more comparable levels of impairment (Cohen's d = 1.17) (Kern et al., 2010).
Although processing speed and working memory were found to be the most impaired areas of cognition relative to other domains, these findings should be interpreted in the context of the method of measurement used to derive T-scores by the MCCB scoring program. Processing speed and working memory are the only MCCB domains assessed by more than one test. Including multiple tests to assess a particular cognitive domain improves reliability and ability to capture the breadth of a construct. However, a form of measurement bias can occur when comparisons are made with domains assessed by a single test. This is because T-scores for domains which include multiple tests are renormalized versions of the sums of T-scores from the component tests. This ensures that the domain scores have a common metric in the sense that their means and standard deviations in the normative sample are the same regardless of the number of component tests. However, the resulting patients' T-scores for domains assessed by multiple tests tend to be lower than those assessed by a single test. This measurement artifact occurs for the following reason. If the component tests are perfectly correlated, the renormalization results in the mean difference between the patient and control domain scores being the average of the group differences on the individual tests. Otherwise, the group difference in the domain score is adjusted by a factor that increases as a) the correlations between the component tests decrease and b) the number of tests increases. Intuitively, this is because when the correlations are imperfect, each test contributes additional unique variance in defining between group differences. As a group, schizophrenia patients generally perform worse than the normative sample on the individual MCCB tests, and the correlations between tests within a given domain (e.g., speed of processing) are less than 1.0 since the tests were designed to capture different facets of functioning. As a result, the combined patient and control group score distributions shift further apart with the inclusion of each additional test, and interpretation of findings between MCCB domains must include consideration of this measurement artifact. However, pertinent to interpretation of the present findings, it is noteworthy that the mean T-scores for the individual tests most representative of speed of processing and working memory were only slightly higher (showing less impairment) than the T-score composites for the corresponding domain (BACS symbol coding = 35.2 (12.0) vs. speed of processing = 33.4 (11.9); Letter-Number Span = 36.7 (12.0) vs. working memory = 35.4 (12.1)).
Speed of processing and social cognition best distinguished persons with schizophrenia from community residents. The finding for speed of processing is not surprising. Symbol coding measures are among the most sensitive to detection of cognitive dysfunction (Lezak, 1995; Wechsler, 2008). These measures have a longstanding history in detecting early signs of cognitive impairment across a wide range of neurological disorders (Storandt and Hill, 1989; Strauss and Brandt, 1986). The test places demands on cognitive processes involved in sustained attention, working memory, graphomotor speed, as well as strategy formation (Glosser et al., 1977). One hypothesis about its sensitivity as a measure of cognitive impairment is based upon the number of cognitive processes involved in task performance. That is, because the test is polyfactorial, its sensitivity may be increased relative to measures that involve fewer performance limiting processes. Patient vs. community resident discrimination was further improved by the inclusion of a measure of social cognition, the MSCEIT Managing Emotions branch. Though this test assesses a narrow element of social cognition, perhaps impairment in this area is particularly central to schizophrenia. Alternatively, the MSCEIT's unique algorithm-based scoring method, which differs from other MCCB tests, may have contributed to its importance as a discriminator.
Speed of processing, visual learning, and attention/vigilance contributed to distinguishing persons with schizophrenia who were competitively employed vs. those who were not. Interestingly, the combination of cognitive domains and cut-off scores that identified workers differed from those that identified non-workers. Marked impairments in processing speed (below the 2nd percentile) along with impairments in visual learning ability best distinguished non-workers from workers. In contrast, level of cognitive functioning appeared less critical to distinguishing workers from non-workers. Other studies have also found processing speed, learning and memory, and attention to be related to employment status with a possible role for executive functioning as well (Bellack et al., 1999; Bryson and Bell, 2003; Evans et al., 2004; Gold et al., 2002; Lysaker et al., 2005; Milev et al., 2005), and the findings for processing speed may extend more broadly to prediction of a number of areas of functional outcome (Harvey et al., 2009). Bellack et al. (1999) found persons with schizophrenia with good vocational histories (GVH) to perform better from those with poor vocational histories (PVH) on a broad array of cognitive tests assessing processing speed, learning and memory, attention, and executive functioning, as well as general intelligence. Classification of GVH vs. PVH using discriminant analyses revealed that cognitive measures that best identified GVH differed from those for PVH, a finding similar to ours. Based on these findings, it appears easier to identify cognitive determinants of unemployment than it is to identify them for employment.
The authors are grateful to the study participants for their time and effort devoted to participation in this study; and we wish to thank the research staff at the five performance sites for their work in recruitment, testing, scoring, and data management. We thank Kellie M. Smith, M.A. for her assistance in the preparation of the manuscript.
Role of funding source
Funding for the MATRICS Initiative was provided through Contract N01MH22006 from the National Institute for Mental Health to the University of California, Los Angeles (Dr. Marder, PI; Dr. Green, Co-PI; Dr. Fenton, Project Officer). Funding for this study came from an Option (Dr. Green, PI; Dr. Nuechterlein, Co-PI) to the NIMH MATRICS Initiative. The NIMH had no further role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Conflict of interest
Author Kern is an officer for MATRICS Assessment, Inc. and receives financial compensation for his role within the non-profit organization; Authors Green and Nuechterlein are officers within MATRICS Assessment, Inc. but do not receive any financial remuneration for their respective roles. No other authors have any conflicts of interest with respect to this manuscript.
Authors Green, Nuechterlein, and Marder designed the study and wrote the protocol. Authors Sugar, Lee, and Kern performed the data analyses. Author Kern wrote the first draft of the manuscript with contributions from authors Gold and Dickinson. All authors contributed to and have approved the final manuscript.
1Note: The MCCB scoring program uses a regression-based approach for determining ageand gender-corrected T-scores based on a linear effects model. The MCCB scoring program will produce T-scores for individuals outside the age range of the normative sample by extending these linear age effects. For small extension beyond the normative age range, this modeling is likely still appropriate. However, we caution assumption of this linear correction for ages far outside the normative range.