We found that 3 tests—TMT B, category fluency, and digit symbol—correlated .86 with the larger FES and CHOR batteries and correlated between .73 and .82 with the total scores from the batteries excluding the B-CATS tests. The correlations of B-CATS with the larger battery total scores are within the range of the typical test-retest correlation coefficients for these measures. The tests themselves have small-to-moderate inter-item correlations ranging from .27 to .48 across both the FES and the CHOR batteries. Thus, the B-CATS should provide a valid estimate of general cognitive ability. The B-CATS has an expected administration time of 10–11 min in contrast to the full batteries of the FES, CHOR, and CATIE studies, which have administration times ranging from 90 to 210 min.
Our decision to examine the correlation of the B-CATS with the total score from the full neuropsychological battery and the battery excluding the B-CATS tests serves as both an undercorrection and an overcorrection of the problem of part-whole correlation. The actual correlation of the B-CATS with the full batteries is likely somewhere in between. An ongoing validity study (see below) will address this issue more fully by comparing the B-CATS with a separately administered neuropsychological battery.
The alternate B-CATS composed of category fluency, digit symbol, and letter-number sequencing calculated for comparison to the CATIE battery also correlated very highly with the global score, although the estimated administration time is 1.5–2.5 min longer. This suggests that other combinations of brief tests may also correlate highly with the total score of a larger battery. The tests we chose are not unique in capturing a large proportion of the variance of the total score, although they are 3 especially brief tests. But if other B-CATS were needed—for computer-based administration, for targeting of a specific domain, for emphasis on the highest correlation between global scores, with less regard to administration time—similar approaches could construct multiple combinations of tests for use as BCA batteries.
A combination approach could also be used combining expert input with empirical derivation. For instance, if the goal in constructing the B-CATS had been domain diversity, instead of brevity, only tests from different domains might have been considered for inclusion. The construction of the above battery also involved expert opinion. For instance, the choice of the VPM as a criterion for test inclusion was based on our opinion that it provided the best measure of efficiency per test. Our decision to include 3 tests, instead of 1 or 2, was based on our belief that 3 tests were “better.” This was because 3 tests correlated more highly with the global cognitive score than did 1 or 2 tests while remaining within the desired administration time. If we had gone strictly by VPM, we would have chosen only 1 test because VPM dropped as the number of tests were added.
Because our goal in this instance was to design an instrument for clinical use, the administration time of the B-CATS was a crucial consideration. Despite the fact that over 20 years of research have demonstrated pervasive and profound cognitive deficits in schizophrenia,17
clinicians remain unable to measure cognition in their patients. And while neuropsychological evaluation is available to some, the need to refer patients to neuropsychologists, and the cost of comprehensive neuropsychological testing, frequently results in a lack of any form of cognitive assessment. Those patients may therefore benefit from a more integrated brief assessment within their regular appointments. A typical outpatient psychiatric practice allots 15–20 min for medication management appointments. An administration time of even 15 min would likely leave clinicians without enough time to enquire about symptoms status and overall functioning, medication compliance, and side effects and monitor for the potential metabolic and movement side effects of antipsychotic agents. Our tool is estimated to take between 10 and 11 min. Furthermore, the administration time may be shortened by reducing category fluency to animal fluency alone (eliminating 2 min of testing time). Both the CHOR and the FES studies included only animal fluency under category fluency. The effect of including only animal fluency is being evaluated during our ongoing validity study. Furthermore, a web-based program (see below) for scoring and interpretation of the results will reduce evaluation time and increase usability of the tool.
The B-CATS can also provide researchers with an approximate global cognition score without requiring a full neuropsychological battery. While a brief battery such as the B-CATS is not a substitute for a comprehensive neuropsychological battery, many clinical trials that do not include cognition as a primary target may still gain value from an estimate of global cognition that is brief and easily administered. The B-CATS can provide an estimate of general cognitive ability in about 10 min, saving significant resources and reducing testing strain on subjects. Furthermore, several pharmacological agents with specific cognitive domain targets (for a review of such potential targets, see Tamminga18
) are currently under development. Testing of these agents will require comprehensive assessment of the domain of interest. The B-CATS could provide a brief but sufficient estimate of general cognitive ability, freeing time and resources for a more extensive assessment of the domain or domains of interest.
The B-CATS is not designed to assess cognitive function at the domain level. Currently, there is debate in the field about the pattern of cognitive deficits in schizophrenia. In particular, whether performance on all cognitive domains predicts general cognitive function in aggregate or whether generalized cognitive ability hierarchically informs the performance on subdomains of cognition. While some research and expert opinion support a domain model of cognitive deficit in schizophrenia,7,19,20
recent factor and structural equation analyses from large studies with comprehensive neuropsychological batteries9,16,21–23
have demonstrated that a single factor of global cognitive function better explains the deficit pattern of schizophrenia (one of these analyses was conducted on the CATIE data set used in this study16
). Specifically, these studies have shown that the best fit for the neurocognitive performance data of people with schizophrenia is either a 1-factor model where the factor of global cognitive ability explains the majority of the variance in performance on the specific tests included in the battery without significant input from cognitive subdomains21,23
or the global factor hierarchically influences performance on the subdomains, which in turn predict performance on the individual tests in the battery.9,16,22
The measurement of the global factor over more specific domain assessment has functional significance as well. In a 2000 meta-analysis, Green et al24
concluded that global cognition scores (vs individual tests or domains) correlate most highly with measures of functional outcome.
The B-CATS uses 3 existing neuropsychological tests, TMT B, digit symbol, and category fluency. All 3 tests require participation from multiple cognitive domains. For instance, digit symbol requires the contribution of motor and processing speed,25,26
and learning and memory.25,26
Category fluency utilizes, among other areas, verbal fluency and language skills,28,29
processing speed, and various memory processes, including verbal memory and semantic organization.30,31
TMT B is a test that uses a complex series of cognitive skills including set shifting,32
executive function and working memory,33
motor and processing speed,32
and visuospatial scanning14,29,32
(not all involved domains may be assessable, however, if trails B is scored only for administration time and not for errors33
). Despite the interaction of skills from multiple cognitive domains, currently, many researchers consider digit symbol and category fluency to be measures of processing speed,7,19,22
and generally in factor analyses, trails B loads on the processing speed domain.19
Processing speed tests (or rather tests assigned to the domain of processing speed) may be particularly well suited for capturing the cognitive deficits associated with schizophrenia. A recent meta-analysis35
suggests that when comparing the performance of patients with schizophrenia with comparison subjects, digit symbol substitution (and to a lesser degree category fluency) has a cumulative effect size of 1½ times the global across-domain effect size (Hedges g
= −1.57 for digit symbol, −1.41 for category fluency, and −0.98 for global effect size). Furthermore, there is 73% nonoverlap in scores between patients with schizophrenia and healthy control subjects vs 55% nonoverlap for the global effect size. The effect size of digit symbol performance in relatives of people with schizophrenia was also larger than that of the global effect size, suggesting that digit symbol may be a marker of a potential cognitive endophenotype. Another study demonstrated that after controlling for digit symbol, no further significant differences in cognitive functioning existed between subjects with schizophrenia and control subjects.36
Finally, 2 recent studies demonstrated that processing speed deficits are the most highly correlated with community functioning, after controlling for general cognitive ability,37
or the most broadly and directly correlated with several measures of functional capacity and outcome.20
These recent data support the use of a processing speed–based measure for screening and assessment of general cognitive function. However, the domain homogeneity of the B-CATS is a limitation, and different derivations of B-CATS could emphasize domain diversity (although that would likely lengthen administration time).
The BCA, constructed by Velligan et al,14
is an almost identical group of tests, composed of verbal fluency (letters and categories), trails A and B, and Hopkins verbal learning test (HVLT). The BCA correlates .72 with a more comprehensive neuropsychological battery, although without controlling for part-whole correlation. However, the inclusion of both letters (FAS) and categories (animal), and the use of the HVLT, which is a series of 3 trials of 12 words, increases administration time to 15 min or more. The B-CATS is shorter and was derived empirically. On the other hand, the BCA evaluates verbal memory, a cognitive domain greatly affected by schizophrenia (although a recent factor analysis suggests that verbal memory deficits are not separable from the generalized cognitive deficit23
). Finally, the BCA correlates in the moderate range with several measures of functional outcome.14
The similarity of the B-CATS to the BCA provides empirical support for the expert consensus that the included tests capture a high degree of variance in a comprehensive battery’s total score.
While our goal was not to derive the best possible brief battery, we believe that the B-CATS is a reasonable choice for cognitive assessment by clinicians, based on its strong correlation with the global cognitive scores from full neuropsychological batteries and its brevity and ease of administration. We plan to make it or some version of it available for clinicians (and researchers). More work is required, however, before the B-CATS is ready for clinical use. We are currently validating the B-CATS in a sample of 100 subjects with schizophrenia. We are comparing performance on the B-CATS to the global score on the MATRICS consortium consensus battery (MCCB) at 2 time points (to assess test-retest reliability and practice effects). We are also correlating the scores of the B-CATS and the MCCB to performance on a measure of functional capacity, the University of California at San Diego performance-based skills assessment. The administrators of the B-CATS in this study are clinicians and research assistants with limited to no training in psychometrics to model the real-world goal of administration of the B-CATS by clinicians without psychometric training.
We are also constructing a website to simplify administration and scoring of the B-CATS for clinicians. Ideally, the website will provide some structure to assist clinicians without psychometric experience administer and score the tests correctly. Despite the great need for a clinical tool to assess cognition in schizophrenia, there is concern that clinicians without formal training in psychometrics may administer the battery in such a way that it loses interpretability. However, a benefit of the B-CATS is that the test instructions are quite simple. By providing users with access to the B-CATS website, we hope to improve reliability by clinician administrators. The website will allow clinicians to download and print test instructions and alternate versions of the tests, and the site will provide a template to enter age, gender, education, and scores to obtain a normed B-CATS score. Clinicians will also be able to use the website to ask questions about test administration and scoring and watch a brief video demonstration of the correct administration of the B-CATS.
In conclusion, the B-CATS is a brief measure composed of existing cognitive tests that provides a global cognition score. It is highly correlated with the global scores from comprehensive neuropsychological batteries that take between 6 and 20 times as long to administer. The B-CATS has been developed specifically for use by clinicians, and the 3 tests have straightforward and easy instructions for administration and scoring. Researchers may also find it a useful tool for obtaining a fast estimate of global cognitive ability that does not require administration by a psychometrist.