Deficits in the performance of critical everyday functional skills, including social and occupational functioning, residential maintenance, medication management, and basic self-care, are present in many neuropsychiatric conditions.1
These impairments are particularly salient in schizophrenia.2
Disability in schizophrenia occurs even following successful treatment of the clinical symptoms of the illness3
and often sets in immediately after the first episode.4
Throughout the course of the illness, the majority of schizophrenic patients experience some form of impairment in everyday functioning, whether in employment, independent living, or social functioning.5
As a result, disability reduction has the potential to benefit nearly every patient with schizophrenia, yet current treatments for the illness are notably ineffective at reducing disability.6
While disability in schizophrenia appears to be related to the failure to perform critical functions in the real world, this disability is likely caused by multiple factors. Failure to perform may be due to skill deficits, motivational deficits, interfering symptoms, and/or limited opportunities or personal resources.7
Thus, what one does in the real world may not be the perfect index of what one can do, but what one can do under optimal conditions is likely an index of maximal real-world (RW) potential.
We argue that RW functioning is just one element of a more global functional outcomes construct. Factors that influence potential, such as cognitive impairments indexed by neuropsychological (NP) test scores and functional capacity (FC; ie, ability or competence in the performance of everyday living skills), as well as other individual differences such demographic factors and symptoms, including positive, negative, and depressive symptoms, have been shown to predict individuals’ RW functioning in schizophrenia.8
However, reports of RW outcomes vary across informants and contain elements of error, which can be indexed as well. Even “objective” milestones such as employment and marriage are influenced by measurable factors other than ability, such as opportunities and societal incentives and disincentives, and they are often reported inconsistently across informants as described below. Thus, each of the elements of the functional outcomes construct is measured by error-laden indices, and there is no “strict operational” definition of what “RW functional outcome” is.
Arguably, the most consistent element of the functional outcomes construct is NP performance, as measured in recent treatment studies by the MATRICS Consensus Cognitive Battery (MCCB)8
. The battery was developed through expert nominations from the field and a RAND Appropriateness Panel to select measures in several domains for subsequent comparison in a formal psychometric study.9–11
The final consensus battery consists of 10 neuropsychological tests and a measure of social cognition, which met comprehensive standards for criterion-referenced validity and test-retest reliability.
Several recent studies have highlighted the variability in convergence between NP, FC, and RW performance measures. In particular, these studies11–15
examined these convergences in patients with schizophrenia using the University of California San Diego Performance-Based Skills Assessment (UPSA) as the measure of FC. Interestingly, despite the use of different NP performance measures in each of the studies, the correlation between NP performance and the total UPSA score was remarkably consistent, ranging from r
0.60 to r
0.65. Across the same studies, however, the correlation between RW outcomes and UPSA performance varied considerably, ranging from r
0.04 to r
0.50. The studies also showed considerable variance in the correlation between NP performance and RW outcomes: r
0.05 to r
0.54. The lowest correlations were found in studies using only self-report of RW outcomes, and the highest correlation, for both domains, came from a study where the RW outcome used was residential independence measured with a comprehensive assessment involving multiple sources of information. These data suggest 2 conclusions: First, performance-based measures of NP performance and FC are highly convergent with each other regardless of the NP battery employed; and second, different RW outcome measures yield widely variant correlations with corresponding performance-based measures. Because the correlations between the performance-based NP and FC measures were so consistent, the variation in correlations with RW measures and these other domains implicates shortcomings of the RW outcomes measures.
The overriding goal in treating cognitive deficits in schizophrenia is reducing functional disability. However, if the overlap between NP performance, even if measured with a highly reliable and valid assessment battery, and RW outcome is as small as it initially appears, the question remains as to whether successful treatment of cognition can realistically improve RW outcomes. One suggestion, explored below, is that current instruments assessing RW outcomes exhibit intrinsic limitations, at least when in the hands of certain informants. The most global and arguably most significant aspects of RW outcome can be measured with high reliability, admittedly with certain limitations. These include marriage or an equally stable relationship, full-time employment, and self-supported living. However, these outcomes are rare and develop over time; hence, they are impractical for use as outcomes variables in treatment studies, even for trials of treatment effectiveness. Measurement of more subtle aspects of RW outcome in neuropsychiatric conditions (eg, household management, social contacts, and job seeking activities) is rarely direct, and in many research studies, these aspects are often measured through self-report. However, recent research has shown that self-reports by patients with schizophrenia may be unreliable when compared with other sources of information; schizophrenia seems to induce types and degrees of self-report deficits that exceed those of the general population.16
Patients with schizophrenia manifested substantial problems in self-reporting their cognitive impairments, when examined on a structured rating scale that was then related to their performance on an NP assessment.17
Further, the convergence of case manager reports and patient self-reports, even of supposedly objective outcomes such as living situation and time spent working in the past week has been found to be minimal, accounting for as little as 4% of joint variance.18
Patients’ self-report of their functioning in that same study was not as strongly associated with their performance on NP and FC measures than were case manager reports, suggesting that these case manager reports have evidence of more validity than patients self-report.
The modifiable sources of reduced validity for rating RW outcomes are at least 2-fold: first, the characteristics of the informant used and, second, the RW outcome rating scale selected. Variance in reports by informants can be influenced by the amount of contact with the subject and situation specificity of the observation. In the case of self-report, the variation can be influenced by patients’ competence in self-evaluation of the quality as well as the quantity of their performance (see Bowie et al18
for an example of this). It is entirely possible that a substantially greater correlation exists between NP performance and aspects of RW outcomes than has been detected in previous studies where the RW outcome measures may have been deficient. For example, in the Twamley et al15
study, the RW outcome was based on a comprehensive assessment of residential independence, and the correlation between NP performance and this outcome was the highest for any of the studies cited above. Therefore, the next step in the construct validation process would be to evaluate candidate measures of RW outcomes with rigorous process similar to the selection of the MCCB.