|Home | About | Journals | Submit | Contact Us | Français|
Cognitive deficits are associated with disability in people with schizophrenia so treatment of cognitive impairment has been proposed as an intervention to reduce disability. However, studies relying on patient self-report have found very minimal relationships between ratings of real-world functioning and cognitive performance, raising questions about the measurement of real-world functioning as a treatment outcome. The Validation of Everyday Real-world Outcomes (VALERO) study was conducted to evaluate functional rating scales and to identify the rating scale or scales most robustly related to performance-based measures of cognition and everyday living skills.
198 adults with schizophrenia were tested with the neurocognitive measures from the MATRICS Consensus cognitive Battery and performed the UCSD performance-based skills assessment-Brief and advanced finances subtest from the Everyday Functioning Battery. They and a friend, relative, clinician, or case manager also reported their everyday functioning on 6 ratings scales: Social Behavior Schedule, Social Adjustment Scale, Heinrichs Carpenter Quality of Life Scale, Specific Levels of Functioning, Independent Living skills Survey, and Life Skills Profile. Best judgment ratings were generated by an interviewer who administered the rating scales to patients and informants.
Statistical analyses developed an ability latent trait that reflected scores on the three performance-based (i.e., ability) measures and canonical correlation analysis related interviewer ratings to the latent trait. The overall fit of the model with all six rating scales was good: χ2 = 78.100, df = 56, p = .027, and RMSEA = .078. Individual rating scales that did not improve the fit of the model were systematically deleted and a final model with two rating scales fit the data: χ2 = 32.059, df = 24, p = .126, RMSEA = .072. A regression analysis found that the Specific Levels of Functioning was a superior predictor of the three-performance based ability measures.
We found that systematic assessments of real world functioning are related to performance on neurocognitive and functional capacity measures. Of the six rating scales evaluated, the Specific Levels of Functioning (SLOF) was best in this study. Use of a single rating scale provides a very efficient assessment of real-world functioning that accounts for considerable variance in performance-based scores.
Disability in multiple everyday functional domains is common in people with schizophrenia , encompassing social , vocational , and residential  domains. These impairments are present in the majority of patients and are refractory to current pharmacological interventions  and only partially responsive to rehabilitation . Some of the patient characteristics associated with these impairments include cognitive deficits , negative symptoms , and impairments in performance of social and daily living skills (i.e., “functional capacity” ). There are also environmental contributions to real-world functional deficits, which include disability compensation (associated with reduced vocational achievement ), and lack of social opportunities and community resources). Disability, as a consequence, is a complex and multiply determined phenomenon which may have similarities across different severe mental illnesses .
There have been considerable efforts to treat the putative causes of these functional deficits, including pharmacological interventions aimed at cognitive enhancement and reduction of negative symptoms , cognitive remediation interventions , supported employment  and various psychosocial interventions, including efforts to teach social and everyday living skills . These efforts have met with variable success, with combined interventions including cognitive remediation and psychosocial interventions yielding the most success . However, several studies have found that impairments on a variety of ability measures, including both neuropsychological tests and measures of functional capacity, were weakly related to ratings of real-world functioning [15–17]. These relationships raise the legitimate question as to whether improving cognitive or functional abilities have the potential to exert a meaningful influence on real-world outcomes.
As we have suggested before , many studies finding low correlations between ability or capacity measures and ratings of real world outcomes used self-reports of real-world functioning. As shown in several systematic comparisons, patient self reports of both ability (e.g., cognitive performance) and real-world functioning are remarkably unrelated to performance on performance-based ability measures and to informant ratings of ability and real-world functioning. As a result of these findings, the Validation of Everyday Real-World Outcomes (VALERO) initiative was undertaken. The current study in the VALERO initiative is aimed at a direct comparison of the relationship of a set of instruments for rating of real-world functional outcome, by informants or patients themselves, and the ability measures, including both cognitive and functional capacity assessments of people with schizophrenia.
In the present study, six rating scales assessing real-world functional outcomes, previously recommended as the best available by a RAND expert panel, were completed (a) by patients with schizophrenia, (b) an informant (friend, relative, or high-contact clinician), and (c) by a research examiner who conducted the interviews with the patient and informant. The interviewer’s ratings reflected her best estimate of the level of functioning of the patient. Examiners were instructed to base all judgments on what they thought was correct, including discounting any information that they believed was inaccurate. These research examiner ratings were then related to the patient’s performance on the MATRICS Consensus Cognitive Battery (MCCB ) and two functional capacity measures, the UCSD Performance-based Skills Assessment-Brief Version (UPSA-B ) and the Advanced Finances Subscale of the Everyday Functioning Battery (EFB ). The UPSA-B is widely used in studies of functional capacity in schizophrenia, but the EFB is aimed at higher functioning individuals and was included in order to avoid the possibility of ceiling effects in the assessment of higher functioning patients. The goal of this study was to identify the rating scale or scales that measure real-world functioning most strongly correlated with patients’ performance on measures of their ability: cognitive performance and functional capacity. Using statistical techniques described in the on-line supplement, we examined the 6 different scales and found the one most strongly correlated with the performance-based measures.
The study participants were patients with schizophrenia who were receiving treatment at one of three different outpatient service delivery systems, two in Atlanta and one in San Diego. In addition, informants were interviewed concerning the everyday functioning of each of the patients, with these informants either being a high-contact clinician (case manager, psychiatrist, therapist, or residential facility manager; 20% of cases) or a friend or relative (80% of cases). All of these research participants provided signed, informed consent, and this research study was approved by local IRBs in Atlanta and San Diego. In Atlanta, patients were either recruited at a psychiatric rehabilitation program (Skyland Trail) or from the general outpatient population of the Atlanta VA Medical Center. The Skyland Trail patients were receiving treatment because of functional disability, including impairments in both residential and vocational functioning, and these patients were recruited through their case managers. The Atlanta VA Medical Center patients were not selected for the presence of disability and were recruited through advertisements, from another research project, and by word of mouth. The San Diego patients were recruited from the UCSD Outpatient Psychiatric Services clinic, a large public mental health clinic, and other local community clinics and by word of mouth.
All patients were administered a structured diagnostic interview, either the Structured Clinical Interview for the DSM (SCID;  administered at the Atlanta sites) or the Mini International Neuropsychiatric Interview, 6th Edition (MINI  administered at the San Diego site) by a trained interviewer. All diagnoses were subjected to a consensus procedure at each local site. Patients were excluded for a history of traumatic brain injury with unconsciousness >10 minutes, brain disease such as seizure disorder or neurodegenerative condition, or the presence of another DSM-IV diagnosis that would exclude the diagnosis of schizophrenia. None of the patients were experiencing their first psychiatric admission. Comorbid substance use disorders were not an exclusion criterion, in order to capture a broad array of patients, but patients who appeared intoxicated were rescheduled. Inpatients were not recruited, but patients resided in a wide array of unsupported, supported, or supervised residential facilities. Informants were not screened for psychopathology or substance abuse. Descriptive information on patients and informants is presented in Table 1.
All patients were examined with a performance-based assessment of neurocognitive abilities and functional capacity. They also provided self-reports of social, residential, and vocational functioning on six different functional outcomes scales either administered as interviews by a trained rater or completed as a questionnaire. Informants independently completed the same six outcomes scales reporting on the functioning of the patients. The examiner who conducted the interviews with the patient and informant then generated ratings for all six rating scales, based on her impression of the “true” status of the patient. Rating scales were presented in a fixed, counterbalanced order across patients. Clinical ratings of symptoms were also collected for descriptive purposes and are presented in Table 1, along with demographic information.
We examined cognitive performance with a modified version of the MATRICS consensus cognitive battery (MCCB ). For this study, we did not include the social cognition measure, from the MCCB, the Mayer–Salovey–Caruso Emotional Intelligence Test Managing Emotions, because there are several reasons to think that social cognition measures may have a different relationship with everyday outcomes compared to neurocognitive measures. This minor modification of the MCCB would make the results similar to previous work, such as our own [3,11] that did not include social cognition measures. We calculated a composite score, an average of 9 age-corrected T-scores based on the MCCB normative program, as our critical dependent variable.
We administered two different performance-based functional capacity measures. Participants’ functional abilities were assessed using the Brief version of the UCSD Performance-based Skills Assessment (UPSA-B ). The UPSA-B is a measure of functional capacity in which patients are asked to perform everyday tasks related to communication and finances. During the Communication subtest, participants role-play exercises using an unplugged telephone (e.g., emergency call; dialing a number from memory; calling to reschedule a doctor’s appointment). For the Finance subtest, participants count change, read a utility bill, and write and record a check for the bill. The UPSA-B requires approximately 10–15 minutes, and raw scores are converted into a total score ranging from 0–100, with higher scores indicating better functional capacity. We also administered the Advanced Finances subscale of the Everyday Functioning Battery (EFB ), designed to examine financial management in higher functioning individuals. The Advanced Finance test requires individuals to prepare bank deposits and checks to pay bills, maintain a checkbook balance, and organize payments such that a pre-specified amount of money is left available at the end of the task. This instrument was selected because it measures abilities considered important for independent living and at the time the study was planned, we were concerned that younger individuals with schizophrenia might evidence ceiling effects on the UPSA-B. Total scores on the Advances Finances subtest range from 0–13.
As we previously reported , the initial phase of the VALERO study included a RAND panel that selected 6 functional outcomes scales from a much larger group of candidate scales as most suitable for current use at the time of the panel (see  for detailed descriptions of these instruments). These six scales are the Heinrichs-Carpenter Quality of Life Scale (QLS ), Specific Levels of Functioning (SLOF ), Social-Behavior Schedule (SBS ), Social Functioning Scale: (SFS ), Life Skills Profile (LSP ), and the Independent Living Skills Survey (ILSS ).
There are several important features of these functional scales. Two (SBS, SFS) were pure social functioning scales, while two others examined only community functioning (LSP; ILSS). The others (QLS; SLOF) were “hybrid” scales examining social, residential, and vocational outcomes. Of the six scales, 2 were administered as self-report questionnaires (ILSS and SLOF) and the others were administered as interviews using the standard instructions for the scale. Although all of these rating scales have multiple individual subscales, for the purposes of this first report, we examined only total scores. If these scales were used as outcomes measures in a clinical trial, a single, predefined primary outcome would be selected and we were interested in making this information available as straightforwardly as possible.
Some of these instruments were modified by deletion of some subscales following the suggestions of the RAND panel. For instance, the social acceptability and personal care subscales of the SLOF were not used for the total SLOF scores and for the QLS, the intrapsychic foundations subscale was not included in the analyses because it measures deficit (i.e., negative) symptoms. While negative symptoms are known to affect functional outcomes , we were interested in examining the association between performance-based measures and functioning. Similar to our decision to exclude social cognition from the neurocognitive predictor set, we wanted to exclude negative symptoms as an outcome measure.
The primary goal of the data analysis was to find the real world functional outcome scale, or scales, that was most strongly related to the three performance-based indices of functional ability on the part of the patients. This analysis was conducted using the method of robust maximum likelihood parameter estimation, which included all available data and did not assume multivariate normality for observed measures . With this analytic approach, the model is fitted to all available data from all subjects. That is, no subject is dropped from any of the analyses due to not providing data on more than one of the variables in any model.
To accomplish these aims, a structural equation model was developed and fit to the available data from the three sites with the latent variable modeling software Mplus . A single latent trait reflecting the shared variance of the three performance-based “ability” variables was developed using Hierarchical linear modeling. This single trait was then statistically related to examiner-generated total scores on all six of the real-world functional outcomes scales. This overall model with a single ability latent trait and 6 scales as predictors was tested for its goodness of fit with standard indices, including χ2 (chi-square value), degrees of freedom, associated p-value, and root mean square error of approximation (RMSEA). As discussed in the structural equation modeling and confirmatory factor analysis literature , smaller scores on both χ2 and RMSEA are indicators of desirable fit (useful approximation to the analyzed data). Further, shared variance statistics are calculated with an R2 statistic for the shared multiple correlation between real-world functioning measures and the performance-based ability latent trait.
Following the fitting of this model, the real-world outcomes variables were considered for deletion from the model in a sequential order based on the lowest correlations for their loadings on the ability latent trait. After deletion of the real-world functional scale with smallest correlation, the overall model fit was recalculated and if there was room for improvement in the fit, the next scale with a low correlation was deleted. As the dimension of real-world functioning needed to be defined by at least two rating scales, a total of 4 of the 6 scales could be considered for deletion and the final two could be compared with regression analysis for their relative predictability of the ability latent trait.
Demographic, clinical, and treatment characteristics of the patients are presented in Table 1. As can be seen in the table, there were demographic differences between the sites. These included age, education, and racial differences, differences in ethnicity, and in the type of informant. All of the informants at the Atlanta VA were friends or relatives. Further, there were differences in residential status, with the majority of the San Diego and Atlanta VA patients were living in the community; the Skyland Trail patients were more residentially disabled. There were no differences in employment across the samples and the PANNS scores were essentially identical across sites. The scores on the three performance-based measures were very similar across the sites as well, although the Atlanta VA and UCSD patients had somewhat higher UPSA scores. Thus, the demographic differences between the samples and sites did not correspond to major differences in the performance-based measures.
Table 2 presents the overall fit of the complete baseline model with the ability latent trait based on three performance-based indicators and examiner-generated total scores for the 6 real-world functioning scales. As can be seen in the top half of the table, all three performance-based variables were significantly related to the ability latent trait. As seen in the bottom of the table, only one of the 6 rating scales was significantly related to the ability latent trait: the Specific Levels of Functioning in the baseline analyses. Nonetheless, the overall fit of the model still would be considered to be acceptable, in that the RMSEA was small and the p-value close to nonsignificant. However, this model clearly had room for improvement, in that most of the rating scales were not independently related to the performance-based measures.
As a result, we dropped real-world functioning scales in sequence, as presented in Figure 1. The sequence was determined by a rank ordering of the nonsignificant p-values for the factor loadings between the real-world rating measure and the ability latent trait. The order of deletion was Social Behavior Schedule, Independent Living Skills Survey, Quality of Life Scale, and Social Functioning Scale based on these criteria. In each of these models based on sequential deletion, there was still room for improvement in the fit of the model, although some began to show evidence of fitting the data acceptably (e.g., the model with Social Behavior Schedule and Independent Living Skills Survey deleted). When four of the 6 scales had been deleted, the model judged as best fitting was revealed, which suggested that the Life Skills Profile and Specific Levels of Functioning were the two best scales in combination for the prediction of the ability latent trait.
The Life Skills Profile factor coefficient was not statistically significant in the last and best fitting model, suggesting that it did not add any information to that provided by the Specific Levels of Functioning. For this reason, a simultaneous regression analysis was performed, entering the two functional outcomes examiner-rated total scores as predictors of the ability latent trait. The analysis found that the Specific Levels of Functioning total score was significantly related to the ability trait, t(192)=3.09, p=.002, while the Life Skills Profile total score was not, t(192)=1.41, p=.15. In a forced entry regression analysis, when the Specific Levels of Functioning was entered first, the results were significant, t(193)=4.52, p<.001, while the LSP total score did not enter, t(192)=.88, p=.38. When the LSP total score was forced into the regression analysis first, it still did not enter the equation, t(193)=1.75, p=.081, while the Specific Levels of Functioning still contributed variance to the ability latent trait above and beyond the nonsignificant contribution of the Life Skills Profile, t(192)=4.21, p<.001. When total variance accounted for between the Ability Latent trait and Specific Levels of Functioning total scores was examined, it was revealed that the Specific Levels of Functioning ratings accounted for 24% of the variance in the ability latent trait. Thus, reducing the number of real-world functional rating scales from 6 to 1 leads to a reduction in reliable variable of 17%, but allowed for the collection of functional information with a single questionnaire.
The results of this study indicate that real-world functional outcomes rated with an array of preselected rating scales, utilizing information from the patient and an informant condensed into a judgment rating by an examiner, are related on a global basis to performance-based assessments of ability in people with schizophrenia. This is an important finding, because several different studies reviewed above have found correlations between cognitive and functional abilities and real-world outcomes that were modest to negligible, raising questions about the well-accepted relationship between cognitive impairment and disability in people with schizophrenia. These data suggest that this relationship is actually as strong as suggested by Green et al. , but also indicates that both the rating scale and the methods for rating real-world functioning may influence the strength of this association.
Further, the results indicate that many rating scales addressing real-world functioning, even with multiple sources of information and a systematic approach to ratings in optimal research conditions, are not strongly related to indices of functional abilities that are the state of the art outcomes in treatment studies. Finally, it is possible to tentatively endorse a functional outcomes scale (i.e., the Specific Levels of Functioning) that measures social, everyday living, and vocational outcomes and is related to performance on measures of everyday functional ability at a level that provides substantial information about everyday functioning.
In studies that use these performance-based measures as outcomes, it appears as if the Specific Levels of Functioning Scale could be a suitable baseline measure to index ability-relevant real-world functioning or an outcome measure in a long-term study, as well as for use in clinical assessment. Most of the real-world functional outcomes scales seem to largely be redundant with each other when utilized simultaneously and the results suggest that none of them is related to functional abilities above and beyond the relationship of the SLOF. However, this finding does not prove that other rating scales would not be suitable with similar rating methods, Later analyses will examine the utility the subscales of these rating scales for augmenting the SLOF for assessment of ability-relevant real-world functioning and which, if any, of the informant ratings could be substituted for the comprehensive interviewer judgments.
There are a number of limitations to be considered when evaluating these results. As noted immediately above, it is possible that elements of these real-world functioning scales, on either a subscale or item basis, could also relate more efficiently to ability measures than the total scores. Further, the current analyses did not address whether self or informant reports of functioning could possibly be superior to interviewer judgments and whether either of these sources of information could be safely used on their own with the current group of scales. This question will be addressed in detail in subsequent analyses. There are other real-world functional status scales that were not examined. Most of these scales were excluded at either the scale screening or RAND panel level in the early phases of this study because there was not enough data available to evaluate their suitability. Some scales have been developed since that time, such as the Schizophrenia Objective Functioning Instrument , or have had several new studies published, such as the Personal and Social Performance Scale . Finally, since we did not use the MATRICS social cognition measure, we could not compute the composite score with the MCCB normative sample scoring program. This program compares composite scores to those obtained from healthy individuals and leads to composite overall scores in impaired populations that are lower than the average of the individual item scores. For example, in a recently published clinical trial , the screening composite t-score on the MCCB was 24.7, but the average of the 9 tests we administered was 34.2. This score is similar to the average score on the same tests in our sample of 38.6, indicating that this is not a relatively unimpaired sample.
As is known from previous studies by our team and others, there are other factors that add to the prediction of real-world outcomes, with some of these factors occasionally found to have a greater impact on real-world functioning than ability (e.g., disability compensation, depression, or negative symptoms). However, even if other factors also predict outcome, treatments aimed at disability reduction try to improve ability, not to change environmental, cultural, or emotional variables influencing disability. Later phases of the VALERO study will examine the usefulness of different informants for generation of real-world functional ratings on the SLOF and later analyses of the current data set will address issues of site and informant differences in validity of real-world functional ratings. The analyses in this paper are aimed at identification of the most broadly useful real-world outcomes rating scale, when administered to the patient and informant, and followed by a best estimate rating. Such a rating procedure has been successfully implemented in previous treatment studies , suggesting that it is a feasible procedure.
This research was supported by Grants MH078775 to Dr. Harvey and MH078737 to Dr. Patterson from the National Institute of Mental Health.
Dr. Harvey has received consulting fees from Abbott Labs, Bristol Myers Squibb, Cypress Bioscience, En Vivo, Genentech, Merck and Company, Shire Pharma, Sunovion Pharma, and Teva Pharma, during the past year. None of the other authors have any commercial interests to report. This study is unrelated to Dr. Harvey’s Consulting work.