|Home | About | Journals | Submit | Contact Us | Français|
With new treatments targeting features of schizophrenia associated with functional disability, there is a need to evaluate the validity of ratings of everyday outcomes. It is unknown whether patients can validly self-report on aspects of their functional status, which would be a potentially economical method for obtaining outcome data. In this study, 67 older schizophrenia outpatients provided self-ratings of everyday real-world functioning using the specific levels of functioning scale (SLOF). They were also administered assessments of neuropsychological performance, performance-based measures of functional capacity and social skills, clinical symptoms, and quality of life. Case managers, unaware of other ratings, also generated SLOF ratings. Based on discrepancy scores, participants were categorized as accurate raters (n = 24), underestimators (n = 16), or overestimators (n = 27) of their functional status as compared to case managers’ ratings. Patients’ self-rated functional status was correlated with their subjective quality of life, but remarkably unassociated with case manager ratings of functional status or their own performance on functional capacity or social skills measures. Case manager ratings, however, were highly correlated with performance on functional capacity and social skills measures. Patients who underestimated their real world performance had better cognitive skills and greater self-rated depression than those who overestimated. Accurate raters demonstrated greater social skills than both overestimators and underestimators, while overestimators were most cognitively and functionally impaired. Accurate ratings of everyday outcomes in schizophrenia may require systematic observation of real world outcomes or performance-based measures, as self-reports were inconsistent with objective information.
Several studies have demonstrated unawareness of illness (i.e., lack of insight) as a core symptom in schizophrenia. Patients tend to underestimate the severity and misattribute the origin of most psychotic symptoms (Amador et al., 1994). While the literature examining patients’ ability to recognize their psychotic symptoms is well-established, little is known about how patients perceive other important features of their illness and their functioning. For example, patients with schizophrenia have notable impairments in cognitive functioning and in the ability to perform everyday functional skills (Bowie and Harvey, 2005; Heaton et al., 1994; Saykin et al., 1991; Twamley et al., 2002). Previous studies have shown that patients’ self-report of their cognitive functioning tends to be uncorrelated with their performance on cognitive tests (Keefe et al., 2006; van-den-Bosch and Rombouts, 1998). In a study by Keefe and colleagues (Keefe et al., 2006), self-reported cognitive functioning was less strongly related to actual performance than were ratings of an informant or performance on measures of everyday functional skills (i.e., functional capacity).
Compared to mood disorder patients, those with schizophrenia self-report better quality of life, contradicting objective evidence of greater disability (Atkinson et al., 1997). McKibbin and colleagues (McKibbin et al., 2004) found high correlations between patient-reported level of disability and subjective reports of health-related quality of life, but found that neither of these reports was associated with cognitive performance or functional capacity scores. Lack of convergence between self-reported cognitive and functional impairments and information collected from outside observers is not limited to schizophrenia. In fact, patients with traumatic brain injuries (Hoofien et al., 2004) and conditions such as multiple sclerosis (Carone et al., 2005) have also been reported to produce appraisals of their functional and cognitive capabilities that were discrepant from objective performance and informant ratings. Interestingly, in both illnesses, the presence of depression was associated with accurate to underestimated reports of cognitive and functional skills, while greater cognitive impairments and emotional lability were found to correlate with overestimation of performance.
Self-reports of functioning therefore appear problematic and alternative assessment methods may be required. Concerns regarding measurement of functional status include both practicality and feasibility. Prescribers’ reports may be hampered by the fact that they may have little opportunity for direct observation of a patient’s behavior. Relatives may not be readily available or they may have limited contact with the patients. Direct observation may be impractical and limited to only some of the potentially relevant environmental situations. To this end, performance-based assessments of skills have been developed. While it could be argued that the generalizability of this method may be limited because there may be differences between patients’ competence or capacity (i.e., what they can do) and performance in the real world (i.e., what they actually do), functional capacity may also be less likely to be influenced by factors outside the control of treatment interventions (e.g., disability status, familial financial resources). In fact, we have demonstrated that functional capacity (what a patient can do), based on performance in a laboratory setting, is moderately correlated with real world deployment of those skills, but strongly correlated with performance on measures of cognitive functioning (Twamley et al., 2002; Bowie et al., 2006).
The purpose of this study was to examine the convergence of schizophrenia patients’ reports of their everyday functional status (using a self-report of real-world functional outcomes) with the reports of their case managers and to identify the correlates of the level of accuracy of these reports. Older ambulatory patients with schizophrenia completed the specific levels of functioning (SLOF Schneider and Struening, 1983) scale as part of a longitudinal study of cognitive and functional status. For all patients in the study who were receiving case management services, this same instrument was also completed by the case manager. To understand the possible determinants of convergence between self- and case manager reports, we also examined other subjective (quality of life, depression) and objective (performance-based assessments of functional and social skills, cognitive performance, and systematic ratings of symptom severity) measures of features of the illness. We hypothesized that: (1) patients and their case managers would differ in their SLOF ratings; (2) that case managers’ SLOF ratings would be more highly correlated with objective measures of functioning than would patients’ ratings, and (3) better cognitive functioning would be associated with more convergent SLOF ratings among the patients.
Sixty-seven patients with a DSM-IV diagnosis of schizophrenia (n = 51) or schizoaffective disorder (n = 16) and age raging from 50 to 75 (M = 56.6, SD = 7.5) were selected for the present analyses. Most (51) of the participants were male, 41 were Caucasian, 14 were African–American, and 12 were Hispanic. They had a mean of 12.5 (SD = 1.9) years of education, first hospitalization at 22.1 years of age (SD = 4.49), and 6.2 (SD = 7.9) subsequent hospitalizations. Schizophrenia and schizoaffective patients did not differ on demographic factors, cognitive impairments, or functional status ratings (all p-values > 0.05).
Participants in this study were sampled from a longitudinal study of cognitive and functional status in older people with schizophrenia. The long-term goal of this project is to examine the course and correlates of change in functional status among older, ambulatory schizophrenia patients. To be included in the present analyses, participants were required to have a diagnosis of schizophrenia or schizoaffective disorder, to be in outpatient psychiatric rehabilitation treatment at the time of recruitment at a VA, New York State, or academic research site, to see a case manager at least biweekly who reported knowing the patient “very well”, and to be prescribed atypical anti-psychotic medications. Outpatient status was defined as living outside of any institutional setting, including a nursing home. Study subjects were also required to have evidence of continued illness at the time of recruitment, as evidenced by meeting at least one of three criteria: (1) an inpatient admission for psychosis in the past two years; (2) an emergency room visit for psychosis in the past two years; or (3) a score on the PANSS positive symptoms items delusions, hallucinations, or conceptual disorganization of 4 (moderate) or more at the time of their baseline assessment. Exclusion criteria for this study included a mini-mental status examination score below 18 or any medical illnesses that might interfere with the assessment of cognitive functioning. All subjects signed a written informed consent form approved by the institutional review board at each research site after the testing procedures were fully explained.
All subjects met diagnostic criteria for schizophrenia or schizoaffective disorder (DSM-IV). The comprehensive assessment of symptoms and history (CASH Andreasen et al., 1992) was completed by a trained research assistant and the diagnosis was confirmed by a senior clinician. Trained raters at each site performed all testing.
Participants completed all instruments in a fixed order. Measures included a comprehensive neuropsychological battery, the SLOF, functional capacity assessments the University of California, San Diego performance-based skills assessment (UPSA Patterson et al., 2001a) and the social skills performance assessment (SSPA Patterson et al., 2001b), assessments of psychiatric symptom severity positive and negative syndrome scale (PANSS Kay, 1991); Beck depression inventory (BDI Beck et al., 1996), and a self-rated Quality of Life Scale assessing degree of impairment in work, social life, and home responsibilities on a 10-point Likert scale. In addition, a case manager completed the SLOF.
A cognitive composite score was created by z-scores based on age- and education-corrected published norms for individual tests. Variables in this composite were parts A and B of the trail making test (Reitan and Wolfson, 1993), the difference score from the digit span distraction test (Oltmanns and Neale, 1975), scores on the digit symbol coding, digit span, and letter-number sequencing tests from the WAIS-III (Psychological Corporation, 1998), Rey auditory verbal learning test total learning and recognition correct responses (Schmidt, 1996), total unique words on the controlled oral word association test (FAS condition (Spreen and Strauss, 1998)), and the interference score on the Stroop color word test (Golden and Freshwater, 2002). Participants also completed the wide range achievement test (WRAT Wilkinson, 1993) reading subtest as an estimate of their premorbid intelligence (Harvey et al., 2006).
The UCSD performance-based skills assessment battery (UPSA Patterson et al., 2001a) is designed to directly assess functional capacity among people with severe mental illness. This test measures performance in a number of domains of everyday functioning through the use of props and standardized performance situations. The comprehension/planning domain measures the patient’s ability to comprehend written material that describe recreational outings and then plan the activities and list appropriate items necessary to bring to the outings. In the finance domain, the patient must count out given amounts from real currency, make change, and fill out a check to pay a utility bill. The communication domain involves a series of role-play situations that require the patient make emergency calls, call directory assistance to request a telephone number, call the number, and call to reschedule a medical appointment. In the transportation/mobility domain, patients use information from bus schedules and maps to determine appropriate fare, state telephone numbers to answer relevant questions, decide which map to use to get to a certain location, and determine the appropriate route and transfers to reach a destination. For the purposes of these analyses, aiming to reduce the number of total correlations, the UPSA total score was used as the dependent variable. This score is the result of transforming each domain to a score ranging from 0 to 20 and summing the transformed domains for a total score ranging from 0 to 80, as per the author’s directions (Patterson et al., 2001a).
The social skills performance assessment (SSPA Patterson et al., 2001b) is a measure of social skills that was created for use with schizophrenia patients. After a brief practice, the patients are asked to initiate and maintain a conversation for three minutes in each of two situations: meeting a new neighbor and calling a landlord to request a repair for a leak that has gone unfixed after a prior request. These sessions are audiotaped and scored by a trained rater who was unaware of all other data. Dimensions of performance scored include fluency, clarity, focus, negotiation ability, persistence, and social appropriateness. The mean rating of these variables across the two measures was used as the dependent variable in this study. These raters were trained to the gold standard ratings of the instrument developers (ICC = 0.86) and inter-rater reliability was maintained at three months (ICC = 0.87).
The SLOF includes 43 items that rate the patient’s real-world performance in six domains. Each item is rated on a 5-point Likert scale with anchors describing the frequency of the behavior and/or the patient’s level of independence. Patients completed the SLOF with verbal instructions from the examiner to rate their own performance. No other instructions or coaching were given, so that the instrument would be consistent with a true self-report instrument that could be completed independently in other settings, such as a clinical trial. An additional item asks the case manager to rate, on a 5-point scale, how familiar they are with the skills and behavior of the person. Patients in this study were selected on the basis of their case managers indicating that they knew the behavior and skills of their patient “very well”. The SLOF domains include physical functioning (5 items), personal care skills (7 items), interpersonal relationships (7 items), social acceptability (7 items), activities (11 items), and work skills (6 items). The physical functioning domain had ceiling effects in the present sample of community dwelling schizophrenia patients and was excluded from these analyses. A functional skill composite is created by summing the interpersonal relationships, activities, and work skills domains. This composite was comprised using these domains due to lack of variability in the other domains as well as there assessment of higher-level (instrumental) functional skills, as opposed to the basic functional skills assessed by personal care and physical functioning.
For all measurements except for the PANSS, QOL, and BDI, higher scores reflected better performance. Discrepancy scores were calculated in the functional skills composite score by subtracting the self-report ratings from the case manager ratings. Patients were then categorized, according to the direction and degree of deviation of their functional skill composite ratings with those of the case managers, as “accurate estimators”, “overestimators”, or “underestimators”. Accurate Estimators were within 0.75 standard deviations (SD) in either direction on the composite of the case manager rating, overestimators rated their functional skills greater than 0.75 SD of the case manager rating, and underestimators rated their functional skills at less than 0.75 SD of the case manager rating. We used a cutoff of 0.75 SD as a more conservative criterion than 0.5 SD, which has been widely reported as the minimally important amount to detect differences in health-related instruments (Norman et al., 2003).
Two correlational analyses were performed using Pearson correlation coefficients for all domains other than the Personal Care domain, for which Spearman’s rho was employed due to this domain’s non-normal distribution of scores. The Physical Functioning scale was not used in this study due to ceiling effects. To minimize experiment-wise error due to multiple comparisons, Bonferroni correction was applied, setting the alpha level at 0.01 in the first analyses, which examined convergence of self-report and case manager-rated SLOF domains. These analyses are presented in Table 1 and demonstrate a general lack of convergence between these two sources in rating functional performance. However, ratings on the personal care subscale were significantly correlated with each other, suggesting that patients were relatively more adept at appraising their personal care skills.
In the second set of analyses, we used Pearson correlation coefficients to examine the relationships between the self- and case manager ratings of functional skills with the various criterion measures, with the Bonferroni correction set to 0.001 (see Table 2). We did not include the personal care subscale due to lack of an objective criterion measure. The case manager ratings showed good construct validity, with case manager ratings of social skills and work correlating with the SSPA, ratings of everyday functioning correlating with the UPSA, and both work skills and activities correlating with the cognitive composite score. The pattern of significant correlations within the patient sample was quite different. Self-reported functional and social deficits were correlated with self-reported depression and lower quality of life, but not with the cognitive or performance-based functional capacity measures. The test for significant differences between correlations revealed stronger correlations for the case manager, compared to patient ratings of activities with the UPSA and cognition, and of work with cognition. Patient’s ratings were more strongly correlated than case manager ratings on the work domain with depression and a trend for the activities domain with depression. Several other trends for significant differences between the correlations are displayed in Table 2.
Based on the classification system described above, 24 (36%) of the patients were accurate estimators, 27 (40%) were overestimators, and 16 (24%) were underestimators. Univariate analyses of variance were conducted to examine differences between the three groups on criterion measures. See Table 3 for descriptive statistics and group means. The three groups did not differ in age, years of education, or estimated premorbid intellectual functioning (WRAT reading score). Underestimators were more likely than both accurate estimators and overestimators to endorse more severe symptoms of depression. Underestimators also had less cognitive impairment than did the overestimators, though they did not differ significantly from the accurate estimators. Accurate estimators demonstrated better social skills than the other two groups. The overestimators were more functionally impaired based on case manager SLOF ratings than were the other two groups and more cognitively impaired than were the underestimators.
This study addressed unanswered questions in the assessment of functional deficits in schizophrenia. Specifically: Which patients appraise their functioning accurately, and what areas of functioning can they rate? Patients in this study demonstrated convergence with case manager ratings in their personal care skills, despite not appraising other. more instrumental, functional domains such as interpersonal skills, activities in the community, and work skills in a manner consistent with observer ratings. Patient self-reports of these functional domains had small, non-significant correlations not only with the case managers’ ratings of the same items, but also with objective measures of performance, such as the UPSA, SSPA, and neuropsychological performance. The correlations of the case manager ratings of Activities and Work with the UPSA and cognition were significantly greater in magnitude than the self-ratings. The correlations of patients’ self-reported Work skills with depression were greater in magnitude than case manager ratings.
Although 36% of patients could be classified as accurate raters of their functional skills with the SLOF, as compared to their case manager, most were inaccurate, with 40% overestimating their abilities and 24% underestimating their abilities. Accurate raters had better interpersonal skills than inaccurate raters, but were not significantly different from inaccurate raters on overall cognitive performance. Those patients who underestimated their functional skills had the highest level of cognitive ability, but also the highest level of self-rated depression. Moreover, while self-rated functional skills were associated with measures of subjective well-being, such as quality of life and depression, the observer ratings of functional skills were linked to the objective measures of performance on the part of the patients. This is consistent with one of our recent studies (Rieckmann et al., 2005), where we found that depression and cognitive performance were positively correlated with each other. These findings could also be viewed as extending the “depressive realism” (Dunning and Story, 1991) and “sadder but wiser” (Alloy and Abramson, 1979) concepts to a psychotic population. Additionally, these findings are similar to previous studies of multiple sclerosis patients, which also found higher depression in those who underestimated their cognitive ability (Carone et al., 2005).
Across the functional skill domains, case manager ratings were more highly correlated with objective measures such as cognitive performance, UPSA performance, and SSPA performance than were self-appraisals. Patients’ self-ratings tended to be correlated with measures of subjective outcomes, such as depression and quality of life, but less so with the objective measures of functional skills and cognition. Interestingly, neither case manager nor self-reported functional status was associated with the traditional negative and positive symptoms of schizophrenia. These replicate previous findings with schizophrenia patients (McKibbin et al., 2004) that demonstrated that schizophrenia patients give internally consistent self-reports across different domains, but that these self-reports were not associated with objective indices of functioning. Brekke et al. (Brekke et al., 2001) found that a global measure of psychosocial functioning (the global assessment of functioning (GAF)), was associated with subjective measures of self-esteem and life satisfaction in schizophrenia patients with executive functioning deficits. However, similar to the current results, for patients with better executive functioning, higher GAF scores were associated with worse self-esteem and life satisfaction. Taken together, the Brekke et al. (Brekke et al., 2001) and McKibbin et al. (McKibbin et al., 2004) studies and the present findings suggest that when patients with schizophrenia are asked to appraise their everyday functioning: (1) they rely heavily on subjective factors (2) their reports may be dependent on their level of cognitive impairments, and (3) they appear to rely less on objective indicators of functioning.
Cognitive functioning, as assessed by traditional neuropsychological instruments, may moderate the relationship between self-rated and objective functional status. Meta-cognitive impairments may further contribute to erroneous self-appraisal. Assessing one’s functional performance, particularly as it compares to some optimal level of functioning, requires the ability to first appraise one’s own level of functioning and then consider how this compares to an external standard. Multiple types of errors may arise when asking schizophrenia patients to self-rate functional skills, including misunderstanding the items on a rating scale, generate inaccurate self-appraisals, inaccurately conceptualizing normal functioning, and making inaccurate comparisons to external standards. Categorizing patients as accurate estimators, overestimators, or underestimators revealed distinct profiles which crossed over cognitive, functional, emotional, and quality of life domains. Overestimators were characterized by cognitive impairments and more global observer-rated functional deficits. Their tendency to be more cognitively impaired than patients who are accurate or underestimators lends more support to the hypothesis that certain aspects of cognitive impairments contribute to either unawareness of illness, poor self-appraisal, unawareness of external standards or inability to accurately complete the instruments.
The dissociation between accurate appraisal of self-care and other functional skills could be due to a number of factors. Patients in this study were relatively high functioning and had good personal care skills, suggesting that they are able to rate functional ability areas that were relatively more intact. Previous research (Evans et al., 2003) found that this ability area was not related to cognitive impairments as well, probably due to ceiling effects on measures of personal care. Further, external standards for personal care may be more easily discernable than, for instance, social skills to people with schizophrenia, leading to more accurate appraisals. It is important to keep in mind that our gold-standard measure of functioning, the case manager ratings of the SLOF, is not without its own limitations. There is an important but potentially overlooked distinction between an “objective” and a “valid” report. Previous reports have indicated that clinician-rated functional disability can vary widely in their correlations with other symptom measures based on instrument employed (Pyne et al., 2003). Future research should aim to determine the most feasible and valid measure of functional status in schizophrenia, as optimism increases regarding functional recovery as a real treatment goal.
This research was funded by NIMH Grant 63116 (PD Harvey, PI), and supported by the Silvio Conte Neuroscience Center (NIMH MH 36692; KL Davis PI), (NIH M01-RR-00071 GCRC), a NARSAD Young Investigator Award to Dr. Bowie, and the VA VISN 3 MIRECC.