|Home | About | Journals | Submit | Contact Us | Français|
The objective of this study was to examine the sensitivity and specificity of a performance-based measure of functional capacity, the UCSD Performance-Based Skills Assessment (UPSA) for the prediction of independent living status in patients with chronic schizophrenia-related conditions. A sample of 434 adults with schizophrenia or schizoaffective disorder was administered the UPSA and assessed for independent living status. Participants were classified as “independent” if they were living alone in an apartment, house, or single-resident occupancy (e.g., hotel room) and non-independent if they resided in a care facility (e.g., Board-and-Care home, Skilled Nursing Facility). Receiver Operator Characteristic (ROC) curves were calculated with the UPSA and Mattis’ Dementia Rating Scale (DRS) scores as predictor variables and residential independence as the state variable. Of the 434 participants, 99 (23%) were living independently at the time of assessment. The discriminant validity of the UPSA was adequate (ROC area under the curve = 0.74; 95% CI: 0.68–0.79), with greatest dichotomization for the UPSA at a cutoff score of 75 (68% accuracy, 69% sensitivity, 66% specificity), or 80 (68% accuracy, 59% sensitivity, 76% specificity). The UPSA was also a significantly better predictor of living status than was the DRS, based on ROC (z = 2.43, p = .015). The UPSA is a brief measure of functional capacity that predicts the ability of patients with schizophrenia to reside independently in the community.
Many individuals suffering from schizophrenia are not capable of residing independently in the community, instead residing under assisted-living arrangements such as in a Board and Care (B&C) homes or skilled nursing facilities. Although it is likely that multiple factors play a role in independent living skills, research suggests that the cognitive impairments associated with schizophrenia limit the patients’ abilities to live and function in the community. For example, recent research has examined various determinants of real-world functional performance in patients with schizophrenia, including the severity of neurocognitive deficits. These studies indicate that neurocognitive functioning is related to interpersonal skills, engagement in community activities, and work skills (Bell & Bryson, 2001; Bowie et al., 2006; Bryson & Bell, 2003; Bryson et al., 1998). For example, Green et al. showed that neurocognitive deficits significantly impaired patients’ daily functioning abilities (Green, 1996; Green et al., 2000), including success in psychosocial skill acquisition (e.g., learning basic conversation skills and medication management), acquisition of instrumental skills and social problem-solving ability (e.g., role-playing performance of skills), and community outcomes/daily activities (e.g., occupational functioning, social attainment, degree of independent living). Although their review and meta-analysis suggested that neurocognitive functioning was the single best predictor of functional outcome, Green and colleagues (2000) called for more research identifying how neurocognition is related to functional outcome. Recent research has sought to answer this question through examining functional capacity. These studies have examined the ability to perform specific, functionally relevant, everyday living skills in controlled assessment settings. In a study of older patients with schizophrenia, Bowie et al. (2006) examined the role of neurocognitive functioning and functional skills capacity in predicting real-world functional performance (e.g., personal care skills, interpersonal skills, community activities) as rated by case managers with extensive contact with the patients. Functional capacity was measured with the UCSD Performance-based Skills Assessment (UPSA; Patterson et al., 2001), a performance-based measure assessing the ability to adequately perform a number of everyday functional tasks such as comprehending written material, planning activities, and managing money. The results indicated that while neurocognitive functioning significantly predicted everyday functional outcomes as rated by the case managers, this relationship was mediated, in large part, by performance-based measures of functional capacity.
Although these studies indicate that neurocognitive functioning, functional capacity, and “real-world” functional performance are interrelated, few studies have examined the types of concrete outcomes that are relevant to healthcare providers and medical decision-makers, such as the ability to live independently. Along these lines, Revheim and Medalia (2004b) examined the relationship between cognitive functioning, problem-solving skills, and community status (inpatient vs. outpatient) in a sample of 162 individuals diagnosed with schizophrenia or schizoaffective disorder. They found that cognitive functioning was correlated with problem-solving ability, and that problem-solving skills best predicted residential status. Similarly, Twamley et al. (2002), in a study of outpatients with schizophrenia, found that the UPSA performance was correlated with neuropsychological performance, and that UPSA scores significantly predicted level of independence in living.
The ultimate functional expression of problem-solving and cognitive abilities, that is, residing independently in the community, may rest heavily on the ability to perform basic functional tasks such as paying bills, scheduling medical appointments, complying with a work schedule, and shopping for food and household supplies. The development of a “gold standard” measure that captures these domains and is capable of accurately predicting the ability to reside independently in the community is needed for healthcare providers and family members who must make decisions regarding safe living situations for patients (e.g., whether to discharge an inpatient with schizophrenia to a B&C facility or independent living environment). This becomes particularly important when considering the “deinstitutionalization” of people with mental illness over the past 50 years, including the release of persons residing in psychiatric hospitals to alternative facilities in the community (Lamb & Bachrach, 2001). Indeed, Lamb and Bachrach (2001) report that between 1955 and 1998, the number of occupied state hospital beds in the United States was reduced from 339 per 100,000 population to 21 per 100,000 population. In California, the pioneer in deinstitutionalization, this number was 3 per 100,000 (Lamb et al., 2001).
As discussed by Obuchowski (2005), the preferred method for providing clinically useful information is the use of ROC analysis, which has several advantages over other methods. First, the ROC curve provides much more information about how a test performs than just a single estimate of the test’s sensitivity and specificity. Second, ROC curves allow clinicians to examine the costs and benefits of a test’s sensitivity and specificity for various cutoff scores on the test. Third, upon weighing these costs and benefits, clinicians can choose the optimal cutoff for each patient. In addition to providing clinically meaningful information, ROC curves are often the only valid method of comparing two or more diagnostic tests.
The purpose of this study was to assess the ability of the UPSA, a brief measure of functional capacity, to independently predict the level of residential independence in 434 patients diagnosed with primary psychotic disorders (i.e., schizophrenia or schizoaffective disorder). We further compared the UPSA with other measures possibly related to residential independence, such as measures of positive and negative symptoms, depressive symptoms, and neurocognitive performance. As discussed above, comparison of these measures for predicting residential independence was assessed by means of ROC curves. We hypothesized that the UPSA would be significantly better than these measures for predicting residential independence.
Four hundred thirty four individuals diagnosed with either schizophrenia or schizoaffective disorder were the participants for this study. All subjects were ongoing research participants at the University of California, San Diego (UCSD) Advanced Center for Interventions and Services Research (ACISR), which examines, among other aspects, functional abilities of these patients. Diagnosis of schizophrenia-related disorders was established by DSM-IV-based chart diagnosis using the patients’ research charts. All participants volunteered and provided written, informed consent to participate in research studies.
All the participants were administered the UPSA (Patterson et al., 2001), which assesses a person’s ability to perform a variety of everyday living tasks. The UPSA involves role-play tasks similar in complexity to situations that a community-dwelling person is likely to encounter. Patients are first tested in the area of Comprehension/Planning and are asked to read two stories. The first describes an outing to the beach on a hot sunny day and the second depicts a trip to the zoo to see the giant pandas on a cold rainy day. After reading the stories, participants are asked a few questions to evaluate their comprehension of the material and then requested to list five items necessary to bring or wear in order to spend the whole day at the beach/zoo. Points are given for answers deemed appropriate (e.g., swimsuit, towel, picnic lunch, sunscreen, etc. for the beach and umbrella, raincoat, camera, money, walking shoes, etc. for the zoo). This part of the assessment takes about eight minutes to complete.
The area of Finance tests one’s ability to count change and write checks. In the first part of this task patients are provided with real currency (coins and bills) and asked to count out given amounts (e.g., $12.17, $6.73, $1.02) and make change from ten dollars. The second part of the task involves filling out a check to pay a utility bill. Patients are shown a real bill from a utility company (e.g., San Diego Gas and Electric) and are required to make out a check. Points are given for each correct element of this process (e.g., the check is made out to SDG&E, the written amount corresponds to the bill, the check is signed, etc.). These tasks take about eight minutes to complete.
The next area tested is Communication. Patients are provided with an unplugged telephone and asked to role-play a number of scenarios. First they are asked to show the tester what number they would dial if they had an emergency. The appropriate response is to dial 9-1-1. A second task involves calling information to get a specific telephone number and then dialing that number from memory. Next, patients are requested to read a medical appointment confirmation letter and then role-play calling the hospital to reschedule the doctor’s appointment. In addition, subjects are asked to describe how the letter requested them to prepare for the medical appointment (e.g., fast for a blood draw) and what two items they need to bring with them to the doctor (e.g., insurance card and list of medications). There are a total of nine communication subtasks that require about five minutes to complete.
The area of Transportation includes the use of public transportation. Subjects are provided with bus schedule information from the San Diego Transit District and asked questions about the cost of their fare, the telephone number for schedule information, which bus to ride to a specific destination, where they would get off the bus to transfer to another bus, and the location of the trolley stations on a map. This task requires about five minutes to complete.
The final task measures Household Management skills. Patients are provided with a recipe for rice pudding and asked to prepare a written shopping list. They are then presented with an array of 29 items that one possibly would have on hand in their pantry (e.g., pasta, jelly, cereal, soup, rice, canned tuna, toothpaste, canned vegetables, crackers, etc.). Patients are requested to read the recipe, check the pantry, and then prepare a list of the missing items they need to buy in order to make the rice pudding. Points are given for each correct item on the shopping list. This task is timed for five minutes. Overall, the UPSA requires approximately 30 minutes to administer and to score in total. Inter-rater reliability of ratings is excellent (Patterson et al., 2001). Participants receive scores for each of the 5 subscales (range = 0–20), which are summed to create a summary score ranging from 0 to 100. Previous research has established that the UPSA is highly related to neurocognitive functioning (Bowie et al., 2006; Twamley et al., 2002) and performance on a number of functional abilities (Bowie et al., 2006).
Mattis’ Dementia Rating Scale (DRS; Mattis, 1973) was administered as an assessment of overall cognitive functioning. This scale consists of a number of subtests assessing various cognitive abilities including attention, initiation/perseveration, constructional abilities, conceptualization, and memory. Scores on these scales are summed to create an overall score (maximum score = 144), with higher scores indicating better functioning.
The Positive And Negative Syndrome Scale (PANSS; Kay et al., 1987) was used to assess positive and negative symptoms of psychosis. The PANSS is a 30-item instrument with each item rated on a seven-point severity scale. Seven items are summed to reflect overall positive symptoms, another seven contribute to the negative syndrome score.
The Hamilton Rating Scale for Depression (HAM-D; Hamilton, 1969) was administered to evaluate patients’ severity of depressive symptoms. A trained observer provided ratings and on a 0-to-4 or 0-to-2-point scale on each of the 17 items, with higher scores indicating greater depressive symptoms. The HAM-D is a brief yet comprehensive measure of depressive symptoms, with well documented reliability and validity (Hamilton, 1969; 1974).
Participants were classified as “independent” if they were living alone in an apartment, house, or single-resident occupancy (e.g., hotel room) and non-independent if they resided in a care facility (e.g., Board and Care, Skilled Nursing Facility). Our dataset also included 134 individuals who were residing in an apartment or house with another individual (e.g., spouse, family member, etc). However, they were not included in the present analysis because we were unable to determine how much assistance these patients were provided by their “roommates”.
A Receiver Operating Characteristic (ROC) curve was plotted for the UPSA. The ROC curve shows the sensitivity versus one minus the specificity for every possible cutoff point; optimal cutoff points are determined by visually assessing which score combines maximum sensitivity and specificity. The area under curve (AUC) with 95% confidence intervals was used as an indicator of the ability of the UPSA to differentiate patients who were living independently in the community (e.g., living alone in an apartment) or not living independently (e.g., living in a B&C facility). In addition, we used the methods described by Hanley and McNeil (1982; 1983) to compare the AUCs for other measures potentially predictive of residential independence (e.g., cognitive performance; positive, negative, and depressive symptoms). Finally, using the cutoff established by the AUC analysis, we conducted additional analyses to describe characteristics (e.g., age, cognitive functioning) also prevalent in those scoring above and below this cutoff.
Characteristics of those living independently and non-independently are presented in Table 1.
Of the 434 participants, 99 (23%) were residentially independent. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and percent correctly identified as residentially independent for different cutoff scores on the UPSA are shown in Table 2. Briefly, sensitivity refers to how good the test is at correctly identifying people who reside independently, whereas specificity is the test’s ability to correctly identify people who are not independent (Loong, 2003). Similarly, positive predictive value is the proportion of patients with positive test results who are correctly classified as independent, whereas negative predictive value is the proportion of patients with negative test results who are correctly classified as dependent (Altman & Bland, 1994). The ROC curve is shown in Figure 1. The estimated AUC for the UPSA was 0.74 (95% CI: 0.68–0.79), which was significantly greater (p < .001) than the area of no information (an area of 0.50). As seen in Table 2, the best UPSA cutoff scores were 75 and 80. At a cutoff of 75, the UPSA correctly identified 68% of the sample, with a sensitivity of 0.69 and specificity of 0.66. At a cutoff of 80, the UPSA correctly identified 68% of the sample, with sensitivity and specificity of 0.59 and 0.76, respectively.
To establish whether the UPSA was better than other predictors of residential independence, we produced ROC curves (with 95% confidence intervals) for positive and negative symptoms, depressive symptoms, and DRS total scores. For these analyses, PANSS positive scores were available for 427 of the original 434 participants. PANSS negative data were available for 425 participants, DRS scores were available for 355 participants, and HAM-D scores for 421 participants. The estimated AUC for PANSS positive scores was 0.55 (95% CI: 0.49–0.62). The PANSS negative AUC estimate was 0.46 (95% CI: 0.40–0.52). The DRS AUC estimate was 0.65 (95% CI: 0.58–0.73). Using the estimate of average correlation provided by Hanley and McNeil (1983), we compared the AUCs for the UPSA and DRS for the 355 participants who had data available on both tests; the UPSA AUC estimate of 0.74 (95% CI: 0.68–0.81) was significantly greater than that of the DRS detailed above (z = 2.43, p = .015). ROC curves for the UPSA and DRS are presented in Figure 2.
Finally, we compared demographic and functional differences between the patients scoring above versus below an UPSA cutoff of 75. Results of these analyses, presented in Table 3, suggest that, compared to those scoring <75, individuals scoring 75 or above displayed fewer positive and negative symptoms and performed significantly better on the DRS. These findings suggest that in addition to residential independence, scores on the UPSA are sensitive to other clinically meaningful outcomes. However, the findings presented above suggest that UPSA scores are the best single predictor of residential status.
Healthcare providers and other medical decision-makers for patients with chronic schizophrenia and other serious mental illnesses often must make decisions on whether their patients are capable of residing independently or would be better served by assisted care settings such as B&C facilities. Although many factors play a role in this decision, including neurocognitive functioning, level of positive, negative, and depressive symptoms, our study sought to establish the usefulness of the UPSA, a brief assessment of functional capacity, for prediction of residential independence. We found that the UPSA was significantly better than chance and better than classical clinical features of schizophrenia such as positive and negative symptoms and global cognitive functioning at predicting residential independence.
It has been suggested that assessments focusing on functional skills (e.g., the UPSA) do not adequately take other cognitively relevant features of everyday functioning into account. Specifically, Revheim and Medalia (2004a) argued that tests of functional capacity (e.g., observing a patient’s ability to pay bills) do not take into account important functional features such as remembering to pay bills or otherwise initiate functionally relevant tasks. Although not discounting this observation, our data demonstrates that performance on the UPSA is highly related to global cognitive performance (see Table 3) and serves as a significantly better predictor of residential independence than does global cognitive functioning. Thus, if this assessment could incorporate assessments of tendencies toward initiation, it might be even more sensitive.
The UPSA offers several advantages relative to other clinical measures (e.g., DRS, PANSS, HAM-D). For example, the brevity of the UPSA (i.e., 30 minutes) makes it preferable to lengthy batteries assessing multiple cognitive domains. Indeed recent research (Bowie et al., 2006; Keefe et al., 2006) has indicated that the prediction of real-world outcomes is not enhanced by cognitive performance when scores on the UPSA are considered. In addition, unlike assessments of global cognitive functioning (e.g., DRS) healthcare professionals do not need specialized training in order to administer the UPSA. Further, our data suggest that clinical symptoms, particularly negative symptoms, failed to provide any meaningful advantage to predicting residential independence. However, participants scoring 75 or above on the UPSA had significantly fewer positive and negative symptoms, indicating these symptoms are captured by performance on the UPSA. Therefore, we believe the UPSA serves as a brief measure of both functional capacity and as a proxy measure of global cognitive functioning and clinical symptoms that can adequately predict residential independence.
In addition to the brevity and ease with which the UPSA may be administered, there are advantages to establishing an UPSA cutoff for predicting residential independence. For example, researchers conducting clinical trials on the benefits of medications (e.g., cognitive-enhancing drugs, antipsychotic medications) or psychosocial interventions may seek to move participants above the UPSA threshold of 75 as an indication that these treatments improve the likelihood that patients may live independently. Indeed, Patterson et al (2006) found that a behavioral intervention, known as Functional Adaptations and Skills Training (FAST), significantly improved performance on the UPSA, with average scores for patients in the FAST intervention improving from 60 to 70. These results were similar to those of a small pilot study of Hispanic patients with schizophrenia examining the effects of a behavioral intervention for improving functional capacity (Patterson et al., 2005). Although these studies did not examine the percentage of participants moving above scores of 75, they provide evidence that the UPSA is sensitive to treatment and that patients can make substantial gains on the UPSA. We encourage future investigations to examine not only overall change on the UPSA, but the success of treatments for moving patients above a cutoff of 75.
Our study provides preliminary evidence that the UPSA is a valid instrument for predicting independent living status. However, data from this study were cross-sectional rather than prospective in nature. Therefore, one interpretation of our findings is that patients living in independent living situations have greater opportunity to practice the skills assessed on the UPSA (e.g., counting change, scheduling appointments) or that patients living in B&C settings are limited in their ability to practice these skills, thus accounting for differences in UPSA scores. If true, we believe this adds strength to our previous assertion that psychosocial skills-training interventions (rather than psychotropic medications) may be useful for improving patients’ functional abilities, given their emphasis on repeated practice of skills. While the cross-sectional nature of this study is a limitation, we believe assessment of patient ability to perform these tasks may still be useful in terms of discharge planning. Yet, future research is needed to determine if the UPSA serves as a prospective predictor of residential independence in patients with schizophrenia.
Our study did not include a clinical assessment of functioning such as the Global Assessment of Functioning (GAF) scale of the DSM-IV (American Psychiatric Association, 2000). We believe the UPSA would perform particularly well compared to clinical assessments such as GAF scores. Previous studies have found GAF scores to be poor predictors of living status and functioning (Revheim et al., 2004a; Roy-Byrne et al., 1996) and suggest that relying on the GAF to assess patients’ functioning may be problematic. Nonetheless, we recommend that future studies should compare the AUC for clinical ratings of functioning such as GAF scores to that of the UPSA. Future research should also examine whether using the UPSA in conjunction with clinical assessments provides a better assessment than either of these assessments alone. Finally, future work should examine the predictive power of the UPSA prospectively across multiple placement settings to determine success in those settings and generalization to other functional outcomes.
We did not assess other indicators of functional outcome such as employment. We believe future studies should examine whether the UPSA adequately predicts employment status as a means of further validating the UPSA’s usefulness as a clinical tool. We suggest two methods of establishing the UPSA’s validity for predicting work-related outcomes. First, similar to our current study, ROC curves could be used to establish cutoffs for predicting employment status (e.g., employed vs non-employed; full-time vs part-time). Second, survival curves for maintaining employment over time could be calculated. Relevant to this point, it was recently reported (Rosenheck et al., 2006) that work outcomes are tightly linked to disability status and less strongly related to cognitive performance. It would be of interest to determine if UPSA scores were more relevant than cognitive test scores at predicting employment when disability status is also considered.
Cutoff scores between 75 and 80 most accurately predicted residential independence, although this accuracy was 68%. It should be noted, however, that our sample did not include participants who resided in care settings where maximum supervision is necessary (e.g., inpatient settings). Instead, our sample consisted entirely of community-dwelling patients. We believe that inclusion of an inpatient sample would further increase the UPSA’s specificity and sensitivity and might further enable healthcare providers to differentiate cutoffs at which individuals may be safely moved into the community (i.e., higher levels of independence). As such, we strongly encourage this line of research.
In sum, we find that the UPSA serves as an adequate predictor of residential independence for individuals suffering from chronic schizophrenia. Furthermore, the UPSA was superior to measures of psychiatric symptom severity and global cognitive performance in predicting residential independence. Additionally, the UPSA is a brief assessment that can be easily administered to patients.
This research was supported, in part, by MH 62554 by the National Institute of Mental Health to Dr Patterson, MH 66248 by the National Institute of Mental Health to Dr Jeste, and by award MH 63116 by the National Institute of Mental Health to Dr. Harvey, as well as the VA VISN-22 and VA VISN-3 MIRECC.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.