The results of this study provide support for the reliability and validity of the PST. In general, items on each of the four tests are congruent and appear to be measuring the same construct, as indicated by acceptable α levels. In addition, results demonstrated the expected convergent and divergent associations with other measures. As hypothesized, the PST was positively correlated with the ACLS–2000. This finding is consistent with previous evidence of the association between cognition and life skills among people with mental illness. As summarized by Green, Kern, Braff, and Mintz (2000)
, studies have consistently indicated that cognition (including executive functioning, working memory, and vigilance) is moderately correlated with functional outcomes (including skills for daily and community activities). More recent studies have also confirmed the finding that individuals’ daily functioning is positively associated with their cognition (Keefe, Poe, Walker, Kang, & Harvey, 2006
; Kurtz, Wexler, Fujimoto, Shagan, & Seltzer, 2008
; Schultz, Ellingrod, Turvey, Moser, & Arndt, 2003
However, scores on the PST were not significantly correlated with the IES–R. This finding is also consistent with the literature indicating that life skills have no notable correlation with trauma symptoms (Davis & Kutter, 1998
). Likewise, previous research has verified that psychiatric symptoms are generally not significantly associated with daily functioning, even though this finding may not fit with clinical impressions (Green, 1996
; Kurtz, 2006
). As noted earlier, this area needs further exploration.
Regarding floor and ceiling effects, we found no floor effect for all PST tests, and only Safe Community Participation had more than a minor ceiling effect at T2. The result for Safe Community Participation suggests this test may have less sensitivity to capture change or improvement. Although other tests did not have much of a ceiling effect, the responses were all negatively skewed with high mean scores, suggesting that the tests may not be sufficiently challenging for the participants. Thus, the tests may not discriminate well across the full range of life skills knowledge.
Although all test scores except Money Management showed significant change after life skills intervention, none of the tests demonstrated a substantial effect size. This result may also indicate insufficient difficulty of the tests, which was reflected in the high mean scores of each test at baseline. This limitation may affect the tests’ ability to detect improvement because there is not much room for scores to improve.
The results suggest some areas of potential improvement for the PST. Several approaches may be possible. The first approach is to adjust the difficulty of the tests. Although life skills deficiency has been a significant concern with homeless people, the participants obtained relatively high scores on the PST. This finding suggests a need to expand the test by adding more difficult items. Because expanding the length of a test can present problems of feasibility, application of item response theory (IRT) methods, especially with a computer adaptive test (CAT), could be explored. IRT methods can help to identify a pool of items that cover the full range of difficulty and to examine the match between the range of performance in the relevant population and the range covered by the items. Subsequent incorporation of the item pool into a CAT yields a measure that can be short to administer but very precise (Haley et al., 2006
). The difficulty of the tests can also be adjusted by removing the items that are too easy for all participants or by changing some of the response formats from true–false options to multiple choice or open-ended short answer format to reduce the possibility of being correct just by chance.
Limitations and Future Research
This study has some limitations that should be considered. First, although the sample size was sufficient for each analysis, a larger sample would provide more reliable statistical estimates. Because the population studied is unstable and vulnerable in coping with mental illness and ongoing substance use, recruiting a large number of participants into the study and keeping them participating is especially difficult. Second, the study sample may limit the generalization of the results. The participants came from two metropolitan cities. Therefore, whether the study results are generalizable to people who live in rural areas or people with other types of living situations is questionable. Clinicians applying the PST to evaluate improvement of individuals’ life skills after treatment should keep in mind that the change of score on the PST may be in the small to moderate range.
This study’s findings may be used to revise the PST. Future research should also investigate use of the PST with other populations with life skills needs. For example, examining the validity of the PST for people with traumatic brain injury or cognitive disabilities would be useful. In addition, it will be important to examine differences on the PST among subgroups because life skills performance may differ depending on different housing situations, clinical conditions, or cognitive levels. Finally, because the PST is an assessment of life skills knowledge, validating the tests’ results against actual performance of skills in real-life situations will also be critical.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
- The PST is a valid measure of life skills in food management, money management, safe community participation, and home and self-care for people who are homeless.
- Evidence has shown that the PST is moderately associated with a client’s cognitive level but not with trauma symptoms.
- The PST is a valid and useful measurement tool for measuring outcomes of occupational therapy life skills interventions.