Supported Employment Outcomes of a Randomized Controlled Trial of ACT and Clubhouse Models

doi:10.1176/appi.ps.57.10.1406

Psychiatr Serv. Author manuscript; available in PMC 2009 October 10.

Published in final edited form as:

Psychiatr Serv. 2006 October; 57(10): 1406–1415.

doi: 10.1176/appi.ps.57.10.1406

PMCID: PMC2759891

NIHMSID: NIHMS144455

Supported Employment Outcomes of a Randomized Controlled Trial of ACT and Clubhouse Models

Dr Cathaleene Macias, Ph.D., Mr Charles F. Rodican, M.S.W., Dr William A. Hargreaves, Ph.D., Dr Danson R. Jones, Ph.D., Dr Paul J. Barreira, M.D., and Dr Qi Wang, Ph.D.

Dr Cathaleene Macias, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106 (Email: cmacias/at/mclean.harvard.edu)

Contributor Information.

The publisher's final edited version of this article is available at Psychiatr Serv

See other articles in PMC that cite the published article.

Abstract

Objective

In a randomized controlled trial, a vocationally integrated program of assertive community treatment (ACT) was compared with a certified clubhouse in the delivery of supported employment services.

Methods

Employment rates, total work hours, and earnings for 121 adults with serious mental illness interested in work were compared with published benchmark figures for exemplary supported employment programs. The two programs were then compared on service engagement, retention, and employment outcomes in regression analyses that controlled for background characteristics, program preference, and vocational service receipt.

Results

Outcomes for 63 ACT and 58 clubhouse participants met or exceeded most published outcomes for specialized supported employment teams. Compared with the clubhouse program, the ACT program had significantly (p<.05) better service engagement (ACT, 98 percent; clubhouse, 74 percent) and retention (ACT, 79 percent; clubhouse, 58 percent) over 24 months, but there was no significant difference in employment rates (ACT, 64 percent; clubhouse, 47 percent). Compared with ACT participants, clubhouse participants worked significantly longer (median of 199 days versus 98 days) for more total hours (median of 494 hours versus 234 hours) and earned more (median of $3,456 versus $1,252 total earnings). Better work performance by clubhouse participants was partially attributable to higher pay.

Conclusions

Vocationally integrated ACT and certified clubhouses can achieve employment outcomes similar to those of exemplary supported employment teams. Certified clubhouses can effectively provide supported employment along with other rehabilitative services, and the ACT program can ensure continuous integration of supported employment with clinical care.

Supported employment is designed to help adults with serious mental illness obtain jobs in socially integrated settings that pay at least minimum wage (1–3). Randomized controlled trials have established supported employment as an evidence-based practice on the basis of higher employment rates for specialized supported employment teams compared with interventions that do not provide supported employment (4–9). The effectiveness of supported employment has been partially attributed to rapid job placement, which bypasses prevocational training, trial jobs, and sheltered work (9–12).

However, being hired does not guarantee success on the job. Enrollees express interest in employment when entering a supported employment program, yet typically average only a few months of employment after finding a job (1,8,11,13–16). One explanation for brief work tenure is the failure of many supported employment specialists to work closely with treating clinicians (17). Encouraging integration of supported employment with case management and other rehabilitation services, such as supported housing or supported education, also seems beneficial (18,19). Research suggests that mobile teams composed of clinicians, social workers, and supported employment specialists might be especially beneficial for people in need of intensive, integrated clinical care (3,20,21). For those less in need of intensive care, membership in a clubhouse should improve work tenure, given that clubhouses develop relationships with local employers that facilitate on-the-job support (22–25).

This randomized controlled trial compared the employment outcomes for two widely disseminated (yet very different) multiservice programs designed to provide supported employment: vocationally integrated assertive community treatment (ACT), which originated at Mendota Mental Health Institute in Madison, Wisconsin (26,27), and a certified clubhouse modeled on Fountain House, Inc., in New York (28,29). Both provide supported employment services with a focus on rapid placement. A control condition was not necessary because controlled studies have already established supported employment per se as an evidence-based practice.

Our intent with this study was to ascertain, first, whether ACT and clubhouse programs can meet the performance benchmarks set for specialized supported employment teams while providing various additional services and serving other participants not interested in employment and, second, whether these two multiservice programs are effective in different ways. On the basis of previous research (20), we tested the hypotheses that ACT would have higher service engagement and retention rates than the clubhouse and that this superiority in engagement and retention would ensure higher job placement rates, especially for individuals with greater disability. On the other hand, because certified clubhouses are mandated to build strong relationships with local employers (30), we hypothesized that participants who obtained jobs through the clubhouse would earn higher wages and remain employed longer than ACT participants.

Data collection for this study was funded by the Substance Abuse and Mental Health Services Administration through the Employment Intervention Demonstration Program (EIDP) (31) from 1995 to 2000. Each participant’s employment was tracked for 24 months. Data analysis was then funded by the National Institute of Mental Health from 2001 to 2006. Institutional review board approval was provided by Fountain House, Inc. (1995–2000), and McLean Hospital (2001–2006).

Editor’s Note: The papers by Macias and colleagues and by Schonebaum and coworkers were submitted independently a year apart and reviewed separately. Neither set of authors knew of the others’ efforts. When we discovered that both used the same data set, we determined that the papers were complementary and decided to publish them in tandem. The two articles used different methods and a different choice of variables but came to the same overall conclusion. Each article makes a distinct contribution. Macias and colleagues present the independent assessment of evaluators, whereas Schonebaum and coworkers offer the perspective of researchers connected to Fountain House, which created the clubhouse model. Each validates findings of the other, turning this coincidence into what we think is an interesting lesson.

Methods

Experimental programs

The first experimental program of assertive community treatment was a mobile team designed by Leonard Stein, M.D., and Mary Ann Test, Ph.D., of Madison, Wisconsin, that provided out-of-office clinical care, assistance with housing and daily living, substance abuse intervention, and help in finding meaningful activities or employment (32,33). Dr. Stein and Jana Frey, Ph.D., of Madison mentored the experimental program evaluated in this study, and fidelity was verified annually by Dr. Frey and Gary Bond, Ph.D. The second program was a clubhouse run collaboratively by members and staff that emphasized mutual support, self-determination, and therapeutic benefits of voluntary and paid work (29). Various rehabilitation services, including case management, a work-ordered day, supported education, supported employment, transitional employment, and weekend social activities, were continuously available to clubhouse members, but attendance was not mandatory. The International Center for Clubhouse Development certified this second program and ensured fidelity to the standards for clubhouse programs (30).

In both programs, vocational staff who had training in supported employment (34,35) worked closely with other staff to ensure rapid placement into mainstream jobs not reserved by employers for individuals with disabilities. On-the-job training and support were provided whenever needed. Clubhouse members could also work transitional employment jobs, which were above-minimum-wage jobs reserved for the clubhouse by a consortium of local employers (30,36).

Sample characteristics

Sample recruitment was designed by representatives from local agencies and the National Alliance on Mental Illness chapter in Worcester, Massachusetts, with the aim of recruiting a heterogeneous sample of adults with serious mental illness. Referrals were accepted from 42 organizations, and participants were also recruited via flyers or radio and newspaper announcements. Applicants were eligible if they met criteria set by the EIDP: clinician diagnosis of severe mental illness (schizophrenia spectrum disorder, bipolar disorder, or recurrent major depression), currently unemployed, at least 18 years of age, and no recorded diagnosis of severe mental retardation. Except for the Maryland project (Anthony Lehman, principal investigator), other EIDP projects also screened for work interest. Our project could not do so because the ACT and clubhouse programs were designed to serve any adult with serious mental illness, regardless of work interest or ability to work.

A total of 465 applications were received, 310 of which met our eligibility requirements. All eligible applicants were screened for program preference and advised to enroll only if they agreed to participate in the program to which they were randomly assigned. Applicants’ program preferences were recorded for use as a control variable in outcome analyses (37). Of all eligible applicants 177 (57 percent) chose to enroll. Reasons for nonenrollment were almost evenly split between not wanting to risk assignment to a non-preferred program (70 applicants) versus fragility of mental or physical health, lack of interest in the program, or the perceived stigma of participation in a psychiatric study (63 applicants). Each enrollee then received a random assignment to the ACT or the clubhouse programs by drawing a slip from a box of slips, five labeled “ACT” and five labeled “clubhouse.” Three enrollees assigned to ACT were omitted from these analyses because of crossover to clubhouse services (one enrollee) or lack of verification that having no work data represented unemployment (two enrollees).

The intent-to-treat sample of 174 was similar to larger epidemiological samples within the same state (38) in terms of demographic characteristics and health problems (39), as well as in mortality rate (40). The two experimental programs were comparable on most background characteristics (Table 1), and there were no significant differences between participants who were interested or not interested in employment within the whole sample or either program.

Table 1

Characteristics of participants in assertive community treatment (ACT) and certified clubhouse programs

Research measures

Participant characteristics

Four participant background variables were chosen as control variables in all analyses on the basis of their correlation with work outcomes in previous research: age (41–44), severity of psychiatric symptoms (41,45–48), severity of physical health problems (46,49–51), and active substance use (8,46). Gender was added as a control variable because of its practical relevance and occasional correlation with work (42). Ethnicity was not statistically controlled for because this variable was comparably distributed across experimental conditions and work interest groups within conditions. Overall, of the participants interested in employment, 101 (84 percent) were Caucasian, ten (8 percent) were African American, seven (6 percent) were Hispanic, and three (2 percent) were Asian or Native American. In regard to employment, 57 (56 percent) Caucasians, five (50 percent) African Americans, four (57 percent) Hispanics, and one Asian (33 percent) were employed.

Psychiatric symptoms during the project were measured as total scores on the Positive and Negative Syndrome Scale (PANSS) (52) averaged across all interviews completed during each participant’s first 24 months with subscale scores equally weighted. Interviewers were trained by Lewis Opler, M.D., along with other EIDP interviewers and had high interrater reliability (53). Persistent or serious physical health problems (39) were identified through open-ended PANSS probes, as well as through Medicaid claims and interviewer reports, and then coded for severity with the Chronic Illness and Disability Payment System (54), which is based on actual treatment costs for a large multistate sample of Medicaid recipients. Substance use disorders (1, yes; 0, no) were identified through clinician reports, research interviews, and treatment records. Two motivational variables predictive of employment were also included as control variables (37,55): baseline interest in work (1, yes; 0, no or uncertain) and match of random assignment to program preference at the time of application (match to preference, mismatch to preference, or no prior preference). No data were missing on these background variables.

Employment

The two programs kept identical service logs and employment records designed by the EIDP (31). Two service variables, total hours of help with job searches (logged) and one hour or more of on-the-job support (1, yes; 0, no), were derived from daily logs kept by program staff for January 1996 through December 2000. Program employment records were corroborated with interview data and telephone calls to participants and family members. Employment outcomes for this study included all jobs lasting at least five days that met the U.S. Department of Labor’s definition of competitive employment (56,57): mainstream, integrated work paying at least minimum wage. These criteria exclude clubhouse enclave jobs in mainstream settings and in-program work provided by the ACT program but include individually held jobs in clubhouse transitional employment. However, we omitted transitional employment from these analyses so our findings would be comparable with those of other supported employment programs and the combined study outcomes of the EIDP (58).

Data analysis plan

Benchmark comparisons of program performance

We first compared both experimental programs with published outcome data for exemplary supported employment programs to confirm that our ACT and clubhouse programs met the performance standards set for specialized supported employment. Because the samples of all other supported employment studies have been composed almost entirely (95–100 percent) of participants interested in employment at the time of enrollment, we restricted our study sample to the 121 (70 percent) participants who expressed interest in employment when they enrolled in our project. This percentage is comparable with the 60 to 71 percent rates of interest in employment reported for other heterogeneous samples of adults with serious mental illness (43,59,60). About one-half (53 percent, or 28 persons) of the participants not wanting to work reported that they were unable to hold a job because of debilitating psychiatric symptoms, 19 percent (ten persons) reported having competing responsibilities, such as going to school fulltime or caring for infants or preschool children, 13 percent (seven persons) were morbidly obese, and 15 percent (eight persons) had life-threatening physical illnesses, such as AIDS, lymphoma, or hepatitis, which led to four deaths during the study period.

For benchmarks of work performance, we relied on total earnings and total hours of work reported for supported employment programs in five recent publications (Table 2). These five studies are the only supported employment randomized controlled trials that have reported descriptive statistics for work performance in addition to employment rates. All five experimental programs adopted the individual placement and support (IPS) model of supported employment (1,34). Supported employment studies typically calculate mean scores on work performance across all randomly assigned participants, entering values of zero for the work hours and earnings of unemployed participants. Because fewer zeros result in higher mean scores, the figures for “all participants” in Table 2 are composite benchmarks reflecting both work rates and work performance. The “employed only” figures are more distinct benchmarks of work performance.

Table 2

Total earnings and work hours of participants in assertive community treatment (ACT), clubhouse, and five benchmark supported employment programs^a

The first two comparison studies (6,9) listed in Table 2 were conducted concurrently with our study and used the same data collection methods and oversight from the same EIDP coordinating center. We truncated our data to fit the 18-month time frame of the three other studies (4,5,7). Only the 18-month study comparisons of earnings were subject to possible bias from temporal variation in minimum wage, because these non-EIDP studies began and ended one to three years earlier than the EIDP, when minimum wage was $4.25 per hour. By contrast, the minimum hourly wage increased uniformly across EIDP projects from 1996 to 2000: Massachusetts, from $4.25 to $5.25; South Carolina, from $4.25 to $5.15; Connecticut, from $4.27 to $6.15. The second benchmark, total work hours, provides an assessment of work performance that is independent of pay rate.

Although we limited our study sample to participants interested in employment to match the selection criteria of the five other programs, we could not control for additional exclusion criteria adopted by specific studies, such as the New Hampshire, Washington, D.C., and South Carolina studies’ exclusion of applicants who had medical illnesses that might interfere with employment (4,5,61) or the requirement by the New York study that every participant have a relative or close friend willing to participate in multifamily group treatment (7). Because the additional exclusion criteria favored the comparison programs over this study’s two experimental conditions, these benchmark comparisons are stringent tests of the quality of ACT and clubhouse supported employment.

Comparison of ACT and clubhouse outcomes

We tested our research hypotheses using a series of hierarchical regression analyses conducted with SPSS (version 11) (62). To compare rates of program engagement, we conducted a Cox regression (survival) analysis of calendar days from enrollment to active status in the assigned program. To compare the two programs on retention, we conducted a Cox regression analysis of calendar days from enrollment to attrition from the assigned program. We compared employment rates in a Cox regression analysis of days to first job. Participants who were never active, continuously active, or never employed were coded as censored on their 24-month anniversary dates. To compare the programs on work performance, we conducted three least-squares regression analyses, with calendar days of work, total hours of work, and total earnings as dependent variables. All dependent variables were log transformed. Work hours and earnings were Winsorized (63) for one clubhouse outlier, so that each value was reduced to the next highest value.

Each analysis relied on the entry of blocks of conceptually similar variables to control for multiple tests. The first block of each analysis controlled for match or mismatch of service assignment to service preference (reference category: no preference), along with vocational service receipt in all employment analyses. The second block controlled for participant characteristics. The third block tested program assignment as a predictor of outcomes when variables in the first two blocks were controlled for. No statistically significant variable was interpreted unless the omnibus test for that block was also significant (p<.05).

We focused on cumulative measures of work performance in cross-sectional analyses, rather than longitudinal analyses of individual change, because previous research has shown that employment is often short term and sporadic in this population and significant overall temporal change occurs only during the first six to nine months of supported employment (9,10). Also, the ceiling on weekly job hours and employer-imposed time limits, such as layoffs and temporary work, make cumulative measures across jobs more meaningful indicators of individual accomplishment than job-to-job incremental improvements.

Because distributions of continuous measures of retention and work performance had clumping at zero and the two programs differed in counts of zero values, we conducted two-stage analyses (64,65) analogous to two-stage mixed-effects, mixed-distribution methods for longitudinal modeling (66,67). Whole-sample threshold analyses (service engagement and employment rates) were followed by analyses of related continuous outcomes based on non-zero values (retention of engaged participants and work performance of employed participants). This two-stage procedure ensured that mean differences in non-zero values would not be confounded with differences in counts of zeros, allowing independent tests of oppositional hypotheses—that is, better work rates for the ACT program but better work performance for the clubhouse program.

Results

Comparison with benchmark employment outcomes

As shown in Table 2, clubhouse participants had comparable or higher mean total earnings than participants in all four exemplary supported employment programs that reported earnings for employees. Clubhouse participants also had higher mean total work hours than employed participants in two of the three studies that reported work hours. Outcomes of the clubhouse program were also comparable with those of three of the five programs that calculated work performance figures with values of zero entered for unemployed participants. Total earnings and work hours for ACT were comparable or higher than two of the four benchmark programs that reported means for employed participants and three of the five programs that entered values of zero for unemployed participants. Only the South Carolina EIDP study (9) reported total earnings and hours of work with more statistically appropriate median figures (not in table). For employed participants in that study, earnings and total hours worked (median = $2,855 earned and 485 hours worked) were comparable with clubhouse work performance (median = $3,456 earned and 494 hours worked) but higher than for ACT work performance (median = $1,252 earned and 234 hours worked). General comparability in total work hours between this study and earlier non-EIDP studies suggests that comparisons of earnings were minimally biased by cross-study variation in minimum wage. Among benchmark programs, mean hourly wage varied from $5.64 in the concurrent South Carolina study to $6.34 in the earlier New York study, compared with $6.45 for the ACT program and $7.94 for the clubhouse program.

Two benchmark studies reported total weeks of employment. Participants in the Washington, D.C., program were employed a mean of 16.5 weeks compared with mean work durations of 30.9 weeks for clubhouse and 21.4 weeks for ACT over 18 months. Participants in South Carolina’s combined individual placement and support and ACT programs were employed a median of 31.5 weeks compared with median durations of 29 weeks for clubhouse and 15 weeks for ACT over 24 months.

The 64 percent (N = 40) employment rate for the ACT program and 47 percent (N = 27) rate for the clubhouse program are within the 27–78 percent range reported for supported employment programs in other randomized controlled studies. The overall employment rate for the two multiservice programs combined was 55 percent (N = 67), which is nearly identical to the benchmark employment rates of 51 percent (68) and 55 percent (10) derived from meta-analyses, with the ACT rate being 9 to 13 percent higher and the clubhouse rate being 4 to 8 percent lower.

Comparison of ACT and clubhouse outcomes

We conducted full-sample (N = 174) omnibus regression analyses to check for interaction effects that would indicate a need for sample disaggregation (69,70). On the basis of previous research (55), we expected that baseline interest in employment would moderate program effectiveness. The interaction between program and work interest was statistically significant (p<.05) in the analyses of work duration, total hours worked, and total earnings. We tested study hypotheses separately for participants interested in employment and for those not interested in employment to interpret these interactions and to allow direct comparisons of study findings with the results of other supported employment projects.

Participants interested in employment at baseline

Table 3 presents the results of three Cox regression analyses for participants interested in employment. In keeping with a philosophy of assertive outreach, the ACT program had significantly better service engagement (98 percent, N = 62, versus 74 percent, N = 43) and 24-month retention of active participants (79 percent, N = 49, mean±SD = 650±180 days, versus 58 percent, N = 25, 569±228 days) than the clubhouse program, which had no required attendance and a philosophy of consumer self-determination. The Cox-regression model for time to first job showed no significant program difference. One-half of both ACT and clubhouse participants began their first jobs within six months (median = 6.67 months versus 4.23 months, respectively). Although the absolute difference in program work rates (ACT, 64 percent, N = 40; clubhouse, 47 percent, N = 27) approached significance (p=.06) in an uncontrolled logistic regression analysis, this trend toward significance was reduced when participants’ service preferences and background characteristics were statistically controlled for (p=.33).

Table 3

Cox regression analyses to predict service participation and employment outcomes of assertive community treatment and clubhouse programs over a 24-month period (N=121)

In the controlled linear regression analyses (Table 4), compared with the 40 ACT participants, the 27 clubhouse participants were employed significantly longer (264± 214 days, median = 199, versus 173± 164 days, median = 98) for more total hours (784±717 hours, median = 494, versus 592±695 hours, median = 234) and earned more ($6,202±$5,894, median = $ 3,456, versus $3,948±$4,888, median = $1,252).

Table 4

Linear regression analyses to predict employment outcomes of assertive community treatment and clubhouse programs over a 24-month period (N=67)

Participant background variables were generally not predictive of engagement, attrition, or employment, but receipt of job search services, such as career planning, networking, job hunting, job development, and transportation to interviews, predicted higher work rates (Wald = 5.77, hazard ratio = 1.16).

Wage as an incentive for work success

When average pay rate was added as a fourth block to each linear regression analysis (not in table), the beta for this variable was significant (p<.001) and the beta for program assignment was substantially reduced. This result suggests that one reason clubhouse participants remained employed longer, worked more total hours, and earned more than ACT participants is because they generally held higher-paying jobs. Clubhouse participants earned about $1.50 more per hour each week than ACT participants ($7.94±$2.71 versus $6.45±$1.66; t = 2.79, df = 65, p<.01), although participants in both programs earned well above the minimum wage.

Participation in transitional employment

Five clubhouse participants in these analyses also held transitional employment jobs, three of whom first worked in a transitional employment job for close to a year, then accepted a permanent position with the same employer. Nevertheless, participants in transitional employment had about the same mean tenure on non–transitional employment jobs as other employed clubhouse participants (207±121 days versus 276±230 days). Nor did the program comparison findings change when transitional employment was included as an outcome. There was no program difference in time to first job, and clubhouse participants worked longer and earned more than ACT participants (p<.05). This was most likely because six-month time limits on transitional employment were not strictly observed, and transitional employment jobs resembled non–transitional employment jobs in that they were individually held jobs in local businesses that paid above the minimum wage (mean pay $6.85±$1.14 per hour; median pay $7 per hour). However, the program difference in total work hours was not significant (p=.06) when transitional employment was included, because transitional employment jobs, by definition, were all half-time positions (30).

Participants not interested in employment at baseline

As with the work interest sample, the Cox regression analysis showed that ACT participants engaged in services faster than clubhouse participants (p<.05). Cox regression models for time to attrition and first job did not reach statistical significance, and because only 12 of the 53 participants with no work interest held a non–transitional employment job (five clubhouse participants and seven ACT participants), there was insufficient power to conduct program comparisons of work performance. However, once employed, these 12 participants who had been reluctant to work stayed employed more days (median = 172 days versus 124 days) and worked more total hours (median = 474 hours versus 288 hours) than participants who were interested in employment when they enrolled in the project, with earnings (median = $3,173 versus $2,000) that approximated benchmark outcomes for supported employment (Table 2).

Supported employment programs for specific types of participants

The question as to whether the ACT program was more vocationally effective for participants who were in greater need of clinical care was addressed with moderated multiple regression by dividing the sample into subgroups based on type and level of disability. The strong integration of vocational and clinical services by ACT appeared to be particularly effective for participants who had co-occurring physical health problems or severe substance use disorders, while the clubhouse was more effective for participants who did not have these co-occurring disorders. [A summary of these findings is available online at ps.psychiatryonline.org]

Discussion

The relatively good employment rates and work performance for the ACT and clubhouse programs suggest that integrating supported employment into multiservice programs is a feasible way to achieve wider dissemination of supported employment without reducing vocational service quality. We do not present cost data in this article, but the comparatively high operating cost for specialized employment programs (71,72) suggests that providing supported employment through multiservice certified clubhouses would be especially cost-effective (73), and vocationally integrated ACT could be reserved for consumers who need intensive clinical care. This assumption needs to be tested within the context of total service system costs.

Regardless of cost concerns, our findings suggest that the availability of supported employment should be expanded to reach individuals who lack the confidence to enroll in a specialized supported employment program but who might be willing to receive such services in a less formal, more spontaneous way from generalist staff they already know well. Participants who lacked a baseline interest in work were unlikely to become employed, but those who did work stayed employed longer and earned more than participants who had expressed interest in work when they enrolled in this and the other supported employment studies listed in Table 2. The success of these particular consumers suggests that multiservice programs that do not focus exclusively on employment may ameliorate the apprehension that prevents some individuals from entering supported employment.

As predicted, the ACT program was uniquely effective in keeping participants engaged in program services, and the clubhouse program was uniquely effective in keeping participants employed. Overall, participants interested in working who were assigned to the clubhouse program stayed employed longer, worked more total hours, and earned more—a result partially attributable to higher-paying jobs. Only one research hypothesis was not fully supported. The ACT and clubhouse programs did not differ significantly in overall work rates when the analysis controlled for background characteristics and program assignment preference.

There were several limitations to the study. Statewide funding of a new supported employment initiative in the project’s second year may have restricted local work opportunities for both ACT and clubhouse participants so that study findings underestimate the potential for these two multiservice programs to provide supported employment. The study design also was not ideal for assessing the effectiveness of either ACT or clubhouse because study variables were selected by the EIDP steering committee. The main intent of the EIDP was to evaluate the specialized supported employment teams that were the focal interventions for most EIDP projects. Many clubhouse participants classified as unemployed in our study actually worked individually held, above-minimum-wage jobs in mainstream businesses that met the Department of Labor’s definition of competitive employment (56) but not the definition set by the EIDP (10). Whether these individuals would have worked at all if they were enrolled in a specialized supported employment program rather than a clubhouse is a question for future research.

Likewise, our study cannot answer the question of whether the relatively good employment outcomes for ACT participants who had health problems or severe substance use disorders were attributable to the quality of ACT clinical care, substance abuse treatment, and employment supports or to the intensity, continuity, and integration of these services. The ACT and clubhouse models are designed to differ on all of these service characteristics. The models were similar in that vocational staff in both programs performed generalist roles. We could learn more about ACT effectiveness through comparisons with other interdisciplinary teams that differ from ACT on only one or two of these distinct measures. For instance, would these particular ACT participants have had equally favorable employment outcomes if they had been assigned to supported employment specialists who met frequently with psychiatrists, case managers, and substance abuse treatment professionals from the same agency (74,75)?

Beyond these limitations to the research design, the reader is cautioned that generalizability of the study findings is limited to ACT and clubhouse programs that have high fidelity to these two service models. Generalizability is also limited to the predominantly Caucasian, urban region of the northeastern United States where the study was conducted. This geographical limitation is mitigated by the sample’s heterogeneity and demonstrated representativeness of adults with serious mental illness who reside in the state of Massachusetts (39).

Conclusions

Although this study is the first randomized controlled comparison of the ACT and clubhouse models and the first randomized controlled evaluation of a certified clubhouse, overall study outcomes may be even more important than program differences. Whether someone went to work was not predicted by program assignment per se but rather by whether the participant was assigned to the program he or she preferred and the amount of job search services subsequently received. Cross-study comparisons suggest that both vocationally integrated ACT and certified clubhouses can achieve employment outcomes that are generally comparable with those of the specialized teams that represent the evidence for evidence-based supported employment.

Acknowledgments

Data were collected from 1996 to 2000 through grant SM-51831 from the Substance Abuse and Mental Health Service Administration. Data analysis was supported from 2001 to 2006 by interdisciplinary research grant RO1-MH-62628 from the National Institute of Mental Health (NIMH). This project would not have been possible without supplemental support from the state of Massachusetts, the University of Massachusetts Medical School, the van Ameringen Foundation, and the John D. and Catherine T. MacArthur Foundation. These views represent the opinions of the authors and not necessarily those of the supporting agencies. The authors thank their NIMH project monitor, Ann Hohmann, Ph.D., and social psychology coinvestigators. Appreciation is extended to members of the project’s advisory council for oversight of sample recruitment, especially National Alliance on Mental Illness members Mary Query, Dorothy LaPointe, Grace LaPearl, and Ann Healy, and Kenneth Hetzler, M.D. The authors also thank project staff Debra Giza, Kathy Smith, M.S.W., and Julia Vera, Andrew Schonebaum, M.S.W., and Colleen McKay, M.A., and the late Ronald F. Kinney, Jr., M.S. Appreciation also to the directors, staff, and consumers of the two focal interventions, ACT at Community Healthlink, Inc., and Genesis Club, Inc., in Worcester for diligent record keeping. Valuable insights into service model operations were provided by Leonard Stein, M.D., and Jana Frey, Ph.D., for ACT, and by Rudyard Propst for the clubhouse.

Contributor Information

Dr Cathaleene Macias, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106 (Email: cmacias/at/mclean.harvard.edu)

Mr Charles F. Rodican, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106.

Dr William A. Hargreaves, The Department of Psychiatry, University of California, San Francisco.

Dr Danson R. Jones, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106.

Dr Paul J. Barreira, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106.

Dr Qi Wang, Community Intervention Research, McLean Hospital, 115 Mill Street, Belmont, MA 02478-9106.

References

1. Bond G. Supported employment: evidence for an evidence-based practice. Psychiatric Rehabilitation Journal. 2004;27:345–359. [PubMed]

2. Drake RE, Becker DR. The Individual Placement and Support model of supported employment. Psychiatric Services. 1996;47:473–475. [PubMed]

3. Bond GR, Becker DR, Drake RE, et al. Implementing supported employment as an evidence-based practice. Psychiatric Services. 2001;52:313–322. [PubMed]

4. Drake RE, McHugo GJ, Becker DR, et al. The New Hampshire study of supported employment for people with severe mental illness. Journal of Consulting and Clinical Psychology. 1996;64:391–399. [PubMed]

5. Drake RE, McHugo GJ, Bebout RR, et al. A randomized clinical trial of supported employment for inner-city patients with severe mental disorders. Archives of General Psychiatry. 1999;56:627–633. [PubMed]

6. Mueser KT, Clark RE, Haines M, et al. The Hartford study of supported employment for persons with severe mental illness. Journal of Consulting and Clinical Psychology. 2004;72:479–490. [PubMed]

7. McFarlane W, Dushay R, Deakins S, et al. Employment outcomes in family-aided assertive community treatment. American Journal of Orthopsychiatry. 2000;70:203–214. [PubMed]

8. Lehman AF, Goldberg RW, Dixon LB, et al. Improving employment outcomes for persons with severe mental illnesses. Archives of General Psychiatry. 2002;59:165–172. [PubMed]

9. Gold PB, Meisler N, Santos AB, et al. Randomized trial of supported employment integrated with assertive community treatment for rural adults with severe mental illness. Schizophrenia Bulletin. 2006;32:378–395. [PMC free article] [PubMed]

10. Cook JA, Leff HS, Blyler CR, et al. Results of a multisite randomized trial of supported employment interventions for individuals with severe mental illness. Archives of General Psychiatry. 2005;62:505–512. [PubMed]

11. Crowther RE, Marshall M, Bond GR, et al. Helping people with severe mental illness to obtain work: systematic review. British Medical Journal. 2001;322:204–208. [PMC free article] [PubMed]

12. Corrigan PW. Place-then-train: an alternative service paradigm for persons with psychiatric disabilities. Clinical Psychology: Science and Practice. 2001;8:334–349.

13. Xie H, Dain B, Becker D, et al. Job tenure among persons with severe mental illness. Rehabilitation Counseling Bulletin. 1997;40:230–239.

14. Salyers MP, Becker DR, Drake RE, et al. A ten-year follow-up of a supported employment program. Psychiatric Services. 2004;55:302–308. [PubMed]

15. Becker DR, Drake RE, Bond GR, et al. Job terminations among persons with severe mental illness participating in supported employment. Community Mental Health Journal. 1998;34:71–82. [PubMed]

16. Latimer EA. Economic impacts of supported employment for persons with severe mental illness. Canadian Journal of Psychiatry. 2001;46:496–505. [PubMed]

17. Drake RE, Becker DR, Bond GR, et al. A process analysis of integrated and non-integrated approaches to supported employment. Journal of Vocational Rehabilitation. 2003;18:51–58.

18. Mowbray C, Bybee D, Collins ME. Integrating vocational services on case management teams: outcomes from a research demonstration project. Mental Health Services Research. 2000;2:51–66. [PubMed]

19. Murphy AA. Enhancing individual placement and support: promoting job tenure by integrating natural supports and supported education. American Journal of Psychiatric Rehabilitation. 2005;8:37–61.

20. Dixon L. Assertive community treatment: twenty-five years of gold. Psychiatric Services. 2000;51:759–765. [PubMed]

21. Test MA, Knoedler W, Allness DJ, et al. In: Comprehensive community care of persons with schizophrenia through the Programme of Assertive Community Treatment (PACT), in Toward a Comprehensive Therapy for Schizophrenia. Brenner HD, Boker W, Genner R, editors. Seattle, Wash: Hogrefe and Huber; 1997.

22. Warner R, Huxley P, Berg T. An evaluation of the impact of clubhouse membership on quality of life and treatment utilization. International Journal of Social Psychiatry. 1999;45:310–320. [PubMed]

23. Dorio J, Guitar A, Solheim L, et al. Differences in job retention in a supported employment program: Chinook Clubhouse. Psychiatric Rehabilitation Journal. 2002;25:289–298. [PubMed]

24. Stein LI, Barry KL, Van Dien G, et al. Work and social support: a comparison of consumers who have achieved stability in ACT and clubhouse programs. Community Mental Health Journal. 1999;35:193–204. [PubMed]

25. Reed SJ, Merz MA. Integrated service teams in psychiatric rehabilitation: a strategy for improving employment outcomes and increasing funding. Journal of Applied Rehabilitation Counseling. 2000;31:40–46.

26. Russert MG, Frey JL. The PACT Vocational Model: a step into the future. Psychosocial Rehabilitation Journal. 1991;14:8–18.

27. Frey JL. Long term support: the critical element to sustaining competitive employment: where do we begin? Psychosocial Rehabilitation Journal. 1994;17:127–133.

28. Beard JH, Propst RN, Malamud TJ. The Fountain House model of psychiatric rehabilitation. Psychosocial Rehabilitation Journal. 1982;5:1–12.

29. Anderson SB. We Are Not Alone: Fountain House and the Development of Clubhouse Culture. New York: Fountain House; 1998.

30. Propst R. Standards for clubhouse programs: why and how they were developed. Psychosocial Rehabilitation Journal. 1992;16:25–30.

31. Cook JA, Carey MA, Razzano L, et al. The pioneer: the Employment Intervention Demonstration Program. New Directions in Evaluation. 2002;no 94:31–44.

32. Allness D, Knoedler W. The PACT Model: A Manual for PACT Start-Up. Arlington, Va: National Alliance for the Mentally Ill; 1998.

33. Stein LI, Test MA. Alternative to mental hospital treatment: I. conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry. 1980;37:392–397. [PubMed]

34. Bond GR, Vogler K, Resnick SG, et al. Dimensions of supported employment: factor structure of the IPS fidelity scale. Journal of Mental Health. 2001;10:383–393.

35. Trach JS. Supported employment program characteristics. In: Rusch R, editor. Supported Employment: Models, Methods and Issues. Sycamore, Ill: Sycamore Press; 1990.

36. Lodin S. The meaning of transitional employment. Psychosocial Rehabilitation Journal. 1992;16:85–86.

37. Macias C, Barreira P, Hargreaves W, et al. Impact of referral source and study applicants’ preference for randomly assigned service on research enrollment, service engagement, and evaluative outcomes. American Journal of Psychiatry. 2005;162:781–787. [PMC free article] [PubMed]

38. Dickey B, Normand S-L, Weiss R, et al. Medical morbidity, mental illness, and substance use disorders. Psychiatric Services. 2002;53:861–867. [PubMed]

39. Jones DR, Macias C, Barreira PJ, et al. Comparing the prevalence, severity, and co-occurrence of chronic physical health problems of persons with serious mental illness. Psychiatric Services. 2004;55:1250–1257. [PMC free article] [PubMed]

40. Dembling BP, Chen DT, Vachon L. Life expectancy and causes of death in a population treated for serious mental illness. Psychiatric Services. 1999;50:1036–1042. [PubMed]

41. Goldberg RW, Lucksted A, McNary S, et al. Correlates of long-term unemployment among inner-city adults with serious and persistent mental illness. Psychiatric Services. 2001;52:101–103. [PubMed]

42. Cook JA, Pickett-Schenk SA, Gray D, et al. Vocational outcomes among formerly homeless persons with severe mental illness in the ACCESS program. Psychiatric Services. 2001;52:1075–1080. [PubMed]

43. Mueser KT, Salyers MP, Mueser PR. A prospective analysis of work in schizophrenia. Schizophrenia Bulletin. 2001;27:281–296. [PubMed]

44. Wewiorski NJ, Fabian ES. Association between demographic and diagnostic factors and employment outcomes for people with psychiatric disabilities: a synthesis of recent research. Mental Health Services Research. 2004;6:9–21. [PubMed]

45. Anthony WA, Rogers ES, Cohen M, et al. Relationships between psychiatric symptomatology, work skills, and future vocational performance. Psychiatric Services. 1995;46:353–358. [PubMed]

46. Razzano LA, Cook JA, Burke J, et al. Clinical factors associated with employment among people with severe mental illness: findings from the employment intervention demonstration program. Journal of Nervous and Mental Disease. 2005;193:705–713. [PubMed]

47. Slade E, Salkever D. Symptom effects on employment in a structural model of mental illness and treatment: analysis of patients with schizophrenia. Journal of Mental Health Policy and Economics. 2001;4:25–34. [PubMed]

48. Regenold M, Sherman M, Fenzel M. Getting back to work: self-efficacy as a predictor of employment outcome. Psychiatric Rehabilitation Journal. 1999;22:361–367.

49. Razzano L, Hamilton MM. Health-related barriers to employment among people with HIV/AIDS. Journal of Vocational Rehabilitation. 2005;22:179–188.

50. Dixon L, Goldberg R, Lehman A, et al. The impact of health status on work, symptoms, and functional outcomes in severe mental illness. Journal of Nervous and Mental Disease. 2001;189:17–23. [PubMed]

51. Druss BG, Marcus SC, Rosenheck RA, et al. Understanding disability in mental and general medical conditions. American Journal of Psychiatry. 2000;157:1485–1491. [PubMed]

52. Kay SR, Fiszbein A, Opler LA. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophrenia Bulletin. 1987;13:261–276. [PubMed]

53. Salyers MP, McHugo GH, Cook JA, et al. Reliability of instruments in a cooperative, multisite study: Employment Intervention Demonstration Program. Mental Health Services Research. 2001;3:129–139. [PubMed]

54. Kronick R, Gilmer T, Dreyfus T, et al. Improving health-based payment for Medicaid beneficiaries: CDPS. Health Care Financing Review. 2000;21:29–64. [PubMed]

55. Macias C, DeCarlo L, Wang Q, et al. Work interest as a predictor of competitive employment: policy implications for psychiatric rehabilitation. Administration and Policy in Mental Health. 2001;28:279–297. [PubMed]

56. Department of Labor. Job Training Program Act, Disability Grant Program Funded Under Title III, Section 323, and Title IV, Part D, Section 452. 63 Fed Reg. 1998.

57. Workforce Investment Act of 1998. Pub L. pp. 105–220.

58. Cook JA, Leff HS, Blyler CR, et al. Results of a multisite randomized trial of supported employment interventions for individuals with severe mental illness. Archives of General Psychiatry. 2005;62:505–512. [PubMed]

59. McQuilken M, Zahniser JH, Novak J, et al. The Work Project Survey: consumer perspectives on work. Journal of Vocational Rehabilitation. 2003;18:59–68.

60. Chandler D, Meisel J, Hu T-W, et al. A capitated model for a cross-section of severely mentally ill clients: employment outcomes. Community Mental Health Journal. 1997;33:501–516. [PubMed]

61. Williams OH, Meisler N, Gold PB, et al. Charleston, State of South Carolina: Department of Mental Health and Medical University of South Carolina; 2001. South Carolina Employment Intervention Demonstration Project: rural-based supported employment approaches.

62. SPSS for Windows. version 11. Chicago: SPSS; 1999.

63. Dixon WJ, Tukey JW. Approximate behavior of the distribution of Winsorized t (trimming/Winsorization 2) Technometrics. 1968;10:83–98.

64. Delucchi KL, Bostrom A. Methods for analysis of skewed data distributions in psychiatric clinical studies: working with many zero values. American Journal of Psychiatry. 2004;161:1159–1168. [PubMed]

65. Lachenbruch PA. Analysis of data with excess zeros. Statistical Methods in Medical Research. 2002;11:297–302. [PubMed]

66. Xie H, McHugo GH, Sengupta A, et al. A method for analyzing longitudinal data with many zeros. Mental Health Services Research. 2004;6:239–246. [PubMed]

67. Tooze JA, Grunwald GA, Jones RH. Analysis of repeated measures data with clumping at zero. Statistical Methods in Medical Research. 2002;11:341–355. [PubMed]

68. Twamley E, Jeste DV, Lehman A. Vocational rehabilitation in schizophrenia and other psychotic disorders: a literature review and meta-analysis of randomized controlled trials. Journal of Nervous and Mental Disease. 2003;191:515–523. [PubMed]

69. Bryk AS, Raudenbush SW. Heterogeneity of variance in experimental studies: a challenge to conventional interpretations. Psychological Bulletin. 1988;104:396–404.

70. Lyons J, Howard K. Main effects analysis in clinical research: statistical guidelines for disaggregating treatment groups. Journal of Consulting and Clinical Psychology. 1991;59:745–748. [PubMed]

71. Latimer EA, Bush PW, Becker DR, et al. The cost of high-fidelity supported employment programs for people with severe mental illness. Psychiatric Services. 2004;55:401–406. [PubMed]

72. Dixon L, Hoch J, Clark RE, et al. Cost-effectiveness of two vocational rehabilitation programs for persons with severe mental illness. Psychiatric Services. 2002;53:1118–1124. [PubMed]

73. Macias C, Barreira P, Alden M, et al. The ICCD benchmarks for clubhouses: a practical approach to quality improvement in psychiatric rehabilitation. Psychiatric Services. 2001;52:207–213. [PubMed]

74. Cook JA, Lehman AF, Drake RE, et al. Integration of psychiatric and vocational services: a multisite randomized, controlled trial of supported employment. American Journal of Psychiatry. 2005;162:1948–1956. [PubMed]

75. Cook JA, Razzano LA, Burke-Miller JK, et al. Reply to letter to the editor: Integration of mental health care and supported employment. American Journal of Psychiatry. 2006;163:940.