|Home | About | Journals | Submit | Contact Us | Français|
Client functioning and treatment engagement were examined in relation to staff attributes and organizational climate across a diverse sample of drug treatment and outreach programs in England. Self-rating assessments were obtained from 1,539 clients and 439 counselors representing 44 programs, and results were interpreted using comparable data from studies of treatment programs in the United States. Client scores on treatment participation and counseling rapport in England were directly related to their higher levels of motivation and psychosocial functioning, as well as to staff ratings of professional attributes and program atmosphere. By linking records from English clients with their counselors in each program, findings also indicate these relationships are rooted in the personal interactions between clients and their counselor. Standardized assessments of treatment structure, process, and performance used across therapeutic settings and national boundaries show there is generalizability in the pattern of clinical dynamics, including the relationships between organizational functioning and quality of services.
International pressures to identify and implement evidence-based health care practices are growing, spurred by concerns about effectiveness of these services. For instance, substance use treatment systems in the United States and United Kingdom seek more consistent dissemination and implementation of these practices to raise standards of care (Institute of Medicine, 2006; National Treatment Agency, 2005). These objectives require better sequential action models for identifying and disseminating enhancement protocols as well as better performance assessments for measuring and monitoring treatment delivery systems and outcomes.
A barrier to reaching these goals in the U.S. has been organizational instability related to leadership and staff turnover (McLellan, Carise, & Kleber, 2003). Not surprisingly, it also is becoming clearer from the literature that some substance use treatment programs do a better job of engaging and retaining clients than others (Broome, Simpson, & Joe, 1999; Joe, Broome, Rowan-Szal, & Simpson, 2002). Such programs likewise show better gains in psychosocial functioning by their clients during the course of treatment delivery and after its completion (Gossop, 2006, May; Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007; Simpson, 2004; Simpson et al., 1997). To help understand why some are more effective than others, treatment process research has expanded in its focus during recent years to give greater attention to organizational aspects of service settings and related staff characteristics (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Greenhalgh, Robert, MacFarlane, Bate, & Kryriakidou, 2004; Simpson, 2002; Simpson & Flynn, 2007a). Special interests include factors such as program management, treatment setting, counselor education and training, and characteristics of the clients being served (D’Aunno & Vaughn, 1995; Etheridge, Hubbard, Anderson, Craddock, & Flynn, 1997; Friedmann, Alexander, & D’Aunno, 1999; Heinrich & Lynn, 2002; Institute of Medicine, 2006; Knight, Broome, Simpson, & Flynn, 2008; Roman & Johnson, 2002; Sun, 2006).
Evidence showing organizational context and functioning are relevant to quality of treatment services is growing. Moos and Moos (1998) found that a supportive and goal-directed treatment program climate is related to improved client treatment participation, greater satisfaction with treatment, and better treatment outcomes at discharge. Quality of organizational functioning likewise influences the use, effectiveness, and efficiency of drug treatment interventions (Broome, Flynn, Knight, & Simpson, 2007; Lehman, Greener, & Simpson, 2002). Therefore, several recent studies of treatment adoption and implementation have measured organizational structure, climate, and staff attributes of treatment organizations by using the Organizational Readiness for Change (ORC) questionnaire. Lehman et al. (2002) described its rationale and original scale development, together with its psychometric properties. Several of these scales are predictive of treatment satisfaction and client-practitioner rapport. While the ORC has been examined in national and statewide surveys of addiction services in the U.S. (Fuller et al., 2007; Saldana, Chapman, Henggeler, & Rowland, 2007), it has only recently been applied in Europe, including research on its cross-cultural applicability in the Veneto region of Italy (Rampazzo, De Angeli, Serpelloni, Simpson, & Flynn, 2006).
Training and staff preparations for organizational change in relation to client performance are now being examined in England, consistent with guidance from growing international evidence for the impact of improving treatment engagement and retention. It is being conducted under the auspices of the National Treatment Agency for Substance Misuse (NTA; National Treatment Agency, 2004), a special health authority established within the National Health Service in 2001. The agency mission is to improve the availability, capacity, and effectiveness of treatment for drug use (National Treatment Agency, 2005). Retaining and engaging individuals in treatment services are high priorities, and some of the tools and techniques summarized below are being adapted for a British context from models developed by the Institute of Behavioral Research at Texas Christian University (Simpson, 2002, 2004).
NTA innovations being developed and implemented include psychosocial and cognitive-based interventions for the International Treatment Effectiveness Project (ITEP), which has been shown to increase client participation in treatment, rapport with keyworkers (i.e., counselors), and perceived support from peers in treatment (National Treatment Agency, 2007). In this research, assessment of client engagement in treatment (satisfaction, participation, and counseling rapport) has been made through the Client Evaluation of Self and Treatment (CEST; Joe et al., 2002). The importance of modeling dynamic components of treatment engagement is established through programmatic studies showing that stronger therapeutic relationships between counselors and clients are related to more frequent attendance at sessions and less drug use during treatment (Joe, Simpson, Greener, & Rowan-Szal, 1999; Simpson, Joe, Rowan-Szal, & Greener, 1995), as well as to less drug use and criminal involvement at post-treatment follow-up (Simpson, Joe, Greener, & Rowan-Szal, 2000). Joe, Simpson, Dansereau, and Rowan-Szal (2001) likewise have shown that higher counseling rapport during treatment predicts better post-treatment outcomes, defined by less drug use and criminal activity.
The purpose of the present study was to examine client-level and program-level functioning assessments addiction services and their clients sampled from diverse treatment settings in England. In particular, it was of interest to determine if client motivation and psychosocial functioning as well as organizational functioning domains (representing staff attributes and organizational climate) are related to treatment engagement indicators in a manner similar to those observed for clients in the U.S. As the U.K. treatment system is being pressed to develop and implement service innovations (focused especially on heroin users) for delivering a variety of harm reduction procedures, studies of the process involved are expected to help identify generalizable enhancement strategies.
Data were obtained in collaboration with treatment agencies located in the North West and West Midlands regions of England, coordinated through the National Treatment Agency (NTA) over a period of 2 years. The subject pool consisted of clinical staff and clients at participating drug services. Some agencies included multiple treatment units — that is, different program sites under the oversight of the same parent organization. Because individual treatment units have their own unique organizational attributes and climate, these units were assessed separately. Methods and procedures for collecting data were carried out in accordance with protocols approved by the Institutional Review Board at Texas Christian University and by regional authorities responsible for data collection in England. Participation in the assessments was voluntary and a passive consent procedure was used in which completing an assessment and returning it implied consent. The NTA research staff (in collaboration with TCU staff) edited the original CEST and ORC forms for British spelling and semantics and also coordinated data collection activities. Data files were later shared with TCU staff for conjoint responsibilities in conducting analyses.
A package of CEST assessments was administered in 2006 at each participating treatment unit. The services consisted of both voluntary (private) and statutory (National Health Service, NHS) agencies and offered treatment services for both alcohol and other drug clients. The agencies represented a broad spectrum of community services, ranging from day care, outreach programs, community drug teams, and Drug Intervention Programme (DIP) teams. Typically, treatment units received forms based on the approximate number of clients currently in treatment at the facility. These were to be administered sequentially to clients as they presented for treatment services. A total of 1,539 CEST forms were collected and completed during the same time period (2-week window) that staff completed ORC assessments. This resulted in an average of approximately 33 CEST forms per treatment unit. Although less than half of the total clients in many of the participating programs were assessed in this cross-sectional and time-limited sampling design, it offered a more focused snapshot of service delivery systems than using a long-term sampling strategy that can be complicated by system changes. Based on client self-reported demographic data, 78% were Caucasian, 8% were Black, and 14% listed as Other race. Average age of clients was 35 years, and 68% were male. Thirteen percent were in treatment for under a month, 18% were in treatment for 1 to 3 months, 27% in for more than 3 months (but less than a year), and 42% had been in treatment for a year or longer.
A total of 439 ORC assessments were collected from staff at the 44 treatment units located in the Greater Manchester, Birmingham, and Wolverhampton area of England that volunteered to be part of this study. An average of 10 ORC forms was collected from each treatment unit. Based on demographic information, the staff members reported being 82% Caucasian, 10% Black, 4% Asian, and 4% other. Seventy-three percent were female, and their average age was 38 years. About a third (35%) had at least 5 years of experience in drug treatment services, and a majority (74%) had been on their present job for 3 years or less (40% were in their present position for 1 to 3 years).
A recent study reported by Greener et al. (2007) includes similar data from a large sample of treatment programs in the United States that could be used for making general comparative interpretations of the U.K. findings. It was based on participating agencies from nine states and includes outpatient drug free, residential, methadone maintenance, and therapeutic community programs. More than half were free-standing clinics, along with a number of community mental health centers and hospitals or university settings also represented. A total of 163 programs provided both CEST and ORC records that were cross-linked and aggregated for use in correlational analyses between program-level and client-level indicators. This included 3,475 CEST forms completed by clients and 531 ORC forms completed by staff from these 163 programs.
The client sample was 60% male and the race-ethnic composition was 15% Hispanic, 21% African American, 58% Caucasian, and 5% other or not reported. At the time of the survey, 38% of clients were in treatment less than 30 days, 27% had been treated 31-90 days, 20% had been treated 91-360 days, and 6% had been in treatment over a year. The staff sample was 73% Caucasian, 16% African American, and 5% Hispanic; 63% were female. About 65% had a bachelor’s degree or higher, with 28% having a Master’s. Just over half (54%) had at least 5 years of experience in drug abuse counseling, and 28% had been on their present job for at least 5 years (with about 23% on their present job for less than a year).
The CEST includes 16 scales (each defined by 5 to 12 items) representing client motivation and readiness for treatment, psychological and social functioning, and treatment engagement (Joe et al., 2002). Motivation for treatment factors include desire for help, treatment readiness, and treatment needs, while psychological domains include self esteem, depression, anxiety, decision making, and self efficacy. Social domains include hostility, risk taking, and social consciousness (i.e., representing personal acceptance of social norms). Finally, treatment process scales include treatment satisfaction, counselor rapport, and treatment participation, along with client perceptions of peer support and social support (from family members).
Scale scores are calculated based on client self-administered ratings using 5-point Likert-type items (agree-disagree). Responses on items for each scale are averaged and then multiplied by 10. Means could range from 10 to 50, with scores above 30 indicating overall agreement (and higher scores indicating stronger agreement). For the U.K. sample, coefficient alpha reliabilities on the 16 scales of the CEST showed six fell below an alpha level of .70; these included treatment needs (.69), self efficacy (.68), risk taking (.68), treatment readiness (.60), desire for help (.59), and social consciousness (.43). By comparison, the U.S. sample include two scales with alpha values below .70 (i.e., self efficacy, .63, and social consciousness, .52).
The ORC assessment of organizational needs and functioning (Lehman et al., 2002) includes 18 scales (defined by 4 to 8 items each) from four major domains — needs and motivation for change, resources, staff attributes, and organizational climate. Needs and pressures (motivation for change) factors include program needs, training needs, and pressures for change, while program resources are represented in regard to office facilities, staffing, training, equipment, and Internet access. Organizational functioning dynamics include scales on staff attributes (growth, efficacy, influence, and adaptability) and program climate (mission, cohesion, autonomy, communication, stress, and openness for change). These are indicators that can help programs identify potential problem areas (Simpson & Dansereau, 2007).
In computing scale scores, counselor ratings using 5-point Likert-type items (agree-disagree) for each scale are averaged and then multiplied by 10. Means could range from 10 to 50, with scores above the neutral point of 30 indicating overall agreement (and higher scores indicating stronger agreement). Coefficient alpha reliabilities for the ORC scales from the U.K. sample showed 11 of the 18 scales had values below .70. These included pressures for change, equipment, adaptability, and autonomy in the range of .50 to .60, and staffing, training, internet, growth, efficacy, mission, and change in the range of .61 to.70. For the U.S. sample reported by Greener et al. (2007), seven scales have coefficient alphas below .70, including pressure for change, offices, growth, and efficacy at .68 or .69, training and adaptability at .63 or .64, and autonomy at .56. (More detailed psychometric information and scale score distributions for U.S. samples also are available at www.ibr.tcu.edu, listed under “Forms”.)
Motivation scales from the CEST for the U.K. sample ranged from 32.5 (sd = 7.3) for treatment needs to 39.8 (sd = 5.2) for treatment readiness. Psychological functioning scales ranged from 30.6 (sd = 7.8) for self esteem to 34.7 (sd = 5.3) for decision making, while social functioning scales ranged from 25.1 (sd = 8.2) for hostility to 35.1 (sd = 4.8) for social consciousness. Finally, treatment engagement scales ranged from 34.5 (sd = 6.9) for peer support to 41.8 (sd = 5.6) for counselor rapport.
Although client self-evaluations by the U.K. and U.S. samples were examined statistically for distributional similarities before conducting correlational analyses, the samples were not drawn from programs with the same types of field services and therefore direct cross-cultural interpretations of mean score differences on the CEST are premature. Therefore, these data are not described in detail. Using Cohen’s D-index (Cohen, 1988) as the measure of effect size (where small, medium and large effects are denoted by .2, .5, and .8, respectively), however, it was found that most of the psychological scales along with one social scale had medium to large effect size differences. These included self esteem, depression, anxiety, self efficacy, and social consciousness scores. Examination of score values indicates the present U.K. sample had poorer levels of functioning on these scales than the U.S. sample. All of the treatment engagement scales had effect sizes in the small to medium range (ES = .26 to .43), with U.K. clients reporting higher treatment satisfaction and rapport, but lower participation. Overall, these results suggest CEST scale score distributions for English clients were acceptable and also that there might be some U.K. versus U.S. variations in psychological functioning that warrants closer study in future research.
In terms of staff comparisons on the ORC, motivation for program change by the U.K. counselor sample ranged from 29.4 (sd = 7.1) for training needs to 32.9 (sd = 5.2) for pressures for change. Scales for program resources ranged from 29.4 (sd = 6.3) for equipment to 33.0 (sd = 8.4) for internet, while staff attribute scales ranged from 33.5 (sd = 6.2) for influence to 38.1 (sd = 4.9) for efficacy. Organizational climate scales ranged from 32.9 (sd = 6.8) for communication to 35.5 (sd = 5.7) for mission.
Inspections of U.K. and U.S. scale scores indentified differences that were statistically significant but none of the ORC effect sizes were in the medium or large range. There were seven scales in the small to medium range, including pressures for change, offices, training resources, equipment, growth, efficacy, and influence. Score values for these scales showed the U.S. sample reported lower pressures for change and more positive ratings of resources and staff attributes. There were no differences in the organizational climate scales with an effect size over .10.
Correlations were first examined to assess relationships in the U.K. sample of the major CEST treatment engagement scales (treatment participation, and counselor rapport) with client self-ratings of their motivation for treatment, psychological functioning, and social functioning. Table 1 shows that almost all client motivation and psychosocial functioning scales were significantly and consistently correlated with engagement (treatment participation, and counselor rapport). These results are similar to those found in U.S. samples.
In order to gauge the relative importance of the CEST scales to each of these aspects of engagement, multiple regressions were performed. For treatment participation, Table 1 shows significant predictors in the U.K. sample included desire for help, treatment readiness, treatment needs, self esteem, anxiety, decision making, self efficacy, and social consciousness; collectively, these scales accounted for 47% of its variance [F(8,1483) = 165.22, p < .0001]. Similar results were found for counselor rapport, for which desire for help, treatment readiness, self esteem, depression, anxiety, decision making, and social consciousness accounted for 33% of its variance [F(8,1478) = 89.64, p < .0001]. For the U.S. sample, similar results were found for treatment participation [F(10, 8759) = 983.27, p < .0001, R2 = .53] and counselor rapport [F(11, 8746) = 314.53, p < .0001, R2 =. 28]. Generally, the motivation for treatment, psychological, and social functioning domains all included significant predictor variables.
Relationships between program-level organizational functioning from the ORC and client engagement indicators from the CEST also were examined using data aggregated to the program level. The number of U.K. programs included was too small to perform multiple regressions, so correlational analyses only are reported in Table 2. For the U.K., treatment participation scores were significantly correlated with training resources, staff attributes scales (growth, efficacy, and adaptability), and two climate scales (i.e., positively related to mission and negatively related to stress). Counselor rapport was significantly related only to one staff attributes scale (growth) and none of the organizational climate scales. In the U.S. sample, treatment participation was significantly related to counselor ratings of low program needs, perceived level of influence in their program, and clarity of mission. Counselor rapport was related to low program and training needs, high resources (for offices, staffing, equipment, and internet), more perceived influence, and all six of the organizational climate scales (positively for mission, cohesion, autonomy, communication, openness to change, and negatively for stress).
Inconsistencies (in Table 2) between findings for U.K. and U.S. samples in relationships of counselor ratings with client engagement raise several questions. The scores were based on counselor and client data aggregated to the program level. The composition of U.K. programs, however, was highly diverse in comparison to the U.S. sample in that it included brief HIV/AIDS outreach as well as criminal justice referral services that involved limited client contact and were not equally appropriate for program-level analyses of engagement in therapeutic relationships. Because the U.K. data collected from the Greater Birmingham region were uniquely linked so that programs and specific counselors could be matched to their clients, it was possible to reexamine data for a reduced subsample of 142 counselors and 858 matched clients from a set of 22 more comparably focused treatment service settings. Two-level hierarchical linear modeling (HLM version 6.0; Raudenbush & Bryk, 2002) was used to assess the relationship between each of the 18 ORC scales with each of the treatment process measures (treatment participation and counselor rapport). The estimates were based on a bivariate approach, where the first level of each analysis consisted of client scores on one of treatment process variables and the second level consisted of the treatment program mean on one of the ORC scales.
Table 3 shows there was a broad change in the pattern of relationships. For treatment participation, only two program resource scales (offices and equipment) and two organizational climate scales (staff cohesion and openness to change) were significant. Counselor rapport as measured from the CEST was positively related to counselor perceptions (measured from the ORC) of low program needs, higher program resources (staffing, training, equipment), and better organizational climate (mission, cohesion, communication, and openness to change). These revised results are highly consistent with the pattern of program-level findings reported in Table 2 for the U.S. sample, suggesting that the diversity in U.K. program composition indeed influenced the earlier findings based on the total sample of aggregated program-level records. More importantly, this more precise matching within these service delivery services strengthens the evidence base that counselor attributes and client engagement are directly interrelated at a personal dyadic level.
The findings suggest an integrated set of assessments for studying interrelationships of client-level (i.e., the CEST) and program-level (i.e., the ORC) functioning is applicable in England. When considered within the context of a comprehensive framework for treatment process (Simpson & Joe, 2004), the interactions and sequential relationships of client needs, readiness, engagement, and retention stages within the context of treatment system dynamics should help provide “diagnostic” insights (Gossop, Stewart, & Marsden, 2003; Joe, Simpson, & Broome, 1999; Simpson & Joe, 2004).
There also was consistency in the way client motivation and psychosocial self ratings were associated with indicators of program participation and rapport with counseling staff. Namely, stronger therapeutic engagement was linked directly to higher treatment motivation and readiness scores as well as to better psychosocial functioning profiles (i.e., higher self esteem, decision making, self efficacy, and social consciousness, along with lower anxiety, hostility, and risk taking). Regression analyses indicated these relationships are substantial, accounting for one-fourth to one-half of the variance in the client engagement measures. Within a treatment process framework (Simpson, 2004, 2006), therefore, this suggests early interventions designed to improve treatment readiness, mental health, and anger-related problems are appropriate in England.
Analyses of program settings for these services gave evidence that staff attitudes in England (like in the U.S.) have influences on client responsiveness to treatment as well. These factors include counselor perceptions of program needs, resources, professional skills, and organizational climate (also see Greener et al., 2007). Comparisons of the U.K. with U.S. findings, each based on aggregated counselor and client ratings within programs (shown in Table 2), however, showed the smaller U.K. sample differed from the U.S. sample in some of its specific predictor profiles. The number of U.K. programs available was only 44, and this relatively small sample size may have had restrictive effects on the results of the program-level aggregate analyses conducted. It also may have accentuated the weight of smaller agencies in the correlational analyses, an issue that was addressed more appropriately and possibly connected by the multi-level counselor-client linked analyses.
A related problem seemingly introduced by the small and highly diverse U.K. sample is reflected in the scale scores and their reliabilities, especially for program-level measures from the ORC. Overall findings for mean scale scores for client and counselor ratings showed they were generally comparable to U.S. profiles but their reliability indices were poorer. These scales were originally designed to include minimum numbers of items in order to gain administration efficiency, but a few scales even in the larger U.S. studies (especially for client self-efficacy, social consciousness, and counselor autonomy) fall below .60 and require caution in using them. The U.K. counselor ratings used to calculate program-level indicators yielded 11 scales with marginal to low reliabilities, including four below .60, that raised questions about their validity and interpretative applications.
However, the U.K. data included records for clients that were linked with those for their respective counselors and it was possible to carry out a more refined and powerful multi-level (HLM) analysis. This had the benefit of excluding data collected from clients and counselors in outreach and service referral programs wherein personal relationships were limited. Results from this refined analysis increased the number of significant staff functioning predictors (defined by their beta weights) in the U.K. sample from one to eight, including four organizational climate scales (i.e., for clarity of mission, cohesion, communication, and openness to change). These results closely resemble findings from the U.S. (see Simpson & Flynn, 2007b), but most importantly, they introduce unique methodological refinements beyond the previous studies using U.S. samples that have been limited methodologically to the use of unmatched counselor and client records within programs. By establishing evidence for the moderating effects of key counselor attributes on client engagement at this more personalized level, the present results from the U.K. illustrate a method for studying dyadic clinical matching of counselor skills with client needs and functioning profiles.
The U.K. treatment system for substance use problems currently focuses mainly on heroin addiction within a tradition of individual counseling approaches. In contrast, the U.S. treatment system is addressing a long-term trend towards more stimulant than opioid drug use and emphasizes “group counseling” approaches that can offer peer-related therapeutic benefits as well as higher appeal in terms of cost benefits (Etheridge et al., 1997; Knight et al., 2008). These and other variations across therapeutic systems guided by different philosophies and policies, especially when viewed across international borders, may contribute to differences in treatment process dynamics.
Nevertheless, the overall findings from this study contribute to an international evidence base for using common assessments to measure client and staff needs and functioning dynamics. Interim and measurable criteria are identified that can be useful in testing and refining treatment improvement protocols and thereby are suitable for exploring interrelationships of scales, including patterns of associations between client and staff assessment scores. By aggregating client scores within programs (or within more specific client-by-treatment subgroups), meaningful measures can be calculated based on these assessments for monitoring clinical patterns and response, emerging client needs, and program evaluations (Simpson, 2006). Further work is in progress for extending these assessments across time and examining provider characteristics in relation to client changes.
This work was jointly supported by NIH/National Institute on Drug Abuse MERIT Award Grant R37 DA13093 and the NHS/National Treatment Agency for Substance Misuse (NTA). The Birmingham Drug Action Team in the West Midlands also provided crucial help and support in the data collection process. The interpretations and conclusions, however, do not necessarily represent the positions of NIH/NIDA in the United States or NHS/NTA in England. The authors thank the staff, managers, and clients from treatment agencies that participated and the research staff that assisted in data collection. More information (including intervention manuals and data collection instruments that can be downloaded without charge) is available on the Internet at www.ibr.tcu.edu, and electronic mail can be sent to ude.uct@rbi.