|Home | About | Journals | Submit | Contact Us | Français|
Despite the fact that the number of Hispanic individuals in need of treatment for substance use problems is increasing internationally, no studies have investigated the extent to which therapists can provide empirically supported treatments to Spanish-speaking clients with adequate fidelity. Twenty-three bilingual Hispanic therapists from five community outpatient treatment programs in the United States were randomly assigned to deliver either three sessions of motivational enhancement therapy (MET) or an equivalent number of drug counseling-as-usual sessions (CAU) in Spanish to 405 Spanish-speaking clients randomly assigned to these conditions. Independent ratings of 325 sessions indicated the adherence/competence rating system had good to excellent interrater reliability and indicated strong support for an a priori defined fundamental MET skill factor. Support for an advanced MET skill factor was relatively weaker. The rating scale indicated significant differences in therapists’ MET adherence and competence across conditions. These findings indicate that the rating system has promise for assessing the performance of therapists who deliver MET in Spanish and suggest that bilingual Spanish-speaking therapists from the community can be trained to implement MET with adequate fidelity and skill using an intensive multisite training and supervision model.
The need for efficacious substance abuse treatment services among Hispanics is increasing on an international level as larger numbers of Hispanics experience significant problems with substance use and seek treatment (Burgos & Collazo, 1990; Canino et al., 1993; Fleiz et al., 2007; Lee, 1998; Medina-Mora et al., 2006; Substance Abuse and Mental Health Services Administration; (SAMSHA), 2001; 2004). However, Hispanics have been underrepresented in treatment research and, for those treated, there is little evidence regarding the extent to which treatment integrity can be maintained in empirically validated therapies adapted for use with Hispanic substance users (Bernal and Scharrón-Del-Río, 2001). Moreover, there are no available psychometrically sound instruments that can be used by trainers, supervisors, or researchers to evaluate Spanish-speaking therapist fidelity (i.e., adherence to manual guidelines, skill in delivering the treatment, discrimination among therapies) and the adequacy of training efforts for empirically validated therapies.
In this report we provide data on the psychometric properties of a therapist adherence/competence rating system adapted from a previous English language multisite randomized clinical trial of Motivational Enhancement Therapy (MET; Ball et al., 2007; Martino et al., 2008), and used in an independent Spanish version of the trial (Carroll et al., in press). Both studies were conducted within the U.S. National Institute on Drug Abuse Clinical Trials Network (CTN). Identical in most ways (other than language and relevant cultural adaptations) to the protocol conducted in English (Ball et al., 2007), the Spanish MET protocol examined the effectiveness of a three-session adaptation of MET used in Project MATCH (Matching Alcoholism Treatments to Client Heterogeneity; Miller, et al., 1992) compared to counseling-as-usual (CAU) in five U.S.-based community treatment programs. The protocol was intended for community drug treatment programs that treated large numbers of monolingual Spanish-speaking clients. Portions of the MET manual and all assessments were translated into Spanish and both treatment conditions were delivered in Spanish. All clinical and research staff participating in the study were bilingual and most were of Hispanic descent (Suarez-Morales et al., 2007; see also for method translation). Outcomes in the Spanish MET trial revealed that while both the MET and CAU interventions resulted in reductions in substance use during the 4-week treatment phase, MET resulted in sustained reductions during the subsequent 12-weeks of the study, whereas CAU was associated with increases in substance use over this follow-up period. Relative to CAU, MET led to a greater reduction in frequency of alcohol use in the subgroup of participants whose primary substance was alcohol (Carroll et al., in press).
MET was selected for the Spanish MET trial, in part, because its nonjudgmental and empathic style contribute to a collaborative relationship between the therapist and client, which is particularly important when working with ethnic minority groups (Añez et al., 2008; Comas-Diaz, 1996). Indeed, recent studies have suggested that adaptations of motivational interviewing such as MET show larger treatment effects among ethnic minority participants (Hispanic and African Americans minority participants) relative to non-minority samples in clinical trials (Hettema et al. 2005; Winhusen et al. 2008).
A measure to establish bilingual therapists’ MET fidelity when the approach is used with monolingual Spanish-speaking Hispanic clients has not been developed to date, nor have the psychometric properties of existing English-language MET fidelity measures when adapted for MET delivered in Spanish. For example, Martino and colleagues (2008) developed an adherence and competence rating scale, called the Independent Tape Rating Scale (ITRS) to evaluate MET fidelity in the English MET trial. Based on independent ratings of 425 audiotapes drawn from that trial, the ITRS demonstrated excellent interrater reliability (mean intraclass correlation coefficient estimates were .89 for adherence and .81 for competence). Moreover, the 10 items describing key aspects of MET converged as intended to form two a priori defined skill factors conceptually related to MET, namely, fundamental skills that underpin the empathic and supportive elements (open-ended questions, affirmations, and reflections) and advanced skills (multiple strategies for evoking clients’ self-motivational statements). These factors discriminated between MET and CAU therapists, with MET therapists having significantly higher fundamental and advanced MET skill adherence and competence and lower MET inconsistent adherence (Martino et al., 2008). It is unclear if the ITRS would perform similarly in the Spanish MET trial.
What also remains unclear is the relationship between Hispanic cultural factors and MET adherence and competence. The current investigation afforded a unique opportunity to investigate Hispanic cultural factors assessed in the Spanish MET trial (e.g., level of acculturation of therapists and clients and therapist and client ethnicity) that may be associated with MET therapist adherence and competence. Past studies have shown that Hispanic clients may view therapists with similar ethnic/cultural backgrounds as more empathic (Alegria et al., 2006; Karlsson, 2005) and in some studies therapist-client ethnic matching predicted positive treatment outcomes (Sue et al., 1991).
In this paper, we evaluated whether the psychometric properties (interrater reliability and factor structure) of the ITRS were retained when used to evaluate the implementation of MET in Spanish and with tapes rated by bilingual raters. In addition, we evaluated the extent to which MET would be discriminable from CAU in the Spanish version of the trial. Thus, we expected significantly higher scores on the two MET skill factors for therapists assigned to MET versus CAU as well as lower scores on the items intended to be inconsistent with MET. We predicted that given the substantial level of initial training and ongoing clinical supervision provided during the trial, these differences would be consistent across the five program sites, the 23 therapists within conditions, and the three therapy sessions. Additionally, we conducted a series of exploratory analyses that evaluated the relationship between therapist and client cultural factors (e.g. therapist Hispanic/non-Hispanic ethnicity, therapist–client ethnic match, level of therapist Hispanic and American acculturation) and MET adherence and competence scores.
Participants in this study included therapists, clients, and tape raters. Client participants provided written informed consent, therapists provided either written permission or informed consent depending on local Institutional Review Board requirements, and tape raters signed a confidentiality agreement. Characteristics of each participant group are described below.
All the 23 therapists that began the study completed the study. Therapists were employed in one of five non-methadone licensed outpatient substance abuse treatment programs that provided an array of services for English-and Spanish-speaking clients. Each of the five participating agencies, located in Colorado, Florida, Oregon, New Mexico, and New York, had a minimum of four Spanish-speaking therapists on staff who delivered substance abuse treatment in Spanish to Hispanic clients. Details on therapist exclusion criteria, procedures to ensure therapist Spanish language fluency and comprehension, and therapist characteristics and demographic information are presented in prior reports (Carroll et al., in press; Suarez-Morales et al., 2007). Therapists were ineligible to participate in the protocol if they did not pass a Spanish fluency test or had received formal MET training within the 3 months prior to protocol initiation, a timeframe in which MET skills are likely to diminish without ongoing performance feedback and coaching (Miller et al., 2004). Overall, therapists had an average of 6.0 hours (sd = 7.4) of prior motivation interviewing or MET training; no baseline differences existed between conditions. On average, therapists had been employed at their agencies 3.9 years (sd = 2.8), had 6.4 years (sd = 4.9; range = 0–20) of counseling experience, and 15.6 years (sd = 4.7) of education. Therapists were born in a range of countries, the most common being South American countries (12%), Mexico (12%), Dominican Republic (12%), the U.S. Commonwealth of Puerto Rico (12%), Cuba (6%), and Jamaica (6%). Most therapists identified Spanish as their primary language (62%) and self-reported a high degree of biculturalism. Fourteen of the 23 therapists identified themselves as Hispanic, six identified themselves as Caucasian, two identified themselves as having an unspecified ‘other’ ethnicity, and one elected not to report ethnicity.
Four hundred five outpatient clients participated in the Spanish MET trial. They were primarily monolingual Spanish-speaking who sought outpatient treatment for a substance use problem and had used alcohol or an illicit drug at least once in the 28 days prior to randomization. As in the English MET protocol (Ball et al., 2007), minimal exclusions (severe medical or psychiatric instability prohibiting outpatient treatment enrollment, residential instability or imminent incarceration, seeking detoxification) were placed on potential participants to obtain a representative community sample. On average, clients were in their early 30’s (M = 32.5, sd = 9.1) and had completed on average 9.6 years of education (sd = 3.2) with slightly less than two-thirds currently employed (62%). The majority of the sample (94%) was of Hispanic descent (Mexican 37%; Puerto Rican 14% Cuban 9%; Guatemalan 3%; Other 5%; multiracial 2%). African Americans and Caucasians comprised 1% and 5% of the sample, respectively. The remaining 24% of the sample indicated they were of Hispanic descent, but did not report specific cultural data. Site differences emerged in the clients’ Hispanic ethnic composition,χ2 (24, N = 405) = 490.6, p < .001. Sites 1, 2, and 4 were predominantly Mexican; site 3 was predominantly Puerto Rican and US born Hispanic, while site 5 was predominantly Cuban, Central American, and US born Hispanic. Primary substance use problems included alcohol (60%), cocaine (22%), marijuana (9%), opiates (6%), and methamphetamine (3%). Further details about the client characteristics and treatment outcomes are presented in another report (Carroll et al., in press).
Nine independent tape raters were trained in the methods used to evaluate the 325 session audiotapes generated within the protocol. Raters were on average 39.0 years old (sd = 10.5). They were of Hispanic descent, fluent in Spanish (assessed in the same manner as the protocol therapists), mostly female (67%), and on average completed 18.0 years (sd = 2.1) of education and achieved a Master’s degree in a clinical profession (89%). Raters had 5.8 years (sd = 6.6) of substance abuse treatment experience. Sixty-seven percent reported prior training in MET, typically in a workshop format for an average of 7.4 hours (sd = 8.2).
The Spanish MET protocol was an independent replication of the completed CTN English MET protocol (Ball et al., 2007), but with all aspects of treatment and research delivered in Spanish. In brief, primarily Spanish-speaking clients seeking treatment for any substance use disorder in five community treatment programs were randomly assigned to receive either three individual sessions of manual-guided MET or three CAU sessions. At least four therapists from each site were randomized to continue CAU or to be trained in MET. The MET therapists and program-based supervisors received a 16-hour intensive workshop training, followed by audiotaped practice cases supervised by MET experts until they demonstrated minimal performance certification standards (i.e., at least half of the MET-consistent items rated at a minimum of “4” or above in terms of adherence and competence) in three sessions. After therapists were certified in MET, they began to treat randomized clients in the protocol and receive biweekly supervision from their supervisors who provided the therapists with MET adherence and competence rating-based feedback and coaching after reviewing audiotaped client sessions. Therapists in both conditions audiotaped all protocol sessions for independent treatment fidelity assessment (see Carroll et al., in press). In addition, in the MET condition, supervisors encouraged therapists to address culturally specific issues that might inform the therapists’ understanding of the clients’ motivations to change their substance use. Thus, therapists and clients sometimes discussed issues such as migration (e.g., experience coming into the US), acculturation and stigma (e.g. language barriers, feeling disrespected by others), trauma history (e.g., leaving family behind, imprisonment), or obligations to family members residing outside of the country to appreciate how these factors might affect the clients’ readiness to change.
Therapist adherence and competence was evaluated using the 39-item Independent Tape Rater Scale (ITRS; Ball et al., 2002) adapted from the Yale Adherence and Competence Scale (Carroll et al., 2000). As in our previous report (Martino et al., 2008), this study focuses on the 30 items that address specific therapeutic strategies involving MET consistent interventions, MET inconsistent interventions, and general substance abuse counseling interventions.1 For each item, raters evaluated the therapist on two dimensions using a 7-point Likert scale. First, they rated the extent to which the therapist delivered the intervention (adherence; 1 = not at all, to 7 = extensively). Second, they rated the skill with which the therapist delivered the intervention (competence; 1 = very poor, to 7 = excellent). Item definitions, rating decision rules, and recording procedures were specified in a detailed rating manual that was used by supervisors and independent raters (Ball et al., 2002). Table 1 lists the scale items for each therapeutic category and provides the corresponding means, standard deviations and interrater reliabilities for each item by adherence and competence dimensions.
Raters were carefully prepared to assess the therapists’ adherence and competence in the sessions (see Martino et al., 2008 for detailed description). In brief, these procedures involved didactic training and review of the rating manual, rating practice items, performance feedback, and a set of 15 calibration tapes used to evaluate interrater reliability via estimates of intraclass correlation coefficients (ICC; Shrout and Fleiss, 1979). Combining the didactic and calibration tape components, each rater had about 44 hours of training. To ensure ongoing rater reliability, a randomly selected common tape was rated on three separate occasions approximately four months apart. Raters were aware of this procedure, but not its timing. Individual ratings were compared to expert ones and feedback was provided to all raters. One rater drifted (i.e., rated more than half the items more than two scale points above or below the expert ratings) and did not continue to rate tapes in the study.
A large number of tapes were rated from each condition (325 sessions; 160 MET: 165 CAU). While most of the tapes selected for analysis represented participants who had attended all three sessions (69% of total sample), the remaining 31% of tapes selected for analysis were derived from participants who completed 1 to 2 sessions only. We primarily focused on participants who had completed their assigned treatment condition so that outcome analyses could examine therapist adherence and competence while attempting to control for the possible confounding effect of the amount of treatment clients received. Following this selection procedure, we then randomly selected additional tapes from any therapists who were not included through this process (n = 2, or 9% of MET therapists) to ensure we had ratings for all therapists in the study.
This six-item self-report instrument developed for this study gathered relevant demographic information about the country of origin of the therapists and the clients and each of their parents, language use, and the individual’s length of residence (years) in the United States. Therapists/clients country of origin was determined by asking them to indicate their birthplace (e.g., Mexico, U.S. Commonwealth of Puerto Rico, Cuba, several South American countries, United States). A dichotomous birthplace variable was created for this study, coded 1 for a birthplace in Latin America, including Puerto Rico, and coded 0 for a birthplace outside of Latin America. Birthplace match was assigned if the client’s birthplace matched that of the therapist.
The BIQ (Szapocznik et al., 1980; Guo et al., in press) is a 24-item scale that assesses the individual’s level of acculturation/involvement, a process whereby an individual adjusts and integrates features of both the original (Hispanic) and dominant (American) cultures (LaFromboise et al., 1993; Tadmor and Tetlock, 2006). The BIQ is one of the few bidimensional acculturation measures designed specifically for Hispanics (Zane and Mak, 2003). Half of the items are Hispanic-oriented and half are American-oriented. The items assess comfort with the English or Spanish language in specific settings (e.g., home, work, with friends) and enjoyment of American or Hispanic cultural activities. Items are answered using a 5-point Likert scale (1 = not at all comfortable/not at all to 5 = very comfortable/very much). A score is computed for each cultural dimension (i.e., Americanism and Hispanicism). In the present sample, the Cronbach’s alpha coefficients for the Americanism and Hispanicism scores were .98 and .76, respectively for therapists and .88 and .85, respectively for clients.
Interrater reliability was estimated using Shrout and Fleiss (1979) intraclass correlation coefficients (ICC’s) two-way mixed model (3.1) with item ratings as the fixed effect and raters as the random effect. To confirm factor structure for our two hypothesized MET consistent subscales (fundamental and advanced skills), we conducted confirmatory factor analysis using the maximum likelihood estimation method for structural models with AMOS (6.0) software (Arbuckle, 2005) with several indices to determine the acceptability of model fit (Kline, 1998; Marsh et al., 1988; Yadama and Pandey, 1995): nonsignificant (p > .05) chi-square goodness of fit index, a χ2/degrees of freedom ratio < 2, normed fit index (NFI), incremental fit index (IFI), and comparative fit index (CFI) > 0.9, and root mean square error of approximation (RMSEA < .05). Because in larger samples (n > 200), the chi-square test usually is significant and often detects trivial differences between sample covariance and fitted covariance matrices (Hu and Bentler, 1995), we relied on the preponderance of evidence from the other indices in determining the best fitting model.
To evaluate internal validity (e.g., predicted differences in adherence and competence ratings on the two MET scales (fundamental and advanced) by treatment group (i.e., four comparisons), we conducted ANOVAs using a Bonferroni-corrected α of .0125 (.05/4). The two mean CFA-derived MET consistent factors were analyzed as separate dependent variables, treatment condition and program site as the fixed factors, and therapists (nested within condition) as a random factor. We also repeated the ANOVAs with session number as an additional fixed factor. Multivariate ANOVAs were used to compute estimates (Roy’s theta) of the proportion of variance accounted for by treatment condition, program site, session number, and therapist (within condition) effects, with the respective mean adherence and competence scores entered simultaneously in separate analyses (Harris, 1985). These models included two MET and two CAU therapists from each site who had five or more unique client sessions that had been independently rated (n = 267/325 or 82% of rated sessions). This approach was used to provide an adequate representation of therapists’ treatment adherence and competence and sufficient balance and variance at the therapist level to evaluate therapist as a random factor.
Finally, exploratory analyses were conducted to evaluate the relationship between cultural factors (i.e., therapist Hispanic/non-Hispanic ethnicity, therapist –client ethnic match, level of Hispanic and American acculturation) with MET adherence and competence scores, as well as site differences that might represent regional differences in the clients’ Hispanic ethnic makeup, using univariate analytic procedures (ANOVAs) and Pearson r correlations. These analyses were conducted across and within conditions.
Table 1 presents the means, standard deviations, and ICC reliability estimates of the 30 adherence and competence ITRS items across treatment conditions (see Table 1 footnote for rating score definitions). A reliability sample of 14 randomly selected tapes (7 MET and 7 CAU) rated by 9 raters (n = 126) indicated that for both the adherence and competence dimensions, 25 of the 30 ITRS interventions showed good to excellent reliability in that ICCs were .60 or above (Cicchetti, 1994). Five interventions (skills training, cognitions, psychodynamic, risk behavior reduction, unsolicited advice or direction giving) showed poor to fair reliability (ICCs < .60), which likely reflected their very low level of occurrence across sessions.
Within the MET condition, the most frequently occurring interventions were primarily MET fundamental skills: open-ended questions (M = 5.8, sd = 1.1), motivational interviewing style (M = 5.2, sd = 1.3), and reflective statements (M = 5.1, sd = 1.6). The next most frequently occurring interventions included client-centered problem discussion and feedback (M = 3.9, sd = 1.9), motivation for change (M = 3.4, sd = 2.0), affirming strength and self-efficacy (M = 3.3, sd = 1.6), discussion of pros, cons, and ambivalence (M = 3.2, sd = 2.3), change planning (M = 3.0, sd = 2.1), and fostering a collaborative relationship (M = 2.9, sd = 1.5). With the exception of social functioning (M = 3.8, sd = 1.8) and assessing substance use (M = 3.3, sd = 1.8), MET inconsistent and general drug counseling interventions were rated as largely absent.
Within the CAU condition, the most frequently occurring interventions were open-ended questions (M = 4.1, sd = 1.4), client-centered problem discussion and feedback (M = 3.5, sd = 1.7) and motivational interviewing style (M = 3.4, sd = 1.4). The next most frequently occurring interventions included reflective statements (M = 2.7, sd = 1.5), affirming strengths and self-efficacy (M = 2.3, sd = 1.4), and fostering a collaborative relationship (M = 2.2, sd = 1.3). With the exception of unsolicited advice (M = 2.6, sd = 1.7), MET inconsistent items were largely absent in the ratings. General drug counseling interventions occurred more often within sessions: assessment of social functioning (M = 3.5, sd = 1.7), assessment of substance use (M = 2.9, sd = 1.7), psychoeducation about substances (M = 2.3, sd = 1.6), and program orientation (M = 2.1, sd = 1.6). Mean competence ratings across ITRS items and treatment conditions indicated an ‘adequate’ to ‘good’ therapist skill level.
Confirmatory factor analysis was used to evaluate whether the 2-factor structure of the MET items was supported in this sample as in the English version of the study (Martino et al., 2008). Thus, we expected 5 items (open-ended questions, reflective statements, affirmations, fostering a collaborative relationship, and motivational interviewing style) would form an independent ‘fundamental’ MET skills factor and 5 items (client-centered problem discussion and feedback, pros/cons and ambivalence, heightening discrepancies, change planning, and motivation for change) would converge to form an ‘advanced’ MET skills factor. Table 2 reports the fit indices of our two predicted models for the combined set of three sessions and within each of the three sessions.2
The hypothesized MET fundamental skills factor demonstrated a good fit in the combined set of three sessions and in sessions 1 and 3. The hypothesized MET advanced skills factor fit very well in session 2 and partially in session 1, although it exhibited poor fit in the combined set of the three sessions and session 3. Reliability analyses of the two MET skill factors suggested they retained excellent inter-rater reliability, consistent with their individual components (adherence ICC: fundamental skills = .93; advanced skills = .89; competence ICC: fundamental skills = .88; advanced skills = .90). Pearson product-moment correlations between fundamental and advanced MET skill adherence and competence mean scores showed that these factors were positively associated with each other in expected ways (r ranged from .33 to .73, p < .001), but were not overlapping. We used these factors in the ANOVA tests of discrimination between conditions, sites, therapists, and sessions.3
Table 3 presents the multivariate ANOVAs examining differences by treatment condition, program site, and therapists in fundamental and advanced MET skill adherence and competence. The analyses revealed significant differences by treatment condition, program site, interaction of treatment and site, and therapists (all p’s < .05). Univariate analyses showed that for both the MET fundamental and advanced scale scores, MET therapists were rated as having significantly higher levels of adherence and competence relative to CAU therapists (p’s < .001). The amount of variance accounted for by treatment condition for MET fundamental and advanced adherence and competence (adherence theta = 33%; competence theta = 26%) was considerably higher than the amount of variance accounted for by site (adherence theta = 3%; competence theta = 9%), treatment by site interaction (adherence theta = 5%; competence theta = 4%), and therapist (adherence theta = 7%; competence theta = 5%). Session number, as a fixed factor in the ANOVAs, did not differentiate MET adherence or competence or interact with treatment condition or program site.
Given site differences in the clients’ Hispanic ethnic composition and the potential that variations in MET fidelity might be influenced by the unique ethnic and cultural factors at each site, we conducted post hoc analyses to examine where the site differences in MET adherence and competence occurred. These analyses revealed no significant differences between sites on MET adherence for either factor, although site 3 serving primarily Puerto Rican and US born Hispanic participants was found to have significantly lower fundamental (M = 4.6, sd = .8) and advanced (M = 4.5, sd = .8) MET competence scores relative to site 4 serving primarily participants from Mexico (fundamental: M = 5.3, sd = .8; advanced: M = 5.4, sd = 1.0) and significantly lower advanced MET competence scores relative to site 5 serving primarily Cuban, Central American, and US born, mostly Mexican ancestry, Hispanic participants (M = 5.1, sd = .8).4
We evaluated whether three cultural factors (therapist Hispanic/non-Hispanic ethnicity, therapist–client ethnic match, and level of therapist Hispanic and American acculturation) were associated with MET adherence and competence. No significant differences in MET adherence scores were found between Hispanic and non-Hispanic therapists. However, Hispanic therapists exhibited significantly lower MET fundamental competence scores relative to non-Hispanic therapists, F (1,10) = 5.1, p < .05). Similarly, therapists who were matched with participants on Hispanic ethnicity (e.g. Mexican, Puerto Rican, Central American) exhibited significantly lower MET fundamental, F (2,58) = 4.5, p < .02) and advanced, F (2,58) = 3.5, p < .04) competence scores relative to therapists who were not ethnically matched. Correlations between levels of therapist Hispanic and American acculturation with MET fundamental and advanced skills were not significant. To determine whether site differences in MET competence could account for these effects, these analyses were re-run without site 3. Differences previously associated with Hispanic cultural characteristics were no longer significant. Repeating these same analyses within treatment conditions yielded the same pattern of results.
This study replicated, in part, the psychometric properties of an adherence/competence rating scale developed for community therapists and used within an independent randomized clinical trial of MET versus CAU, delivered in Spanish, to a large sample of Spanish-speaking individuals seeking treatment in five treatment programs. Similar to results from the parallel Clinical Trial Network English MET trial (Ball et al., 2007; Martino et al., 2008), the ITRS 1) had excellent interrater reliability, and 2) discriminated between treatment conditions as predicted for both fundamental and advanced MET skills, and demonstrated a good fit for a fundamental MET skill factor. However, unlike in the English MET trial, support for the advanced MET skill factor was weak in the Spanish MET trial.
To our knowledge, this study is the first to document the reliability and validity of a therapist MET fidelity scale for bilingual therapists working with Spanish-speaking substance using clients. Given the widespread implementation of MET internationally (Miller and Rollnick, 2002), the ITRS may prove useful for establishing treatment discrimination in future MET trials conducted in Spanish and for the purposes of training and evaluating the performance of Spanish-speaking therapists in research or practice settings. Further investigation of the capacity of the ITRS to perform these functions is warranted.
In this study, analysis of interrater reliability indicated that 83% of the ITRS items exhibited good to excellent inter-rater reliability, with the majority of the ITRS items (67%) in the excellent range. The interrater reliabilities are consistent with those reported by Martino and colleagues (2008) from the parallel English language multisite trial, and are comparable to the Motivational Interviewing Treatment Integrity scale developed by Moyers et al. (2005) as well as other motivational interviewing fidelity measures (cf. Moyers et al., 2003; Madson et al., 2005). In addition, some support was found for the ITRS’s construct validity when used in the Spanish MET trial. Specifically, a fundamental MET skill factor fit the data adequately. This factor included several items that capture the empathic and collaborative style or spirit of motivational interviewing, often delivered through the frequent use of reflective listening and open-ended questions. Support was weaker for the advanced MET skill factor, though present in session two where the use of a decisional balance activity to explore the pros and cons of using versus not using substances was the prescribed activity within the MET treatment manual.
One potential explanation for the failure to find strong evidence for the two-factor structure shown in the English MET trial may be the way in which therapists utilized MET to explore and evaluate their clients’ motivations to change their substance use. Anecdotally, the therapists reported that they were less inclined to use the more directive strategies very early in treatment for eliciting clients’ self-motivational statements (i.e., those strategies that comprised the advanced MET skill factor) than those strategies that served to build a therapeutic relationship with clients (i.e., fundamental MET skill factor). The therapists in the Spanish MET trial may have relied more on the fundamental skills of MET before moving toward its more directive components. Fundamental MET skills that demonstrate collaboration, support, and empathy are fully consistent with adherence to particular Hispanic cultural values such as personalismo (preference for developing relationships with individuals rather than with institutions), respeto (respect and mutual deference), and confianza (trust and intimacy in a relationship) deemed necessary to enhance therapeutic alliance early in treatment and are often conveyed in a manner consistent with the client-centered nature of motivational interviewing or MET (Añez et al., 2008; Atdhian and Vega, 2005). The confluence of attention to Hispanic cultural values and MET adherence and competence requires careful examination in future studies.
Additional support for the validity of the ITRS comes from the univariate ANOVA tests of MET adherence and competence discrimination between conditions, which indicated that the MET therapists delivered fundamental and advanced MET skill strategies significantly more often and with greater competence than CAU therapists. While there were significant site, treatment by site, and therapist differences in the fidelity ratings, multivariate ANOVAs indicated that treatment condition accounted for substantially more variance in overall adherence and competence scores than any of these factors or the specific session delivered. These findings are particularly important given that CAU therapists were implementing a variety of interventions that were consistent with MET strategies, such as open-ended questions, motivational interviewing style, and client-centered problem discussion and feedback, albeit to a moderate degree. CAU therapists’ use of these skills may have been a reflection of their attention to Hispanic cultural values aimed at increasing client engagement early in treatment or their familiarity with basic MET skills they had acquired through prior training. Notably, across conditions therapists rarely used MET inconsistent strategies as these techniques are likely to be antithetical to Hispanic culture in the context of a helping relationship. Thus, the ITRS was able to discriminate the adherence and competence of bilingual therapists trained to use MET among a group of therapists already prone to interact with clients in a MET consistent way. Moreover, the findings suggest that the intensive MET workshop training and follow-up supervision delivered by on-site, program-based supervisors who had been trained to rate sessions using the ITRS and to provide therapists with feedback and coaching to improve their performance may be an effective approach for teaching bilingual therapists MET. A randomized controlled trial comparing supervision models is needed to test this assumption.
Our failure to find associations between cultural factors examined in this study and MET fidelity was somewhat surprising. Given the suggested potential synergy between ubiquitous Hispanic cultural values and MET strategies, we expected Hispanic therapists, those ethnically matched with clients, and those self-reporting high levels of Hispanic acculturation to show the highest levels of MET fidelity. Initially, our results suggested the opposite may be true in that Hispanic therapists and those ethnically matched to their clients demonstrated less competence implementing MET. However, this association was specific to one site where all therapists were Hispanic and ethnically matched with the majority of clients they saw in the trial. Thus, the poorer proficiency of the specific therapists implementing MET at one site, rather than specific cultural factors per se, seems to be the most parsimonious explanation for the results. Moreover, the relatively small number of therapists and their high level of Spanish fluency and biculturalism likely limited the variability in cultural factors that might relate to therapists’ ability to implement empirically supported therapies such as MET. In addition, even within race and ethnicity, personal characteristics (e.g., age, education) and experiences (e.g., discrimination, family supports) of the therapists and clients may differ substantially and these differences may limit the relevance of broad racial and ethnic matching analyses. Future studies with larger samples of bilingual therapists working with Hispanic clients are needed to better evaluate these issues.
Our finding that the use of interventions associated with general counseling techniques as well as empirically validated approaches other than MET (such as emphasis on abstinence, self-help group involvement, skills training, case management, and risk behavior reduction) occurred very rarely in CAU and did not increase over time is consistent with our findings from the English-language versions of these protocols (Santa Ana et al., 2008) and highlights the limited penetration of empirically validated therapies in standard addiction treatments. Further work is needed to develop effective strategies for teaching therapists how to deliver empirically validated substance abuse treatments to Hispanic clients.
Overall, the results of this study extend the initial positive findings for the psychometric properties of the ITRS and suggest that it is a reliable and valid measure for assessing MET skills among bilingual therapists who treat monolingual Spanish-speaking clients seeking treatment in community treatment programs. Strengths of this investigation include the generalizability of the ITRS psychometric properties to Spanish-speaking therapists who work with Hispanic clients and the careful assessment of MET adherence and competence in a wide range of program sites, therapists, and clients and evaluated by independent raters fluent in Spanish. This study has several limitations. First, therapists and their supervisors participating in the CTN may be more amenable to training in and evaluation of MET than other community substance abuse treatment programs not participating in research trials (Roman et al., 2006). Second, the participating sites may have been unusual in that they had a minimum of four bilingual therapists, a fairly large number given the shortage of bilingual Spanish-speaking staff in the US workforce (Atdjian and Vega, 2005; Diaz et al., 2001). Third, the ITRS may need modifications for use with Hispanic therapists and clients in countries outside of the United States since its global applicability cannot be presumed. Fourth, our decision to focus our process analyses more on those participants who completed three sessions may have provided a constricted range of treatment engaged clients, although 1/3 of the sessions analyzed were derived from participants who did not complete the three sessions. Each of these issues limits the generalizability of the findings. Finally, because of the infrequency with which therapists used MET-inconsistent strategies, we were unable to test for hypothesized treatment condition differences for these items. Future studies should address these limitations, as well as evaluate the relationship between therapist adherence and competence to client outcomes in community programs serving treatment-seeking Spanish-speaking clients in order to determine the clinical relevance of therapist MET fidelity.
1Items 31–34 involve 7-point Likert scale general ratings of the therapist (overall skillfulness, ability to maintain the session’s structure, demonstration of frustration during the session, general client discussions and self-disclosures). Items 35–37 involve 7-point Likert scale ratings of the client (unrelated session discussions, difficulty understanding, and working alliance). Future reports will examine treatment condition differences for these items.
2As suggested by Kelloway (1998), we compared our two hypothesized models with two alternative ones to determine if models other than the ones we predicted might provide a better description of the adherence data instead of relying solely on a model’s absolute fit: (1) a 1-factor model in which all 10 MI consistent items form a single MI fidelity construct and (2) a 2-factor model in which MI fidelity is supported by both fundamental and advanced MI skills factors. The 1-factor and 2-factor MI fidelity alternative models had poor fit (all NFI, IFI, and CFIs < .9). These analyses, and the item-level factor score weights and squared multiple correlations of the hypothesized models, are available upon request from the first author.
3Items inconsistent with MET rarely occurred in either condition and were not proscribed in CAU. Therefore, we did not expect these items to form a single construct and had insufficient variability in adherence ratings from which factor structure could be analyzed. Similarly, because general drug counseling items could occur together or separately in both conditions and were expected to vary widely across sites, we did not anticipate that these items would necessarily form a construct. Chronbach’s alpha coefficient (.55) for these items suggested there was little interrelationship among the items. Thus, no factor analyses were conducted on these two sets of items, and they were excluded in tests of treatment discrimination.
4MET adherence and competence site specific means and standard deviations for the MET fundamental and advanced factors are available from the first author.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.