|Home | About | Journals | Submit | Contact Us | Français|
The present study describes the development of the Mindfulness-Based Relapse Prevention Adherence and Competence Scale (MBRP-AC), a measure of treatment integrity for Mindfulness-Based Relapse Prevention (MBRP). MBRP is a newly developed treatment integrating core aspects of relapse prevention with mindfulness practices. The MBRP-AC was developed in the context of a randomized controlled trial (RCT) of MBRP efficacy and consists of two sections: Adherence (adherence to individual components of MBRP and discussion of key concepts), and Competence (ratings of therapist style/approach and performance). Audio recordings from 44 randomly selected group-treatment sessions (50%) were rated by independent raters for therapist adherence and competence in the RCT. Findings evinced high interrater reliability for all treatment adherence and competence ratings, and adequate internal consistency for Global Therapist Style/Approach and Global Therapist Competence summary scales. Ratings on the MBRP-AC showed that therapists in the recent RCT adhered to protocol, discussed key concepts in each session, and demonstrated the intended style and competence in treatment delivery. Finally, components of the Competence section were positively related to measures of therapeutic alliance, and overall ratings on the Adherence section were positively related to measures of change in mindfulness over the course of the treatment.
Mindfulness-based interventions train individuals to practice formal meditation techniques to increase attention to present moment experiences, including thoughts and emotional states, while relating to these experiences in an accepting and nonjudgmental manner. These interventions have been described with increasing frequency in the empirical literature and are being applied to a variety of populations and problem areas (e.g., Bowen et al., 2006; Bowen et al., 2009; Kabat-Zinn, 1982; Kabat-Zinn et al., 1992; Kristellar & Hallet, 1999; Teasdale et al., 2000).
Two mindfulness-based interventions that have garnered significant empirical support are Mindfulness-Based Stress Reduction (MBSR; Kabat-Zinn, 1990) and Mindfulness-Based Cognitive Therapy for depression (MBCT; Segal, Williams & Teasdale, 2002; Teasdale, Segal & Williams, 1995). These 8-week group treatments combine cognitive and/or behavioral techniques with mindfulness exercises to help participants better cope with stress and depressive symptoms, respectively.
Mindfulness-Based Relapse Prevention (MBRP) is a recently developed aftercare treatment for adults with substance use disorders (Bowen, Chawla & Marlatt, in press; Witkiewitz, Marlatt, & Walker, 2005). Although the structure of MBRP is based largely on MBSR and MBCT, MBRP provides an innovative application of mindfulness in the treatment of substance use disorders. Specifically, MBRP integrates mindfulness practice and aspects of relapse prevention (Daley & Marlatt, 2006; Marlatt & Gordon, 1985), a cognitive-behavioral treatment that aims to prevent relapse to substance use.
Findings from the initial RCT on which the current study is based, suggested that MBRP participants significantly reduced substance use and craving during the four months following treatment compared to a treatment as usual control (TAU) group (Bowen et al., in press). Further, MBRP participants showed increases in acceptance and ability to act with awareness compared to TAU participants. Additionally, high participant satisfaction and treatment compliance demonstrated the feasibility of the MBRP program. These results offer promising preliminary evidence for MBRP as an efficacious and feasible aftercare treatment and provide support for the use of mindfulness meditation for treating substance use disorders.
As described by Waltz, Addis, Koerner and Jacobson (1993) adherence refers to the extent to which interventions and approaches that are prescribed by a treatment manual are delivered and those that are proscribed are avoided. Competence, on the other hand, refers to the skill with which therapists deliver the treatment (Waltz et al., 1993). Assessment of these treatment integrity indicators is essential to ensuring the internal and external validity of treatment study findings (Bellg et al., 2004). Treatment integrity assessment helps detect and explain differential adherence and/or competence in treatment delivery (Waltz et al., 1993), which reduces unexplained variability and determines the extent to which the treatment itself accounts for outcomes (Bellg et al., 2004). Further, assessment of treatment integrity facilitates cross-site comparisons and assessment of treatment discriminability (Bellg et al., 2004; Waltz et al., 1993). Finally, the information provided by these measures may contribute to further development and refinement of the treatment (Waltz et al., 1993).
Despite their popularity and the data supporting their efficacy, most studies of mindfulness-based interventions have not assessed therapist adherence or competence. Although several studies on mindfulness-based interventions indicate that therapists were experienced in the delivery of the treatment, procedures for therapist training are often not discussed (Baer, 2003). Further, therapist experience and training do not necessarily indicate that a treatment was delivered as intended. One review of mindfulness-based treatments reported that no studies to date have included information on measures of adherence and competence (Baer et al., 2003). To the best of our knowledge, there is only one published measure of treatment integrity for mindfulness-based interventions (see Segal, Teasdale, Williams & Gamer, 2002). Although this scale is a promising measure of adherence to an MBCT protocol, it does not assess the competence with which the treatment was delivered. The dearth of treatment integrity tools for mindfulness-based interventions, and therapist competence in particular, highlights the need for further measures in this area.
The complexities involved in the development of treatment integrity measures may account for the paucity of treatment integrity assessment in the context of mindfulness-based treatments. As is true of assessing any treatment, it is expensive to hire and train expert raters, as well as time-consuming to rate individual treatment sessions. There are also several challenges unique to mindfulness-based interventions that may further complicate the assessment of therapist adherence and competence. Mindfulness is a multifaceted construct that is difficult to describe and quantify (Ivanovski & Malhi, 2007). Until recently, there was no agreed upon operationalization of mindfulness or the procedures involved in training individuals to practice mindfulness. This ambiguity has made it challenging to operationalize and quantify the techniques and processes used in these treatments. Additionally, many of the study interventions reported in the literature were delivered by their original developers (e.g., MBSR, Kabat-Zinn, 1990), and it is only recently that procedures for training other health care providers have been developed. Moreover, there is no agreed upon standard for the background, training and supervision required of facilitators of mindfulness-based interventions. For instance, some interventions require that therapists have a formal, daily mindfulness practice, which is assumed to be reflected in their therapeutic approach (Kabat-Zinn, 2003). However, this embodiment of mindfulness on the part of the therapist and the qualities with which it is associated (e.g., authenticity, nonjudgment, being in the present moment, etc.) may be difficult to measure with accuracy (Dimidjian & Linehan, 2003). These issues have provided great challenges to the clarification necessary for the measurement of competence and adherence.
The aim of the current study was to develop a reliable and valid quantitative measure of therapist adherence and competence in delivering MBRP. In this study, we describe the development of the MBRP-AC, assess its’ interrater reliability and validity, and use it to assess therapist adherence and competence in the context of an MBRP RCT.
Participants (N = 93) in this study were recruited after completion of either inpatient or intensive outpatient substance abuse programs at a nonprofit treatment agency. Participants comprised a subset of participants who had volunteered to take part in the larger parent RCT (N = 168) comparing the efficacy of MBRP and TAU (for details, see Bowen et al., in press), and consent to complete all study procedures. Participants (35.48% female) had an average age of 40.84 years (SD=1.07). The majority identified as Caucasian (63.44%), followed by African-American (22.58%), Hispanic/Latino/a (6.45%), Multiracial (1.08%), Native American (9.68%), and Asian/Pacific Islander (4.31%).
MBRP (Bowen et al., in press; Witkiewitz et al., 2005) is an eight-week group-based intervention that involves two-hour, weekly sessions. Each session is facilitated by two therapists in a small group format (6–10 participants), and comprises meditation practices and related relapse prevention discussions and exercises. Specific goals of MBRP include increasing awareness of substance use triggers, shifting the relationship and response to discomfort or distress, and interrupting habitual behavioral reactions to substance use cues, thereby reducing the likelihood of relapse. Sessions typically begin with a 20–30 minute guided meditation, and involve a variety of experiential exercises (e.g., “urge surfing” or practicing nonjudgmental acceptance and observation of urges, rather than suppression, practicing use of mindfulness techniques in high-risk situations, and use of “mini-meditations” or “breathing spaces” in challenging situations). Participants are assigned daily exercises and provided with meditation audio recordings for practice between sessions.
The MBRP-AC, which is completed by trained raters reviewing recorded sessions, consists of two main sections: Adherence and Competence, each of which contains two subscales. The items of the MBRP-AC are presented in Appendix 1.
The two subscales of the Adherence section of the MBRP-AC are MBRP Treatment Components and Discussion of Key Concepts. Adherence to MBRP Treatment Components is assessed using a checklist of the major topics within each session of the MBRP treatment manual to determine whether therapists delivered each of the components. The Discussion of Key Concepts subscale is based on several items from the MBCT-AS (Segal et al., 2002), which were modified to fit the content of the MBRP treatment protocol. These items assess the extent to which therapists used the key concepts of MBRP in facilitating discussion of in-session exercises and in responding to questions and comments. This subscale consists of four items, each of which is rated using behavioral counts, or a tally of instances of each behavior.
The Competence section of the MBRP-AC contains two subscales: Therapist Style/Approach, and Overall Therapist Performance. The Therapist Style/Approach subscale was developed based on direct observation and review of MBRP session audiotapes, and was revised based on feedback from experienced providers of mindfulness-based interventions. Items are both indicators of general therapist competence (e.g., therapists’ ability to elicit and respond to feedback by asking open questions, accurate empathy/validation), as well as mindfulness therapist competence (e.g., clarifying expectations and misconceptions about mindfulness meditation). The subscale consists of 4 items, each of which is measured on a 5-point scale, where 1 = low ability and 5 = high ability.
The Overall Therapist Performance subscale is designed to capture raters’ global impressions of the session, such as the ability of the therapists to work as a team and keep the session on topic. It consists of four items that are rated on a five-point scale, where 1 = not satisfactory and 5= excellent.
Therapeutic alliance was assessed using a 12-item version of the Working Alliance Inventory (WAI-S; Horvath & Greenberg, 1989, Tracey & Kokotovic, 1989) that was completed by individuals as part of the larger RCT. It is a self-report measure of the quality of the therapeutic relationship. Items are rated on a 7-point Likert scale where 1 = never, 7 = always (e.g., “I am confident in therapist’s ability to help me,” “Therapist and I are working towards mutually agreed upon goals”). The WAI-S demonstrated good internal consistency (α = 91.4) in the present study.
Mindfulness was assessed using the Five Factor Mindfulness Questionnaire (FFMQ; Baer, Smith, Hopkins, Krietemeyer, & Toney, 2006), which was completed by all participants in the parent RCT. It consists of 39 items assessing the degree to which individuals notice or attend to a variety of internal and external phenomena, engage with full awareness of current experience, allow and accept current experience without evaluation, and notice internal phenomena, such as thoughts and feelings, without reacting. Items are rated on a 5-point Likert scale (1 = never or very rarely true and 5 = very often or always true). The FFMQ has demonstrated good internal consistency and expected relationships with a variety of other constructs (Baer et al., 2006).
Raters were two doctoral-level clinical psychology students, two masters’ level counseling/clinical psychology students, and one bachelors’ level former psychology student. Procedures for rater training were similar to those used to train MBRP therapists: raters first attended or viewed video recordings of an intensive two-day workshop that included a detailed review of the treatment manual as well as the background of MBRP. Raters then coded at least 10 practice sessions, which were evaluated with respect to “expert” consensus until their ratings achieved acceptable reliability. The 10 practice sessions were selected from a pilot MBRP group and were not included in the analyses.
A detailed manual was developed to train raters and establish a consistent and reliable approach to performing ratings. The manual describes general guidelines such as instructing raters to take notes, to keep a running tally of therapist behaviors while listening to each session, and to make overall ratings after listening to the entire session. The manual also provides detailed descriptions of each item, example behaviors that correspond to each item, guidelines for distinguishing between related items, and instructions on making lower versus higher ratings.
The MBRP groups were facilitated by ten therapists with master’s degrees in either psychology or social work. Therapists were experienced in delivery of cognitive-behavioral interventions, and several had an extensive background with mindfulness meditation. Therapists participated in a two-day intensive training workshop, several weeks of additional training, and weekly supervision throughout the course of the groups. Each group was co-facilitated by two therapists.
Audio recordings of 44 group-treatment sessions were assessed. Four of the eight sessions (50%) were randomly selected from each of 12 treatment cohorts (one cohort was excluded due to equipment failure). Each session was rated by two independent raters. The assignment of raters to sessions was randomly determined. All the raters met for periodic calibration meetings to prevent rater drift.
Reliability analyses compared the ratings made by the first and second raters for all MBRP-AC subscales. Average absolute agreement was established using 2-way, mixed model intraclass correlation coefficients (ICC). Consistency analyses for both individual items and summary scores (i.e., for Therapist Style/Approach and Overall Therapist Performance) showed high levels of agreement between raters (see Table 1 for ICCs). Internal consistency for the Therapist Style/Approach (α = .86) and the Overall Therapist Performance (α = .82) subscales reached acceptable levels; thus, these summary scores may be used to represent global therapist style and competence.
Validity of the MBRP-AC was tested using bivariate Spearman correlations to assess whether the Adherence and Competence sections were significantly correlated with participants’ development of mindfulness over the course of the intervention and with participants’ perceptions of therapeutic alliance following treatment. The Working Alliance Inventory summary score at posttreatment was correlated positively with Overall Therapist Performance (ρ = .29, p = .03), and with two individual items on the Therapist Style/Approach subscale: Inquiry (ρ = .26, p = .04) and marginally with Use of Key Questions (ρ = .24, p = .07). The posttest-baseline change score for the FFMQ summary score correlated positively with the total Adherence section (ρ = .36, p = .04), which indicated that adherence to the MBRP manual was positively related to the development of mindfulness during the intervention.
To assess manual adherence, we used a Wilcoxon sign-rank test to confirm whether therapists delivered 100% of the MBRP treatment components listed on the Adherence Subscale. Averaged across groups and sessions, the delivery of MBRP components in the current RCT did not reach the “ideal” 100% delivery of all MBRP Treatment Components, z (N = 11) = −2.89, p <.004. That said, adherence was still relatively high at 90% component delivery (SD = 2%). Adherence as measured by the Discussion of Key Concepts reached 100%: Key Concepts were discussed in all 11 groups (M = 74.82, SD = 22.24, Range = 38–110).
Competence in the current RCT was also relatively high. The Therapist Style/Approach ratings reached a mean of 3.95 (SD = .50), and the Overall Therapist Performance subscale reached a mean of 3.92 (SD = .42). Considering competence ratings for these scales ranged from 1 (low) to 5 (high), the scores indicated that the therapists in this study reached adequate levels of competence on both Therapist Style/Approach and Therapist Performance.
The primary aim of this study was to develop a reliable and valid measure of therapist adherence and competence in delivering a new mindfulness-based treatment for substance use disorders. Results demonstrated high interrater reliability for all treatment adherence and competence subscales of the MBRP-AC, which suggests that it is possible for students at various levels of psychology training to be trained to code MBRP sessions effectively and reliably. This finding is encouraging and would reduce the high cost and time expenditure typically associated with expert-level adherence and competence ratings. Another encouraging finding was the high level of internal consistency for therapist style and competence ratings. This finding indicates that the competence subscales are consistently tapping into key MBRP constructs.
Concerning the validity of the MBRP-AC, competence of overall treatment delivery and use of Inquiry and Use of Key Questions appear to be significantly related to client ratings of working alliance, suggesting that therapists’ skillful facilitation and exploration of clients’ experiences may foster a positive therapeutic relationship with clients. Additionally, the relationship between overall therapist adherence and changes in client mindfulness over the eight weeks of the course suggests that close adherence to the MBRP protocol may facilitate increases in clients’ levels of mindfulness. Given that most exercises and practices in MBRP are designed to increase mindful awareness (Bowen at al., in press), this association is not surprising. Indeed, it suggests that facilitation of these practices is contributing to a primary goal of the intervention. Somewhat surprising is the lack of association between therapist competence and client mindfulness. It is possible that the current MBRP-AC competence subscales do not fully reflect the key factors of competence necessary for optimal delivery of MBRP. Future studies are necessary to determine whether this finding is limited to the current sample.
Once reliability and validity of the MBRP-AC was established, we used this measure to assess the adherence and competence attained in the first RCT of MBRP. Although the MBRP therapists did not reach the hypothesized 100% adherence to the MBRP components, findings suggested relatively high levels of adherence for an initial feasibility and efficacy trial (90%). The Use of Key Concepts subscale of the Adherence section showed 100% adherence to the key concepts of MBRP, which were discussed in all groups and in all sessions. On a scale of 1 (low) to 5 (high), mean ratings of both the Therapist Style/Approach subscale and the overall performance section were close to a four. Taken together, these descriptive findings suggest sufficient reliability and validity of the MBRP-AC to assess therapist treatment delivery. Further, ratings on the MBRP-AC suggest therapists in the recent RCT adhered to protocol, discussed key concepts in each session, and demonstrated the intended style and overall competence intended in the treatment.
Despite these encouraging preliminary findings, there are also several limitations that deserve mention. First, the study did not assess treatment discriminability, or the extent to which the scale can distinguish between MBRP and other types of interventions with which it shares common elements (e.g., MBCT and RP). According to Waltz and colleagues (1993) a key element of measure adherence is the examination of unique and common elements of different treatments. Thus, an important task for future research comparing MBRP to other treatments is to build in further components that will assess prescribed components for MBRP and other, related treatments to optimally assess treatment discriminability. A second limitation of the MBRP-AC is that it only assesses prescribed therapist behaviors and not those that are proscribed by the MBRP treatment manual. Although we asked raters to make a note of therapist behaviors that appeared contrary to the treatment, these were not quantified for the purposes of the current study. Thus, consistent with the recommendations of Waltz et al. (1993), future research would benefit from an added focus on enumerating therapist behaviors that are inconsistent with or violate the treatment guidelines for MBRP.
Overall, these limitations indicate a need to further refine and develop the current measure. However, given that MBRP is one of the first mindfulness-based interventions to address relapse related to substance use disorders, the development and assessment of the MBRP-AC is an important step to assessing and enhancing MBRP treatment integrity for future trials and clinical applications.
This research was supported by National Institute on Drug Abuse Grant R21 DAO 10562.
The Mindfulness-Based Relapse Prevention Adherence and Competence Scale (MBRP-AC)
|2||Expectations for Group and Rules for Confidentiality and Privacy|
|3||Discussion of Group Structure and Format|
|4||Raisin Exercise/Discussion of Automatic Pilot|
|5||What is Mindfulness?|
|6||Body Scan Practice|
|7||Home Practice for the Week|
|Please assess the extent to which therapists used each of the key concepts of MBRP in facilitating discussion of in-session exercises and in responding to questions and comments. Please count the number of instances of each behavior. Please focus on the skill of the therapist, taking into account how difficult the participant seems to be.||Behavior Counts|
|1||NOTICING/AWARENESS OF CURRENT EXPERIENCE:|
To what extent do therapists encourage noticing and being aware of present moment experience?
This includes pointing out and validating client behaviors, if the client is already paying attention to their experience (e.g., “So you noticed the thought that…”; “So you noticed a judging thought.”; “So you noticed your mind wandering.”; “Seems like you were aware of the craving.”), as well as encouraging clients to pay attention to their experience (e.g., “What would happen if you just tried to notice that as a thought?”; “Could you pay attention to the sensation?”).
|2||ACCEPTANCE of CURRENT EXPERIENCE:|
To what extent do therapists encourage bringing curiosity and a non-judgmental attitude to whatever arises in the present moment, regardless of whether it is pleasant, unpleasant, or neutral?
(e.g., paying attention to the experience of sleepiness, restlessness, peacefulness, calm, anger, an itch etc. with curiosity and non-judgment; “Can you just notice what the experience of anger is like?”; “What does an itch really feel like—is it burning, is it hot, pulsing, throbbing?”).
|3||ACCEPTANCE versus AVERSION:|
To what extent do therapists introduce the differences between relating to one’s experiences from a standpoint of acceptance as opposed to aversion?
(e.g., allowing and being with difficult emotional and physical states instead of trying to get rid of them, fight them, fix them or manipulate one’s experience in some way; “Can you just stay with the itch for a moment and get to know it before scratching it, or immediately getting rid of it and having to make it go away?”).
|4||ACCEPTANCE and ACTION:|
To what extent do therapists discuss the importance of stepping out of auto-pilot (pausing, taking a breathing space, evaluating one’s choices etc.) as a means of engaging in mindful action (responding vs. reacting, making choices that are in one’s best interest), and/or to what extent do therapists describe the relationship between acceptance and skillful/mindful action?
|INQUIRY||Therapists’ ability to elicit and respond to both verbal and non-verbal feedback (this may be demonstrated through eliciting reactions to exercises, asking open questions, validating the clients experience and summarizing/making reflections).|| 1 2 3 4 5|
|ATTITUDE||Therapists’ ability to model and embody the spirit of mindfulness (respond to participants in a way that is curious, focused in the present moment, and non-judgmental/accepting of whatever participants bring up).|| 1 2 3 4 5|
|USE OF KEY|
|The overall extent to which the therapists used key questions in eliciting discussion about exercises and home practice.|
(1) Highlighting the participant’s raw experience in the moment: What did you experience in this exercise? What body sensations did you experience during the exercise? Making a distinction between thoughts, feelings, and body sensations.
(2) Distinguishing from typical way of experiencing things: How is this different from how you usually experience things?
(3) Relationship to purpose of program: How does it relate to relapse?
| 1 2 3 4 5|
|The extent to which the therapist addresses and clarifies ideas and misconceptions about mindfulness meditation.|
(e.g., “I’m not doing it right”;” I’m just in a different zone when I practice”; “This practice is great because it makes me feel so relaxed and blissful”).
| 1 2 3 4 5|
Competence: Overall Therapist Performance 1. How would you rate the overall quality of the therapy in this session? 1 2 3 4 5 Not Mediocre Satisfactory Good Excellent Satisfactory 2. How would you rate the ability of the therapists to work as a team? 1 2 3 4 5 Not Mediocre Satisfactory Good Excellent Satisfactory 3. How would you rate the ability of the therapists to keep the session focused and on topic? 1 2 3 4 5 Not Mediocre Satisfactory Good Excellent Satisfactory 4. Please rate the overall quality of delivery of the meditation exercises. 1 2 3 4 5 Not Mediocre Satisfactory Good Excellent Satisfactory