|Home | About | Journals | Submit | Contact Us | Français|
Understanding the implementation of evidence-based practice (EBP) in community service settings is critical for the successful translation of research to practice. However, we have limited research evidence about the impact of EBP implementation on the mental health and social service workforce. In a previous study we demonstrated reduced staff turnover where an EBP was implemented with fidelity monitoring in the form of supportive ongoing supervision and consultation. Other research has shown that staff burnout and emotional exhaustion in particular is associated with poor quality of care and increased staff turnover intentions and turnover. Current research, however, has focused less on the effects that EBP implementation may have on staff emotional exhaustion. The present study investigates the association of EBP implementation and fidelity monitoring with staff emotional exhaustion in a statewide EBP implementation study. The 21 case management teams in this study were randomized in a 2 (EBP vs. services as usual [SAU]) by 2 (monitoring vs. no monitoring) design. The EBP in this study was SafeCare®, a home-based intervention that aims to reduce child neglect in at-risk families. SafeCare was developed from a behavior analysis approach and is based in cognitive behavioral principles. In keeping with our previous research, we hypothesized that providers implementing SafeCare with monitoring would have the lowest levels of emotional exhaustion and those receiving additional monitoring not in the context of EBP implementation would have higher emotional exhaustion relative to the other groups. Results supported our hypotheses in that we found lower emotional exhaustion for staff implementing the EBP but higher emotional exhaustion for staff receiving only fidelity monitoring and providing SAU. Together, these results suggest a potential staff and organizational benefit to EBP implementation and we discuss implications of the findings relative to EBPs and to fidelity monitoring.
Evidence-based practice (EBP) dissemination and implementation processes and outcomes have received increased attention in recent years. For children and youth mental health and social services in particular, addressing the challenges of dissemination and implementation of EBPs has been emphasized as critical for improving the quality of treatment services received as well as for EBP sustainability (Barlow, Levitt, & Bufka, 1999; Burns, 2003; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). With the recognition that implementation of EBPs in real-world settings often represents a complex and challenging process, a growing body of literature has developed that examines factors influencing EBP implementation. While we are witnessing an increased awareness of the value of implementation science as a discipline, there remains a critical gap in our understanding of EBP implementation (Aarons, 2005; Burns, Hoagwood, & Mrazek, 1999; Hoagwood et al., 2001).
In addition, little is known about how the implementation process may affect service providers, and the organizations within which they work, as implementation often encompasses changes in organizational structure, process, and technology (Glisson, 1992). Changes in training, supervision, and practice requirements are examples of how the service providers may be affected and the impact of these changes to service delivery has received little attention (Aarons & Palinkas, 2007). This is of concern because of the growing interest in, and implementation of, EBPs in child and family service settings (Aarons, 2005). For the purposes of this study EBPs are defined as those behavioral and social service interventions that have sufficient support from well-conducted, rigorous research studies and allow for both clinical judgment and consumer choice, preference, and culture (American Psychological Association, 2005; Institute of Medicine, 2001).
In a previous study we found that EBP implementation with fidelity monitoring, provided as ongoing supportive consultation, predicted lower staff turnover rates in a statewide children’s services system (Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009). In the current study, we further explore one potential mechanism that might be implicated in this finding. Specifically, we examine emotional exhaustion among home-based service providers participating in a statewide EBP implementation of a parent-mediated intervention for children at risk of child maltreatment in the form of parental neglect. Emotional exhaustion, one of the three components of burnout (the other components include depersonalization and personal accomplishment) is the extent to which an employee feels that their emotional resources have been depleted (Maslach & Jackson, 1981). We focus on emotional exhaustion because it best captures the “core meaning” of burnout (Cropanzano, Rupp, & Byrne, 2003; Knudsen, Ducharme, & Roman, 2006; Shirom, 1989) and high levels of emotional exhaustion predict important organizational outcomes such as job performance, turnover intentions, voluntary staff turnover (Wright & Cropanzano, 1998). We next describe relevant research on emotional exhaustion.
Emotional exhaustion has been a central research focus for decades and in particular within social and human service organizations. Scholars have consistently demonstrated the physical and emotional hazards of burnout for individuals (e.g. see Lee & Ashforth, 1996; Maslach & Leiter, 1997) and a growing body of research focuses on the organizational implications of emotional exhaustion. For example, emotional exhaustion has been linked to job performance (Cropanzano et al., 2003), turnover intentions (Blankertz & Robinson, 1997; Geurts, Schaufeli, & De Jonge, 1998; Knudsen, Ducharme, & Roman, 2009), and employee turnover (Wright & Cropanzano, 1998). To our knowledge no studies have examined emotional exhaustion within the context of EBP implementation.
Although there have been studies of turnover in the context of EBP delivery (Aarons et al., 2009; Glisson et al., 2008; Sheidow, Schoenwald, Wagner, Allred, & Burns, 2007; Woltmann et al., 2008) an important step in understanding the impact of EBP implementation and fidelity monitoring is examining more proximal staff outcomes from EBP implementation such as emotional exhaustion that may be implicated in work-related behaviors. EBP may reduce emotional exhaustion to the degree that the practice aids in service delivery and if it is perceived fit well with the needs of clients and improve client outcomes. In the implementation literature this is known as innovation-values fit, or the degree to which an innovation, in this case an EBP, fits the values and needs of service providers (Klein & Sorra, 1996). It is also likely that an EBP will be positively perceived to the degree that it is believed to be able to increase service effectiveness compared to services as usual. In addition, the fit of the EBP with services being delivered, and fit with client needs and presenting problems may also mitigate stress and decrease job stress. Additionally, most EBPs require or recommend some type of fidelity monitoring, which may be associated with emotional exhaustion (Edwards et al., 2005), although the degree to which fidelity monitoring is actually conducted in most usual care settings is unknown. We next discuss EBP as a special case of organizational change.
The implementation of an EBP and fidelity monitoring typically represent significant changes to organizational structure and process. Service providers who experience organizational change report that role changes may be inconsistent with their professional identity leading to negative effects on job satisfaction (Neuman, 2003). Implementation of EBP generally requires organizational changes that impact how work is carried out as well as subsequent staff attitudes and behaviors (Aarons & Sawitzky, 2006; Harris & Mossholder, 1996; Howard & Frink, 1996). While attention has been given to job burnout in public sectors services (e.g., Wright & Cropanzano, 1998)), to our knowledge, the effects of organizational change represented by EBP implementation and fidelity monitoring on provider emotional exhaustion have never been explicitly examined – an important first step in examining the potential association of EBP implementation, workforce functioning, and large scale practice sustainability.
Transitioning mental health and social service agencies into the implementation of EBP typically involves changes in organizational processes due, in part, to the need for adherence to a more structured service model. For example, successful translation of laboratory models of child therapy and social services into the field depends upon maintaining protocol adherence (Elliott & Mihalic, 2004). One example of this comes from ongoing dissemination of the Multi-Systemic Therapy (MST) model for youth behavior problems. Henggeler and colleagues (1997) have noted significant decrements in effectiveness associated with drift from the MST protocol, prompting dissemination efforts that emphasize model adherence. Further, Waller (2009) noted a number of reasons why service providers may drift from evidence-based cognitive behavior protocols. He also proposed some practical solutions to drift but emphasized that evidence-based behavioral interventions require attention to model adherence. Fidelity monitoring and feedback facilitates the assessment of and compliance with adherence to an EBP. As such, emotional exhaustion may be even more important in EBP implementation because of increased demands for following procedural specifications related to particular interventions.
Most EBPs require a higher degree of structure relative to usual care. The EBP that is the focus of the present study - SafeCare - has a great deal of structure and is a home-based intervention designed to reduce child-neglect and improve parent-child interactions (Gershater-Molko, Lutzker, & Wesch, 2003; Lutzker & Bigelow, 2002). SafeCare targets proximal parental neglect behaviors and was developed for and evaluated with multi-problem families with young children involved in the child welfare system. SafeCare (with origins in “Project 12-Ways”) grew out of the behavior analysis field, and is manualized, structured, and uses classic behavioral intervention techniques (e.g., ongoing measurement of observable behaviors, skill modeling, direct skill practice with feedback, training skills to criterion). SafeCare is comprised of three components derived from the original 12 units of Project 12-Ways: (a) infant and child health, (b) home safety and cleanliness, and (c) parent-child bonding.
The present study examined the impact of EBP implementation and fidelity monitoring on staff emotional exhaustion in the context of a statewide effectiveness trial. State authorities, in collaboration with academic researchers and service agencies, selected SafeCare as an EBP to implement in the state child welfare family preservation/family reunification service system. The effectiveness trial used a 2 (EBP vs. SAU) by 2 (fidelity monitoring vs. no fidelity monitoring) design to examine intervention effectiveness and the impact of fidelity monitoring (Aarons, et al., 2009). While the effectiveness trial focused on client outcomes, the implementation study is an observational study of implementation process and outcomes. The State of Oklahoma Child Welfare System is divided into six areas or regions for administrative purposes. Because of variation in area population characteristics, areas and sites were assigned to intervention conditions in a deliberate rather than a random manner. Such a logical approach allowed the investigators to come closest to obtaining comparable pre-intervention cell characteristics at the client level. Although this moved away from a classic randomized design, this approach solved some of the problems that randomization of sites would entail while preserving key aspects of a true experimental design. In particular, intervention conditions were assigned, not self-selected, and assignment was made independent of factors related to intervention condition. Teams within each condition (EBP or SAU) were then assigned to fidelity monitoring or no fidelity monitoring.
The SAU approach, while not manualized, included intensive case-management, identifying risk factors, facilitating access to services to address identified risks, family and social support, empowerment strategies, and didactic parenting education. The agencies in the regions selected to implement the EBP had previously been providing SAU, thus the implementation of EBP involved adding SafeCare to their model of care. All regions in the EBP condition received a week-long didactic and interactive training (e.g., role-plays, in-vivo practice) in SafeCare.
Fidelity monitoring and feedback was provided by professionals trained in the service model (SafeCare or SAU) who attended home-based services, observed sessions, and after the session, provided feedback and additional training as needed to providers in the fidelity monitoring conditions. This is a type of supervision that also includes a fidelity monitoring function. The program referred to the fidelity monitors as “ongoing consultants” to enhance receptivity by home-based providers. Ongoing consultants were selected by the agencies and often were providers who were viewed as highly skilled by their colleagues and supervisors. Conscious efforts were made to address service providers’ receptivity and acceptability of the ongoing consultants. For example, to facilitate the providers’ perception of the consultant as a support rather than as someone to “monitor” them, consultants were presented as someone to help the providers learn how to deliver the SafeCare or SAU model with their complex families. The ongoing consultants were employed by the research project rather than the service organizations, and provided supervision that was distinct from that provided by the organizational supervisor. SafeCare consultants were not to provide feedback about providers to agency supervisors, except in rare cases of ethical concerns. However, the consultants did provide supervision regarding client issues and service provision and were indeed fidelity monitors in addition to being supportive coaches. The consultants were trained in evaluating providers’ behavior relative to the SafeCare model, and turned in fidelity data for each session that was observed. Each provider was observed by their consultant for two one to two hour sessions each month.
The ongoing consultants in the two treatment conditions had slightly different roles. In the EBP group, the focus of the consultation was assisting the service providers in applying the SafeCare protocol in the home with their clients. The consultant also was available to model and assist with motivational interviewing and other issues related to the case. In the SAU condition, the role of the consultant was to aid in providing social support, case management resources, and didactic parenting, as well as to assist with motivational interviewing and general supportive counseling skills.
As noted above, our previous study found lower rates of staff turnover where Safecare was implemented with ongoing consultation (Aarons et al., 2009). The present study further explores one potentially active factor that may be implicated in our previous study, namely the impact of implementation on staff emotional exhaustion. For this study we proposed the following hypotheses:
Data used in this study were collected as part of a larger longitudinal study examining organizational factors likely to impact the statewide implementation of an EBP, SafeCare, throughout a statewide network of nonprofit organizations contracted with the Oklahoma State Office of Children Services System (see Aarons et al., 2009 for greater detail). As part of this larger study, home-based service providers and supervisors employed by the contracted agencies (N=21) were asked to complete bi-annual, web-based surveys. The surveys took approximately 45–90 minutes to complete and provider response rates averaged 94.5% over four waves. The organizational participation rate was 100%. For this study, we examined emotional exhaustion at the fourth wave. At wave four, service providers had been in the study condition for two full years, allowing a sufficient amount of time for SafeCare implementation to influence the organizational environment and staff. The response rate at wave 4 was 96.7%. The high response rate is the result of a number of factors including ongoing positive relationships with the agencies and teams, an incentive (electronic gift certificate) for each survey completed, collegial relationships with the state Office of Children’s Services, bi-annual project newsletters, and in-person feedback and communications with the teams and agencies involved. We included all participating direct service staff, resulting in a sample size of 99 home-based service providers.
The web survey incorporated questions regarding home-based provider demographics including age, sex, race, education level, and job tenure (Gregory A. Aarons, 2004). Sex was binary coded to indicate whether the home-based provider was female. For this study, race was treated dichotomously indicating if the home-based provider was Caucasian or non-Caucasian. Provider education was measures as having a college degree, having completed some graduate work, and having a masters degree, but was collapsed into a dichotomous measure indicating whether the participant had received at least some graduate level education for multivariate analyses. Job tenure was calculated using provider self-reported employment start date and the date of survey completion. The natural log of job tenure was used in all multivariate analyses to reduce skewness in the measure.
Emotional exhaustion is the dependent variable in all analyses, and was assessed using the emotional exhaustion subscale from the Children’s Services Survey (Glisson & James, 2002). The scale has demonstrated sound psychometric and measurement characteristics (6-items, α = .92). Examples of scale items include: “I feel emotionally drained from my work,” and “I feel used up at the end of the workday.” Participants indicated their level of agreement with each statement on a 5-point Likert-type scale that included “Not at all,” “To a slight extent,” “To a moderate extent,” “To a great extent,” and “To a very great extent” response options, with higher scores representing higher levels of emotional exhaustion.
A unique aspect of this study is the 2 x 2 experimental design, in which EBP vs. SAU is crossed with the level of fidelity monitoring (monitored vs. non-monitored). In this study, there were 21 teams of home-based service providers (N=99) operating in six regions covering the entire state, with approximately one quarter of the teams operating in each study condition. Efforts to promote stratified randomization of the study sites were pursued by purposefully distributing rural and urban regions within each study condition. For the analyses, models were developed to examine both the independent effects of EBP and fidelity monitoring on employee turnover as well as the interaction of EBP and monitoring conditions on employee turnover. The four experimental groups are defined as follows:
All multiple regression analyses were conducted using the STATA 9 statistical package (StataCorp, 2005). Analyses examined the association between experimental conditions, and controlled for provider demographic characteristics. Because home-based providers were nested in 21 different work teams there was a concern of potential within-group dependency. As such, all regression analyses accounted for clustering at the team level.
Table 1 shows the descriptive statistics for each of the variables used in the study. Of the 99 participants who participated in the study, 84.8% were female. The ethnic backgrounds of the participants were categorized into white (59.6%) and non-white (40.4%). The mean age of the participants was 39.2 years (SD=11.6), and their average job tenure was 29.3 months (SD=43.1). All service providers had at least a college degree (53.5%), a number had completed some graduate work (20.2%), and just over 26% had master’s degree. The average caseload for the home-based providers was 9.6 cases (SD=2.7). A majority of the 99 participants were implementing SafeCare (62.2%) and just under half (44.9%) were in a monitoring condition. The average emotional exhaustion score was 1.1 (SD=0.9), out of a potential 0 to 4 scale, with higher scores signifying higher levels of burnout.
Table 2 displays the correlation matrix for each of the variables in the analyses. As expected, the SafeCare condition had a significant, negative association with emotional exhaustion, suggesting that the EBP implementation may reduce job burnout among home-based service providers. In addition, emotional exhaustion was significantly, positively associated with provider caseload and negatively associated with age.
Table 3 presents the results from the multivariate regression of emotional exhaustion on provider demographic characteristics and emotional exhaustion. As is shown in Model 1, which includes only provider demographic characteristics, older providers have significantly lower levels of emotional exhaustion as do their younger counterparts. Also associated with emotional exhaustion is caseload: for every additional case that a provider carries, emotional exhaustion increases by .08, all else being equal.
In the next model (Model 2), we included the experimental condition variables of implementing SC and having fidelity monitoring. Consistent with our hypotheses, implementing SC was found to be significantly related with lower levels of emotional exhaustion. In specific, providers who are implementing SC score almost a half-point lower on emotional exhaustion than those who providing services as usual (b=-.477, p<.05), all else being equal. On the other hand, and contrary to what we expected, fidelity monitoring was not a significant predictor of emotional exhaustion. Age and caseload remained statistically significant when EB condition and fidelity monitoring were included.
In the final model (Model 3), we separated experimental condition into the four groups (SC/M, SC/Non, SAU/M, SAU/Non) with SAU/Non serving as the comparison group. In this set of analyses, we found that in comparison to providers of unmonitored, usual services, individuals who provided SAU services with monitoring had significantly higher scores on emotional exhaustion (b=.717, p<.05). And, while neither SC condition was significantly associated with emotional exhaustion compared to usual services, both demonstrated negative associations with emotional exhaustion. Thus we found an overall effect of EBP with lowered emotional exhaustion, and higher levels of emotional exhaustion for those providing SAU and also receiving increased monitoring.
The present study examined the primary association of EBP implementation and fidelity monitoring on provider emotional exhaustion in the context of a statewide implementation of an EBP designed to reduce child neglect. Consistent with expectations, EBP implementation predicted lower levels of emotional exhaustion. In addition, higher provider caseload and younger age were associated with higher emotional exhaustion. It could be that the lower emotional exhaustion associated with EBP implementation could mitigate some of the effects of caseload and age and these associations should be examined in future studies.
The implementation of an EBP could be associated with lower levels of emotional exhaustion for multiple reasons. Some explanations may include: the fit of the EBP with services being delivered, fit with client needs and presenting problems, provision of a useful structure for organizing and providing services, and perceived effectiveness of the practice as compared to services as usual. Indeed, our results indicate that it is likely that SafeCare is highly congruent with the philosophy, approach, and desired outcomes of the home-based service providers in this study, representing a good innovation-values fit (Klein & Sorra, 1996). In addition, it is likely that greater perceived effectiveness of the EBP in treating clients further helps to engage and energize service providers.
The highest level of emotional exhaustion was experienced by providers of usual care services who were in the monitoring condition. While further study is needed to examine this issue, it is possible that, although the fidelity monitoring is intended as a mode of ongoing supervision, consultation, and support, it may be perceived as an increase in oversight that reduces job autonomy and adds stress when not accompanied by a change in service provision. That is, increased oversight without a clear rationale may impinge on service providers’ sense of control and autonomy. In addition, fidelity monitoring as a main effect was not significantly associated with emotional exhaustion. It was only when tied to the SAU condition where we saw negative effects. This further supports the notion that a clear rationale for supervision and monitoring should be present. These hypotheses should be examined in future studies.
The findings of the present study are encouraging for agencies considering the implementation of an EBP. Where the EBP fits the needs of clients and providers, implementation may be more effective. Lower levels of emotional exhaustion may be especially beneficial in regard to promoting job satisfaction, organizational commitment, and decreased staff turnover (Cropanzano et al., 2003). Improving these factors is likely to lead to better continuity of care and more positive organizational climate which has been demonstrated to be associated with better client outcomes (Glisson & Hemmelgarn, 1998).
The data used in the present study represent only a cross-section of information. As such, it is difficult to determine causality. The present study examined implementation of SafeCare and results may not generalize for the implementation of other EBPs. In addition, this study examined the impact of EBP and fidelity monitoring with home-based service providers in child social services agencies. The pattern of results may differ in agencies with higher educated and licensed service providers. However, a recent study suggests that there may be even greater openness to EBPs among mental health service providers in other settings (Stahmer & Aarons, in press).
The momentum of EBP implementation in mental health and social services agencies in the United States and abroad has been wrought with challenges. Agency and provider buy-in to EBP models of services is a repeated barrier to implementation, often related to concerns about training demands, impact on work load, decreased job autonomy, and increased stress, which in turn could increase staff turnover. In some cases such concerns can lead organizations to “de-adopt” innovative service models (Massatti, Sweeney, Panzano, & Roth, 2008). Recent studies have found serious workforce problems in mental health and social service fields (Hoge et al., 2005). However, agency directors and managers recognize that for current staff, implementation of an EBP should provide tangible benefits (Palinkas & Aarons, in press). While not directly assessed in the present study, it is likely that potential negative effects of the increased structure of the EBP were mitigated because SafeCare adds structure that directly supports the mission of home-based service providers. The construct of innovation-values fit posits that implementation of innovation will be successful to the degree that the innovation matches the values (in this case theoretical orientation, organizational mission, service provider tasks and duties) of the organization and individuals within that organization (Klein & Sorra, 1996).
In this study, we examined the effects of EBP implementation on provider emotional exhaustion as a step in assessing how EBP affects providers within a complex organizational context. Several logical steps follow. Future research in this area should link emotional exhaustion to turnover intentions and to actual turnover. Also, important questions arise around the role of work attitudes (conceptualized as job satisfaction and organizational commitment) in the relationship between EBP implementation, emotional exhaustion, turnover intentions, and staff retention. Staff retention is a critical concern when considering EBP implementation because of the increased resources required for additional training and clinician support needed to promote adherence to intervention protocols. Continued attention to organizational climate and the work context of mental health and social services is particularly relevant in this area.
Rather than focusing only on clinical outcomes we believe that it is important to better understand the full range of impacts of system and organizational decisions to change practice through EBP implementation. There is a need to identify the ways in which such changes may impact staff and the quality of mental health and social services. Concerns such as workforce selection, retention, training, and service quality are important concerns for agencies providing mental health and social services. While addressing client needs is of critical importance, identifying the ancillary impacts of efforts to improve clinical services is of high importance in large and small agencies and service delivery systems and both urban and rural settings. This study took place in a system encompassing these complex characteristics. Further work should examine how such system change impacts staff in diverse service settings during implementation as well as maintenance and sustainability of EBPs.
The results of this study address the real-world implications of EBP implementation and are relevant for national, state, and local service systems and organizations and also for clinical practice. It is critical to understand how to improve workforce engagement in services, help providers develop positive relationships with clients, and provide the most effective service technologies. Learning how to decrease burnout and emotional exhaustion should be pursued in order to help to accomplish these goals. It may be that some EBPs and/or fidelity monitoring approaches increase staff work stress and burnout while others may lower these factors. This is an area in need of future study as the impetus to implement EBP in large scale service systems continues to grow.
This study was supported by National Institute of Mental Health Grants R01MH072961 (principal investigator: Gregory A. Aarons) and R01MH065667 (principal investigator: Mark J. Chaffin). SafeCare® is a registered trademark. We thank the organizations, supervisors, and service providers that participated in the study and made this work possible. The authors also thank Kristen Honda, B.S. for help with preparation of this manuscript.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.