|Home | About | Journals | Submit | Contact Us | Français|
Measurement feedback systems (MFSs) have been proposed as a means of improving practice. The present study examined the implementation of a MFS, the Contextualized Feedback System (CFS), in two community-based clinic sites. Significant implementation differences across sites provided a basis for examining factors that influenced clinician uptake of CFS. Following the theoretical implementation framework of Aarons, Hurlburt & Horwitz (2011), we coded qualitative data collected from eighteen clinicians (13 from Clinic U and 5 from Clinic R) who participated in semi-structured interviews about their experience with CFS implementation. Results suggest that clinicians at both clinics perceived more barriers than facilitators to CFS implementation. Interestingly, clinicians at the higher implementing clinic reported a higher proportion of barriers to facilitators (3:1 vs. 2:1); however, these clinicians also reported a significantly higher level of organizational and leadership supports for CFS implementation. Implications of these findings are discussed.
Quality improvement initiatives in healthcare are increasingly including the use of electronic tools such as measurement feedback systems (MFSs) to improve service delivery and monitor outcomes (APA Task Force on Evidence-Based Practice for Children and Adolescents, 2006; Bickman et al., 2011; Jensen-Doss & Hawley, 2010; New Freedom Commission on Mental Health 2003; Sapyta, Riemer, & Bickman, 2005). MFSs have been found to have a positive impact on outcomes in different subspecialties of medicine (Duncan & Pozehl, 2000; Goebel, 1997; Holmboe, Scranton, Sumption, & Hawkins, 1998; Leshan, Fitzsimmons, Marbella, & Gottlieb, 1997; Mazonson et al., 1996; Robinson, Thompson, & Black., 1996; Rokstad, Straand, & Fugelli, 1995; Tabenkin et al., 1995), education (Arco, 1997; Furman, Adamek, & Furman, 1992; Mortenson & Witt, 1998; Rose & Church, 1998; Tuckman & Yates, 1980), and mental health (Chorpita, Bernstein & Daleiden, 2008; Chorpita, Bernstein, Daleiden, 2011; Howe, 1996; Lambert et al., 2001; Lambert et al., 2005; Lambert, Hansen & Finch, 2001; Mazonson et al., 1996). While quality improvement tools, such as MFSs (Cebul, 2008), have been successfully applied for several decades (Kluger & Denisi, 1996; Rose & Church, 1998), their use is not widespread within children’s mental health services. In many states, mental health services are being structurally reorganized and integrated into general health systems. Consequently, the use of MFSs to track outcomes of services for youth and families will gain even more traction (Bruns, Hoagwood & Hamilton, 2008).
This paper focuses on the implementation of a specific MFS called Contextualized Feedback System (CFS™; Bickman et al., 2011; Bickman, Kelley, & Athay, 2012) in two outpatient community-based mental health clinics in New York State. Clinicians and clients completed treatment progress information at every clinical encounter. CFS provided feedback to agency personnel (director, supervisors and therapists) on mental health progress and therapy process variables. This paper is a companion to Bickman and colleague’s in this issue. Both papers focus on the implementation of the CFS within the same two clinics. Bickman et al’s paper focuses on the impact of CFS implementation on client outcomes, focusing primarily on the degree of implementation measured quantitatively. In this paper, we focus on the complexities of the adoption and implementation process within these same agencies, examining multilevel factors from both a qualitative and quantitative perspective.
According to the theoretical framework of Aarons, Hurlburt, and Horwitz (2011), the adoption and implementation process is multilevel as well as multiphasic, with many challenges influencing how evidence-based practices are successfully implemented at different stages over time. This may account for the lag between the development of evidence-based practices and their widespread use. The process of adoption and implementation thus appears to be as critical to the overall effectiveness of the program as the specific treatment itself (Aarons et al., 2011; Fixsen, Blasé, Metz, & Van Dyke, 2013; Wisdom, Chor, Hoagwood, & Horwitz, 2013). Both inner context (e.g., organizational structures or processes, priorities, change readiness, openness to adoption, innovation-values fit, provider characteristics), and outer context factors (e.g., sociopolitical, funding, interorganizational networks, client advocacy) influence the adoption and implementation of innovations (Aarons et al., 2011). Studies that have examined barriers and facilitators to the implementation of evidence-based practices (EBPs) in children’s services, have highlighted multilevel factors ranging from organizational level factors (e.g., leadership support for training), the fit between the innovation and the service context (e.g., ease of use), and individual level provider and consumer factors (e.g., views of the usefulness of the intervention, competing demands, logical issues) (Aarons et al., 2007; Aarons et al., 2009; Langley et al., 2010). The present study extends this work to elucidate the interplay between barriers and facilitators that may influence the implementation of a technologically sophisticated measurement feedback system. Given the rapid rise of interest in the use of technology for monitoring mental health outcomes, identification of distinct implementation barriers associated with technology is particularly timely.
In this study, we focus on inner-context factors because CFS specifically pays attention to customizing the training and implementation of the measurement feedback system to fit an organization’s context. Research on organizational social context demonstrates that the context within agencies affects staff work attitudes and thereby an agency’s ability to improve their services (Peters & Waterman, 1982; Osborne & Gaebler, 1992). Organizational social context have been found to affect service implementation and quality (e.g., Aarons and Sawitzky 2006; Brunette et al. 2008; Carr et al. 2003; Glisson 2008; Glisson and Durick 1988; Glisson and Green 2006; Glisson and Hemmelgarn 1998; Glisson et al. 2013a; Glisson et al. 2008a; Glisson et al. 2010; Greener et al. 2007; Guzzo et al. 1985; Neuman et al. 1989; Olin, Williams, et al., 2014; Parker et al. 2003; Robertson et al. 1993; Sheridan1992; Shim 2010). In particular, Glisson and colleagues have found that a more positive organizational social climate improves uptake of new innovations (Glisson & Hemmelgarn, 1998; Glisson, 2002; Glisson & James, 2002). Thus, an effective innovation with proven efficacy in one context may fail to deliver expected effects in another due to barriers encountered in the different context. Importantly, strategies to target aspects of organizational culture and context can improve work environments to create organizational contexts that support the uptake of new practices and improve youth outcomes (Glisson, Hemmelgarn, Green, Dukes, Atkinson & Williams, 2012; Glisson, Hemmelgarn, Green & Williams, 2013; Glisson, Schoenwald et al., 2010).
In line with existing implementation theories, the CFS training focused on engaging leadership in supporting CFS implementation. Perceptions of risk and its management by agency leadership can affect uptake of innovations. Panzano and Roth (2006) applied a risk-based decision-making framework to examine the decision to adopt evidence-based and research-guided practices within agencies. The authors noted that decisions to adopt EBP innovations are likely to be political and complex, representing both strategy and risks. The investigation examined data from 83 projects involving 66 organizations. Overall, they found that adoption practices were related to perceived risks and perceived capacity to manage risks. Compared to non-adopters of EBPs, ‘early-adopters’ perceived fewer risks and saw the risks as more manageable. Such perceptions during the adoption phase influence the implementation process.
The current study followed the theoretical framework of Aarons et al (2011) to examine multilevel factors (organizational, innovation, staff and client level) that influenced the uptake and implementation of Contextualized Feedback System (CFS) (Bickman et al., 2011; Bickman et al., this issue). The differential uptake and impact of CFS at two clinic sites provided a context for examining factors that may be important in implementing measurement feedback systems such as CFS. In this paper, we describe data from qualitative interviews that were purposefully and systematically collected with the goal of augmenting the Bickman et al’s quantitative study (this issue). We hypothesized that implementation of CFS would be influenced by both barriers and facilitators at multiple levels, including characteristics of the outer context, organizational factors, provider factors, and the innovation (CFS) itself. Specifically, we expected that many of the barriers to CFS implementation would be similar across all clinics. Further, following Panzano and Roth (2006)’s concept of risk management and strategic fit, we hypothesized that CFS implementation would be facilitated by leadership capacity to manage these risks, or more specifically barriers associated with CFS implementation. The clinic that implemented CFS better was hypothesized to have leadership support to overcome barriers and align CFS implementation with their agency’s mission.
Thirty clinics with at least 5 clinicians who had been trained on EBPs through a state training initiative were invited to participate. Eleven applied for the project and four were selected based on their application, experience and success in implementing other state initiatives. These four clinics represented 2 agencies and were enrolled in the study. Two clinics from the same agency de-adopted at 9 months into the study. This paper focuses on the remaining 2 clinics from the same agency, who implemented CFS for two years.
CFS training and consultation followed the CFS Individual Site Training and Consultation Model previously described in Bickman, et al (this issue). In the current project, issues related to study resources and software development arose and notably altered the implementation process. First, there was a delay of over six months in the introduction of the technology due to development issues. Second, the developers introduced updates of the CFS program midway through the project. Third, at one of the sites (Clinic R), a senior administrator and her assistant assumed primary responsibility for facilitating participation and understanding of CFS because they were more directly accessible to the staff and were hoping to integrate and sustain CFS within the organization as part of the agency’s quality improvement efforts. At the other site (Clinic U), project staff conducted in-person consultation on a monthly basis to facilitate participation and understanding of the program. Finally, the agency issued a mandate midway through the project for both sites requiring the use of the program on all eligible cases and backing the mandate by including CFS use in performance evaluations. Therapists at both clinics, hereafter referred to as Clinic U and Clinic R, continued to have ongoing consultation either in person or via the telephone on the use of CFS data from the individuals described above.
As previously described in Bickman et al. (this issue), clinicians across both sites implemented CFS differently. Clinicians’ questionnaire completion rate and feedback viewing at Clinic R were 50% higher than clinicians at Clinic U. Mode of data entry was also different between clinics, with clinic R caregivers and clients more frequently using the computerized system instead of paper and pencil. Computerized data entry in both clinics by clinicians was equally high (over 95%). Overall, Clinic R had better implementation as measured for this study than Clinic U (Bickman et al., this issue).
At the end of the project, a list of staff who used the CFS program was obtained from clinic supervisors and administrators totaling 21 participants. Study staff approached 13 clinicians from Clinic U and 5 clinicians from Clinic R (3 clinicians of the initial 8 could not be interviewed because they were interns and had left their position with the agency). Of the clinicians approached to participate, 100% agreed to participate. The three intern-clinicians were not approached to participate because no contact information for them existed. In-depth, semi-structured exit interviews were conducted with a total of 18 clinicians (13 from Clinic U, and 5 from Clinic R).
Demographic and pre-implementation measures were administered to and completed by all 21 participants at the initial training (Clinic U n = 13 and Clinic R n = 8). These measures are described further below.
Clinic U served a primarily urban catchment area, whereas Clinic R served a rural catchment area. Moreover, 82% of Clinic U’s clinical population received Medicaid as compared to 30% of Clinic R’s population. (New York State Office of Mental Health Dashboard, 2011)
The sample of clinician participants at both Clinics was primarily female (75%) and between 26 to 30 years of age (43%). Clinic U had a more ethnically diverse staff (Caucasian 46%, African-American 23%, Multi-Racial 31%, and Hispanic 13%). At Clinic R, therapists were primarily Caucasian (88%) with the remaining 12% identifying as Multi-Racial. While site differences in the age and racial/ethnic composition of clinicians were not statistically significant, there was a trend for clinicians at Clinic U to be younger and more racially diverse than at Clinic R, with the racial composition of clinicians reflecting the racial composition of their clients.
The majority of clinicians held a Master’s degree in Social Work, endorsed an unspecified orientation (48%), and had less than one year in their current position (47%). About a fifth (19%) of the clinicians had no experience at all providing services for children or youths either in their current work place or in any other place before they used CFS. The limited experience may be accounted for by the fact that 2 of the clinicians at Clinic R were interns and thus came with little or no prior experience. Finally, 52% of the clinicians were licensed to practice in the state in which they currently worked.
At pre-implementation, clinicians at Clinic R reported larger caseloads of children over the age of 11 than those at Clinic U which is important to note because this was the target population for the implementation of CFS. At Clinic R, 60% of clinicians had more than ten clients 11–18 years old compared to only 16% of clinicians at Clinic U. Although aggregated pre- implementation differences were not statistically significant (likely due to the small sample), these site differences may be clinically meaningful and may influence CFS implementation.
Interview questions were developed based on Aarons et al (2011) as a broad conceptual framework; further, we used the Klein and Sorras’ (1996) measure that focused on facilitators and barriers to technology implementation to develop more domain-specific questions. The key domains included: General Issues, Quality of Training, User Manuals and Technical Assistance, Management Support, Technical Issues, Clinical Use and Other. These domains guided the development of interview questions in our study (see Appendix A). The project staff who administered the interview were trained in qualitative interviewing techniques. After completing interviews, project staff routinely debriefed.
Both qualitative and quantitative methods were used in this investigation. A trained research assistant conducted individual interviews over the telephone with participating staff at the two clinics over a four-month period. The semi-structured phone interview lasted up to thirty minutes. Participants were asked about the process and experience of using the system including characteristics of the innovation and implementation. Interviews were audiotaped and transcribed verbatim. A content analysis approach was used to analyze the data in which codes were developed based on barriers and facilitators that are commonly identified in theoretical models of innovation implementation (Bernard & Ryan, 2009). The interviews were analyzed by research staff using Atlas.ti 6.2 software, which facilitates the coding, organization, and retrieval of qualitative data (Friese, 2011). To develop the initial code list, an iterative process was used in which codes were developed based on the theoretical model, and modified through a process of independent coding, discussion, and refinement by members of the research team. Once the codes were established, two coders separately coded all interviews. Any disagreements were resolved through consensus. Kappa coefficients show reliability across coders was 0.87 for all interviews based on blinded double-coding of all text. After the completion of all coding, the team categorized coded material related to barriers and facilitators at multiple levels that are consistent with existing implementation frameworks (e.g., Aarons et al. 2011, Wisdom et al., 2013). Factors related to facilitators and barriers of CFS implementation were categorized into broad categories of innovation, organization, and individual (staff and client) levels. An external environment category was not included in this particular investigation because clinics were part of the same agency and experienced similar outer context factors including clinic restructuring. Positive comments were coded as facilitators whereas negative comments were coded as barriers. See Tables 1 and and22 for a list of barriers and facilitator codes and subcodes.
Clinic R was found to be more successful in implementing and using CFS than Clinic U (Bickman, et al., this issue). We thus examined and compared clinician reported facilitators and barriers to CFS implementation across sites, as seen in Tables 1 and and22.
Review of clinician interviews resulted in 169 unique implementation themes of which 119 were coded as barriers. There are thus more than twice as many reported facilitators (n = 50). For a list of barrier and facilitator codes, definitions and quotes elucidating the categories please see Table 4. To account for the unequal number of clinicians across sites, comparisons across sites were based on the mean number of barriers or facilitators reported per clinician. As see in Table 3, the average overall number of coded barriers per clinician (m = 6.61, sd = 2.28) was greater than facilitators (m = 2.78, sd = 2.05). This was true at both clinics across all categories (i.e., innovation, organization, client, and staff, and other). Interestingly, clinicians at the higher implementing clinic (Clinic R) reported a higher barrier to facilitator ratio (almost 3:1, 83 barriers to 30 facilitators) compared to clinicians at Clinic U (almost 2:1 (36 barriers to 20 facilitators).
Staff reported a total of 119 uniquely coded barriers across all categories (e.g., innovation, organization, other, client and staff, see Table 4 for exemplar quotes). There were no differences in the mean number of coded barriers reported by clinicians at Clinic U (m = 6.31, sd = 2.32) compared to those at Clinic R (m = 6.55, sd = 2.68).
By far, the most prevalent type of barrier was related to characteristics of the implemented technology. In total, 48% of the 119 coded barriers were in the innovation category (See Table 3). Within the innovation category, barriers were coded into 6 subcategories that are listed in order of their prevalence as defined by the number of participants that reported a barrier in the coded subcategory (See Table 1). The most common reports of innovation barriers were the system being time consuming (89%) and having a complicated design (83%). Other common innovation barriers included: a) burdensome technical requirements (50%) b) difficult to understand language (39%), and c) difficult to apply clinically (39%) (see Table 4).
The second most prevalent category of coded barriers was organizational, accounting for 19% of the 119 coded barriers (see Table 3). Organizational barriers included two main subscales: Insufficient resources and structure (61%), and Lack of implementation efforts (56%). One clinician described that they often had to complete measures outside of session or on their own time and this is what the agency wanted.
The least prevalent barriers were related to client (13% of all barriers) and staff level categories (5% of all barriers). In the client category, the most frequently reported barriers included a) Lack of readiness/capacity to adopt innovation (reported by 39% of clinicians), and b) Lack of computer literacy (reported by 28% of clinicians). One of the most notable differences between clinics occurred at this level, with four out of five clinicians (80%) in Clinic R, the more rural clinic, noted that clients’ lack of readiness and capacity to adopt CFS was a barrier, compared to only 3 of 13 (23%) clinicians in the more urban site (Clinic U). Only clinicians in Clinic U noted that client speed with completing surveys (2 of 13 clinicians) and lateness to appointments posed a barrier (1 of 13 clinicians). The majority of clients at Clinic U completed the forms in a paper and pencil format (youth= 60.6%; caregiver= 78.2%) versus those at Clinic R who completed them on the computer (youth= 84%; caregiver= 60% respectively).
With regard to staff level barriers, less than one fifth of clinicians across both sites noted barriers at the individual or personal level, such as not seeing CFS as a priority or being uncomfortable with technology. In one example an individual noted, “I am an older staff member and I am not that computer savvy.”
The Other category accounted for 15% of the 119 coded barriers; these included the following subcategories listed in order of prevalence based on number of clinicians reporting: a) Insufficient training (44%), b) Project’s nature as a research study (39%), c) System rolled out too early (17%). Examples describing the insufficient training barrier read, “It was a little too much to take at the training. I was feeling a little lost.” “I think we needed more training, that’s all.”
Across all 18 participants, there were 50 uniquely coded facilitators across the innovation, organizational, client, staff, and other categories (see Table 4 for quotes). Overall, Clinic R participants reported more facilitators on average (m = 4.0, sd = 2.35) than did participants at Clinic U (m = 2.31, sd = 1.80). Interestingly, a difference at the organizational level appears, such that Clinic R clinicians reported more organizational facilitators (m = 1.8, sd = .45) than those at Clinic U (m = .69, sd = .48).
The most frequently reported facilitators by clinic staff were in the organizational category, accounting for 36% of all coded facilitators. Under this category, all clinicians (100%) in Clinic R reported leadership championing of CFS as a facilitator compared to less than half of clinicians at Clinic U (46%). Similarly, four out of the five clinicians (80%) in Clinic R perceived organizational support in terms of implementation efforts as a facilitator compared to 3 of 13 (23%) in Clinic U. As an example of leadership facilitation, one person stated, “They’ve done everything that they could do to make everyone take this absolutely seriously.”
Innovation facilitators accounted for 28% of all reported facilitators. The most frequently noted facilitator in this category related to the clinical applicability/relevance of CFS (33%). One individual described the relevance and clinical use of the program as it pertained to one case where the client divulged a clinical issue on the CFS measures but not directly to their clinician.
Other facilitators accounted for 24% of the total reported facilitators and encompassed the following two subcategories: a) clinical support (39%) and b) training support (28%). As with barriers, client and staff level factors accounted for the lowest proportion of facilitators coded, with each accounting for 6% of the total reported facilitators.
States and other entities have been strong proponents and investors in the dissemination and implementation of evidence based practices to improve care quality. Understanding facilitators and barriers to innovation implementation from the perspectives of staff is critical because they are typically the stakeholder group responsible for and hence challenged with the implementation of such innovations. Given the challenges associated with implementing a new technology like CFS, it is not surprising that clinicians across both clinics identified two to three times as many barriers than facilitators. Notably, the pattern of barriers was similar across both clinics, and both reported proportionately more barriers than facilitators, regardless of level of implementation. In fact, the clinicians in the higher implementing clinic reported a higher ratio of overall barriers to facilitators than those in the lower implementing clinic. The finding that higher implementing clinicians reported many more barriers may reflect their increased effort and experience in working through the challenges in integrating a new practice.
Interestingly, it appears that one distinguishing factor between the 2 clinics pertained to the average number of organizational level facilitators of CFS implementation, with the higher implementing clinic reporting more organizational level facilitators to CFS implementation. This finding is consistent with Panzano and Roth’s (2006) notion that perceived capacity for managing barriers may be key to adoption success and by extension, implementation effectiveness. Thus, despite clinician perception of similar barriers across sites, the clinicians in the higher implementing clinic (Clinic R) reported more leadership support, including champions of CFS and concomitant training and implementation support onsite to help them implement the new technology over time. At Clinic R, an internal senior clinic administrator and her assistant provided the ongoing consultation, clinical supervision and technical support in the use of CFS. The senior administrator provided more immediate oversight of CFS implementation. At Clinic U, a Doctoral level staff person provided regularly scheduled consultation on a monthly or bimonthly basis. While the project staff may have been as technically competent as the internal senior agency staff, it is plausible to assume that the internal staff had greater influence on the behavior of the clinicians. Clinic R having greater day-to-day involvement of senior leadership staff than Clinic U may have generated greater engagement of all staff. Even though staff at both clinics were required to complete CFS measures as part of performance evaluation, this did not occur until well into the implementation phase. Another possibility is that having in-house staff support resulted in accessibility to supports that facilitated clinician implementation behavior; Clinic U had to rely on external project staff to provide in-person support on a scheduled basis. There is emerging evidence that active and explicit leadership for innovations, both in terms of general support and setting expectations can have a positive impact on implementation (Hall & Yip, 2014; Martin et al., 2011; Unsworth, Cowie, & Green, 2012; Wolpert, Curtis-Tyler, & Edbrooke-Childs, 2014).
In this project, the most prevalent category of barriers was the innovation itself. As has been increasingly recognized in the literature, the fit between the innovation system and the organization is a critical factor (Aarons et al., 2011). While the clinical utility of CFS was valued, the design and technological demands of CFS use posed significant challenges for the majority of users. Specifically, clinicians described that it took considerable time and effort to learn, implement, and use feedback effectively. Time is a commodity in short supply at both the clinics. In addition, challenges with CFS design and technology interfered with ease of use. For example, a number of individuals stated that it took too many clicks on the computer to get where they wanted. Over the duration of the study, many of these features were changed and system upgrades were made to address these issues. However, these negative experiences likely created barriers by reducing clinician openness to the system and perception of clinical utility.
Related to innovation characteristics, CFS software was the same in both sites but the way the feedback was implemented was different. Clinic U did not use computers or the provided tablets for youth or caregivers to answer the questionnaires at the end of the session. This resulted in significantly more delay in the clinicians receiving the feedback since the paper forms had to be hand entered. The additional delay could have weakened the effectiveness of the feedback and impacted the implementation effort in Clinic U. Additionally, Clinic U had younger clinicians with a smaller number of relevant caseloads; they reported less organizational leadership; and they apparently experienced more problems or had more difficulties resolving problems with the computerized technology since they used paper instead of web-based entry. These various factors could have affected attitudes and behaviors and hence CFS implementation.
In addition to innovation characteristics, organizational barriers were the second most commonly described. Conflicting priorities with organizational demands such as productivity or other agency-mandated paperwork were often highlighted. Lack of direct support from administration to implement CFS was also a significant barrier for clinicians in the lower implementing clinic. Clinicians also described their internal struggle between prioritizing the completion of CFS according to agency priorities and their belief that, even with the information CFS supplied, the burden of CFS completion sometimes interfered with the content and process of their therapy sessions.
Client and staff level factors were the least frequently mentioned facilitators and barriers perhaps due to response bias. This finding may reflect the emphasis CFS training had on supporting staff and problem solving issues that occurred. On the other hand, this finding may also reflect the powerful influence of organizational social context in innovation implementation; staff may have emphasized organizational factors because of the critical role organizational culture (e.g., work priorities; expectations of frontline providers) played in their ability to take on and implement new technologies. The role of organizational factors in service and implementation quality has been well documented (e.g., Aaron et al., 2011).
This important role of organizational support in innovation implementation is supported in our case study. Organizational factors were the second most commonly reported type of barrier, and also the most commonly reported facilitator of CFS implementation. Thus organizational issues appear to exert both positive and negative effects on implementation. Interestingly, the only significant difference found between the two implementing clinics was related to organizational support. Despite being part of the same agency and operating under the same agency mandate to implement CFS, the higher implementing clinic reported significantly more organizational facilitators than the lower implementing clinic. Consistent with the literature, the agency leadership support and having a champion were endorsed as a facilitator by every clinician in the higher implementing clinic. By contrast, fewer than half of the clinicians in the lower implementing clinic perceived leadership support for this effort despite the mandate. Geographic distance may have been a factor affecting proximity to leadership. Clinic R was co-located with leadership administrative offices while the other clinic was almost 100 miles away, and thus had less direct access to agency support. Availability of leadership support and ongoing efforts by leadership to facilitate implementation likely mitigated clinician perception of barriers, creating a more receptive climate for implementing a new technology.
The Other category of barriers included aspects of the CFS training that may be unique to CFS as a technology and its development. At the time when the CFS training was rolled-out, the most current version of the CFS system was still being finalized. Initially, clinicians and users were trained with screen shots but not a fully working CFS program. After initial training, a Beta version was implemented and updated throughout the program. Participating clinicians reported feeling lost with some aspects of the training especially in the beginning when they were asked to imagine the use of the system in relation to their organization or clinical procedure.
This is a case study that included only 2 clinics and a small sample of clinicians. As mentioned earlier, two of the original four clinics that started the project de-adopted before they implemented the full system thereby limiting our ability to get feedback from them on the implementation process. Moreover, due to clinician attrition at the 2 implementing clinics, the sample did not include all clinicians who implemented CFS. Our findings may thus be biased and have limited generalizability due to the small and restricted sample of adopters. It is possible that de-adopters experience different barriers and facilitators. Findings should thus be interpreted with caution.
Additionally, our crude counts of coded barriers and facilitators do not distinguish the relative importance of these factors in influencing CFS implementation. It is possible that some of the barriers and facilitators had a greater or differential impact on implementation than others. The relative impact of these various factors was not investigated.
As with many long-term implementation efforts, our study was subject to the ever-changing service system. During the course of the study, all clinics across the state were undergoing restructuring efforts that impacted how they were funded and paid. This had a major impact on caseload, length of session, and many other variables within each of the clinics. Interesting, participants did not mention clinic restructuring explicitly as a cause of implementation difficulty but the stressors imposed by the restructuring had and continues to have a large impact on most clinics across the state. This issue may be mitigated by the fact that clinic restructuring likely impacted both clinics similarly since they were part of the same parent agency.
Another study limitation relates to the use of semi-structured interviews rather than standardized measures to understand contextual issues around implementation. While semi-structured interviews have inherent limitations, the use of this methodology provided rich, detailed information not available from standard quantitative methods. Finally, the sole focus on clinicians and not other stakeholders from multiple levels of the organization is a limitation of the current study. Different stakeholders, especially in de-adopting clinics, are likely to have different perspectives about implementation, depending on their role. This would be an important area of future research.
This study highlights the importance of facilitating conditions, particularly the role of organizational support and leadership, in creating conditions that are conducive to integrating new practices. Our study also points to the importance of not focusing exclusively on barriers to implementation. While disparate clinics may be faced with similar barriers, the tipping point for improved implementation may be the presence of key facilitating organizational conditions that help users overcome the almost certain barriers that will undoubtedly arise. Future research on adoption and implementation of innovations should focus on the ratio of barriers to facilitators and the important role of organizational and leadership factors in tipping the balance to improve the installation of new practices in these complex child and adolescent healthcare systems.
Hi, my name is ________. I’m speaking with you today because we want to get more information about aspects of the CFS project that have gone well for you, or aspects that could be improved. We will be recording these calls for note-taking purposes, but all of your responses are confidential and won’t be shared with other members of your agency. Your answers will be used to better understand this quality improvement initiative. We really appreciate your time and willingness to speak with us. Do you have any questions before we get started?
Let’s start with some general questions about your overall experience during this project
Now I’m going to ask you about the clinical application of CFS and how the program fits with your clinical practice.
Those are all of the questions I have for you today. Thank you again for participating! We have one final piece of this interview--it’s an online survey that you should be able to complete quickly. I’ll send you the link to the survey now. It would be great if you could complete the survey as soon as possible, while your experience with CFS is still fresh in your mind.