Closing the gap between research and practice has been a priority for many agencies, including the U.S. National Institutes of Health, Veterans Health Administration (VHA) and the Agency for Healthcare Research and Quality [1
]. Despite the development of effective interventions to improve health care quality, most of these interventions have only been implemented in the academic settings in which they were developed, and few have been successfully disseminated into non-academic-affiliated (i.e
., community-based) organizations [2
]. A primary reason for this research-to-practice gap is the lack of a framework for implementing effective interventions in community-based organizations that maintains fidelity while maximizing transferability when the interventions are adopted across different settings [2
An effective strategy for implementing clinical and health services (i.e
., health care) interventions is needed. Many interventions fail to achieve the outcomes observed when tested in research settings once they are implemented in community-based organizations. This "voltage drop" in effectiveness maybe due to reduced fidelity to the intervention when disseminated outside the academic realm to community-based organizations, as well as lack of guidance in customizing interventions to community-based populations [5
]. Moreover, stakeholders (e.g
., health care purchasers, plans, and providers) increasingly have to choose which interventions to implement for their populations, and need guidance in implementing them in a cost-efficient manner without diminishing the intervention's effectiveness. At the same time, these stakeholders need to ensure that the intervention will be accepted and adopted across different organizations.
A number of strategies for guiding implementation efforts have been proposed [9
], many of which primarily focus on adapting interventions by (a) determining when an organization is "ready" to adapt or adopt an intervention, and (b) working with senior leaders and frontline providers to overcome barriers to adaptation. For example, the VA's Quality Enhancement Research Initiative (QUERI) framework outlines a process for choosing, implementing, and marketing evidence-based interventions in health care organizations [12
]. Simpson describes the underlying organizational characteristics that need to be considered (readiness to change, resources, and culture) when planning an implementation of evidence-based interventions [13
]. Bartholomew et al
. describe Intervention Mapping, which involves planning an implementation of a program based on information from the target population [14
However, these frameworks do not address a fundamental issue in implementation: achieving a balance between adequate fidelity to the intervention and accommodating differences across organizations in order to maximize the effectiveness of the intervention. To date, no implementation frameworks have outlined strategies for maintaining treatment fidelity while providing opportunities to adapt interventions to fit local needs. For example, previous frameworks do not provide specific guidance in replicating interventions for use in community-based organizations, and do not specify training or technical assistance programs for facilitating implementation across different settings. Without the appropriate tools and materials available in a user-friendly format, community-based organizations are less likely to implement an evidence-based intervention; or if implemented, treatment fidelity may be suboptimal.
In this article, we describe an innovative implementation framework: Replicating Effective Programs (REP). In contrast to previously published implementation frameworks, REP provides a roadmap for implementing evidence-based interventions into community-based settings through a combination of intervention "packaging," training, technical assistance, and other strategies to maximize the chances for sustaining the interventions. REP has been empirically evaluated through a randomized controlled trial of its effectiveness in achieving intervention uptake and fidelity across different organizations [15
]. To date, other implementation frameworks have included some but not all of the components of REP [9
], and there have been no evaluations of implementation frameworks that combine strategies to maximize both fidelity and flexibility in implementing interventions.
History and underlying theories of REP
REP was developed and applied by the U.S. Centers for Disease Control and Prevention (CDC) to package and disseminate HIV behavioral and treatment interventions for implementation in community-based service settings, notably AIDS service organizations [17
]. CDC's REP project was initiated in 1996 to address a critical link in bridging research and practice; namely, packaging interventions so that they can be easily implemented in non-academic, community-based settings. Extensive research has focused on developing and evaluating interventions designed to change risk behaviors related to HIV transmission since shortly after the first cases of AIDS were reported in the United States [18
]. The CDC's approach involved: applying rigorous criteria to identify effective interventions [19
]; packaging interventions' complete scientific protocols into non-technical language [20
]; supporting the implementation of the interventions through training and technical assistance [21
]; and providing ongoing financial and technical support to sustain the interventions' implementation [22
The REP framework (Figure ) builds on a systematic literature review and community input [19
]. The concepts underlying the REP packaging process derive from action anthropology (wherein a neutral party mediates interaction and exchange between two cultures – in this case, research and practice) [25
] and principles of health promotion [26
]. The foundational theories for disseminating the interventions are Diffusion of Innovation, which posits that innovations pass through particular communication channels among members of a social system over time [24
], and Social Learning Theory, which describes the relationship between behavior change and persons' beliefs about their ability to change and the results of the change [27
Replicating effective programs framework for health care interventions. This figure outlines the Replicating Effective Programs (REP) process as it can be applied to health care interventions.
There are four phases to REP [22
]: pre-conditions, pre-implementation, implementation, and maintenance and evolution (Figure ). The four phases are described in detail in Table as they would apply to health care interventions. When disseminating interventions, the first three phases of REP (pre-conditions, pre-implementation, and implementation) are most appropriate for the "early adopter" and "early majority" types of community-based settings based on the Rogers Diffusion of Innovation model [24
]. The final phase, maintenance and evolution, is more appropriate for organizations described as the "late majority" or "laggards" based on the Rogers model, as they are more likely to participate once the intervention is already being used by other organizations and with appropriate organizational and financial incentives [24
Outline of REP process for health services-based interventions
Pre-conditions for intervention adoption are: the identification of the need for a new intervention for a target population; identification of an effective intervention that fits local settings (e.g., mission of organization and benefits to the organization) and the intended target population (e.g., behavioral risks and culture); identifying implementation barriers; and drafting a user-friendly manual (i.e., "package") of the intervention.
The first step of the pre-conditions phase is to identify appropriate at-risk populations and suitable evidence-based interventions.
Identifying effective interventions
Interventions are suitable for REP if they have been rigorously evaluated (e.g., via randomized controlled trials or other quantitative studies) and have produced statistically significant positive effects on the health outcomes of interest.
Ensuring intervention fits local settings
In reviewing effective interventions, researchers should be cognizant of whether the intervention has been shown to be effective in patient populations similar to those in the organizations taking part in the implementation process. Many community-based organizations serve a disproportionate number of low income, minority, and co-morbidly ill patients when compared to patients from academic settings. Another consideration is determining what the "usual care" conditions were in the effectiveness studies. Often academic practices have resources that facilitate the conduct of research studies (e.g., staffing, electronic medical records, space), which many community-based organizations lack.
Identifying implementation barriers
To ensure that the intervention is feasible in local settings, researchers should meet with the staff members from participating organizations, introduce the intervention, and conduct an assessment of potential barriers to its implementation. Such meetings also foster buy-in, especially if the researchers develop a list of benefits of participation, from training opportunities to potential cost savings (i.e., the "business case"). During these meetings, researchers should gather information in the form of a needs assessment to benchmark usual care, including resources and functioning characteristics that may directly affect the mode by which the intervention will be implemented and the types of technical assistance that might be needed (e.g., staffing, patient volume, management characteristics, information technology capability). This information can inform the content of the intervention package.
Drafting the intervention package
An intervention package is then drafted that is suitable for implementation in community-based organizations. Training and technical assistance plans are also drafted during this phase. Packaging involves the conversion of intervention protocols into non-academic language and into a user-friendly manual that can be readily disseminated [20
]. The package should be drafted by the developers of the intervention, but edited by non-technical writers to ensure that the intervention package materials are free of technical jargon. Supporting documents, such as a staff training curriculum, a technical assistance guide, and promotional materials are also included. The package is refined and finalized during the pre-implementation phase of REP based on input from the pilot-testing organizations.
The REP intervention package conveys the intervention's theoretical foundation (i.e
., core elements), components, and methods. Core elements – the critical features of the design and intent of the intervention that are thought to be responsible for the intervention's effectiveness – are essential for maintaining fidelity. Still, while core elements are standardized, the mechanisms by which they can be operationalized may vary across organizations. The package can therefore articulate menu options for adapting intervention delivery across different organizations to allow flexibility in implementation. For example, menu options for organizations with a substantial number of minorities may include culturally appropriate options for delivering the intervention, such as outreach to families or community groups. Prior evidence from the management literature suggests that articulating a priori
the core elements and adaptation options of a new product to potential users of the product is necessary for its successful adaptation [28
]. Thus, having the core elements detailed, while also providing options for implementing these core elements, is vital for optimizing both fidelity to the intervention and flexibility in its implementation.
The REP packaging process has distinct advantages over current strategies for disseminating interventions. For example, a number of behavioral health intervention toolkits (i.e
., descriptions of the intervention components) have been published or posted on websites, such as the U.S. Substance Abuse and Mental Health Services Administration's National Health Information Center Evidence-Based Practices website [29
]. However, unlike REP packages, these toolkits often lack a detailed description of the intervention's set-up procedures, underlying theories and logic flow, scripts, and other specific materials, as well as instructions for implementing these materials across different settings. In contrast, REP packages provide specific details regarding the intervention as well as operationalized options for adapting delivery of intervention core elements to local organizations in a way that does not compromise the intervention's core elements. The REP packaging process has already been widely used by the CDC to successfully implement and disseminate HIV interventions. For example, between 1996 and 2005, the REP project packaged ten interventions for use by health departments, clinics, and community-based organizations; and more packages are now being prepared. CDC and state health departments fund the implementation of REP-packaged interventions by over 500 prevention organizations nationwide [30
Pre-implementation involves 1) input from a Community Working Group (CWG) on developing the package, training, and technical assistance programs, 2) package pilot test, and 3) orientation and logistics in preparing the intervention for implementation.
Community working group – develop package
To accomplish the aims of the pre-implementation phase, the intervention developer convenes a group of stakeholders from organizations serving the target population to participate in a Community Working Group (CWG). We use a comprehensive definition of stakeholders based on the Pincus multi-level 6-P framework that describes the different levels of health care: populations (i.e., communities), purchasers, plans, practices, providers, and patients (consumers) [30
]. Given that these different levels play an important role in the use of REP to implement interventions, representatives from all levels should be considered for active participation in the CWG. For example, providers would be knowledgeable of the day-to-day barriers to implementing the intervention, consumers can comment on participant recruitment and burden, and health plan leaders can provide input on how to sustain the intervention beyond the REP implementation phase (e.g
., reimbursement of intervention services). CWG participants should be persons who have influence over their peers (i.e
., opinion leaders), yet at the same time provide input on the real-world experiences and have the clout to leverage any necessary changes to minimize barriers to intervention implementation.
The CWG members meet regularly throughout the pre-implementation phase to review the intervention materials, finalize the prototype package, advise on the staff training and technical assistance plans, and plan the logistics of the implementation. The goal of these meetings is to review and advise on the package content and develop options suitable for customizing the intervention to specific institutions (e.g., different options for organizing group sessions, care manager communication protocols with providers, guideline dissemination techniques). The intervention developers or researchers familiar with the science and core elements of the intervention should also participate in the CWG in order to provide expertise in the intervention. Notably, core elements and menu options are refined based on CWG input, but the final decisions to include menu options are made by the intervention developers/researchers. Whenever possible, notes should be taken at the CWG meetings to preserve insight and feedback on the package and implementation process garnered from the CWG members.
The final package should include the following components: the intervention technical manual, training curriculum, and guidance for the technical assistance. The technical manual contains a general overview (operationalized core elements, recruitment tips, and selling points for organization staff and leaders on the return on investment); intervention materials (verbatim scripts, and recommendations such as selecting an appropriate venue to conduct the intervention), as well as session workbooks and exhibits; a document describing staff members' roles, time and resource requirements, job qualifications for staff involved, and supervision guidelines; electronic files of printable materials pertinent to the intervention; and pocket cards, flowsheets, and other materials to facilitate the intervention implementation. All materials should be able to be photocopied and easy to read.
Pilot testing the package
Once the package has been reviewed by the CWG, it can be tested for clarity and functionality within a few intervention sites. During this process, the organizations can identify five to eight patients to participate in a full intervention cycle, and study investigators should collect information on feasibility, acceptance, and any problems with the overall package, so that the package can be refined based on their input.
Orientation and logistics
Based on input from the site visits, needs assessments, pilot test, and CWG, the prototype intervention package, training, and technical assistance plans should be finalized for distribution. The package can be distributed through orientation meetings, which can be helpful in bringing recipient organizations together as well. Prior to package distribution, a program champion should be identified at each organization. Program champions can help identify appropriate staff members who can implement the intervention and mobilize support for the intervention within their organization. Program champions, along with key staff members who will be implementing the intervention, should be asked to attend the orientation meeting.
The implementation phase begins upon the dissemination of the REP package to organizations, and continues with intervention training, technical assistance, and evaluation (e.g., fidelity and outcomes monitoring). The CWG continues to be involved in this phase to assist researchers in interpreting feedback from the evaluation process.
Training of selected staff members at participating organizations is essential in implementing and sustaining the intervention. The training should begin soon after package dissemination and should review the manuals and include role playing to showcase specific communication skills with participants. Booster training sessions should also be implemented later in the implementation process.
The technical assistance (TA) component usually occurs after training and involves regular phone calls with the organization representative within a month after training is complete. Regular calls are necessary in order for the TA experts to be proactive rather than reactive when problems regarding implementation need to be solved. The trained TA specialist takes notes during the call and documents implementation progress at each organization. He or she also advises on how to maintain fidelity (e.g., discerning core elements from menu options), integration with existing services, and troubleshooting the implementation process. One of the biggest challenges is ensuring that the core elements are maintained (fidelity) while its implementation is customized and adapted to local design specifications (flexibility). The TA specialist should be encouraged to discuss the tension between fidelity and adaptability with organizations, stressing that the essential core elements can be implemented using different methods while remaining consistent with the intervention's intent, and to provide positive feedback on the organization's efforts. The TA specialist should also be trained to address fidelity issues with both more- and less-experienced employees, as the former may be reluctant to change their traditional ways of patient care, while the latter may require more intensive training.
Four types of evaluation strategies ought to be considered: interpretative evaluation of the intervention implementation process; measurement of intervention fidelity at the organization and patient level; patient-level outcomes; and return on investment (e.g., costs). Interpretative evaluation involves collection of data via qualitative interviews of providers and consumers to determine how the intervention was actually implemented, and notes from training and TA calls to determine to what extent these programs were useful for the organizations. Intervention fidelity measures should be developed to determine whether core elements were successfully implemented. For example, medical charts and staff logs can be used to count the frequency and intensity of services provided, as well as patient attendance and adherence, and staff interviews can be used to gather additional information on whether core elements were implemented correctly. Patient-level outcomes, including processes of care and clinical and functional outcomes, should be chosen based on which outcomes the intervention was designed to impact. The return on investment (e.g., cost-benefit, or how does the cost of implementing the intervention compare to the savings in patient care) is one of the most important evaluations to be conducted, and represents a key argument in making the business case for the intervention to stakeholders (e.g., health care purchasers, plans, and consumers).