|Home | About | Journals | Submit | Contact Us | Français|
This study examines adoption and implementation decisions among organizations that purchased Project Towards No Drug Abuse from 2001 to 2004. Telephone interviews were conducted with 120 organizations nationwide. The most common reason for adopting the program was its evidence base. In schools, classroom teachers were more likely to deliver the program than other types of implementers, and in non-school organizations, prevention specialists and counselors were more common (p <.05). Most organizations (73%) reported that they delivered all of the program sessions. The limitations of the study, as well as the implications of the findings for future research and wide-scale prevention program dissemination, are discussed.
One of our most critical health challenges is to develop effective interventions that address priority health issues, and to assure that those interventions are being disseminated, implemented, and sustained, once proven effective. The need for translation of effective interventions from research to real-world settings is especially noteworthy in the area of substance abuse prevention (Rohrbach, Grana, Sussman, & Valente, 2006). There is now substantial empirical evidence indicating that a number of school-based programs are effective in preventing or reducing substance abuse among adolescents (Gottfredson & Wilson, 2003; Tobler & Stratton, 1997; Tobler et al., 2000). Nevertheless, studies examining the actual use of evidence-based programs in schools have demonstrated low rates of utilization and program implementation fidelity, or the extent to which the program is implemented as intended by program developers (Dusenbury, Brannigan, Falco, & Hansen, 2003; Ringwalt et al., 2009). For example, in a recent survey of use of substance use prevention programs among U.S. schools with middle school grades, Ringwalt and colleagues found that only 43% of schools were using an evidence-based program (Ringwalt et al., 2009). Furthermore, when implementation fidelity has been measured, results indicate that teachers commonly eliminate some of the key curriculum points, objectives, and/or modules (Botvin, Baker, Dusenbury, Tortu, & Botvin, 1990; Rohrbach, Graham, & Hansen, 1993; Tappe, Galer-Unti, & Bailey, 1995; Tortu & Botvin, 1989); are less likely to use the interactive teaching methods that are key elements of the program (Ennett et al., 2003; Tappe et al., 1995); and generally deviate from the program as written (Pentz et al., 1990; Ringwalt, Ennett, Rohrbach, & Vincus, 2004).
During the past decade, considerable effort has been devoted to disseminating information about evidence-based substance abuse prevention programs. These efforts have been strengthened by public policy such as the Safe and Drug-Free Schools and Communities Act, which was first authorized in 1994 (U.S. Department of Education [DOE], 2005). Beginning in 1998, this policy mandated that federal funds be used to support only those programs that provide evidence of effectiveness in reducing illicit drug use and violence. Several governmental agencies have established criteria for evaluating research on program effectiveness and developed lists identifying effective interventions (e.g., National Registry of Evidence-Based Programs and Practices (NREPP) of the Substance Abuse and Mental Health Services Administration (SAMHSA), 2005; Safe and Drug Free Schools Program U.S. DOE, 2001). Other non-profit agencies have developed similar types of registries (e.g., Blueprints for Violence Prevention, Elliott & Mihalic, 2004) or published standards for evidence of the effectiveness of prevention interventions (e.g., Flay et al., 2004). While the criteria for identifying a specific program as “effective” vary across these lists, in general, the evidence must be based on experimental or quasi-experimental evaluation studies and must demonstrate positive effects behavioral outcomes related to tobacco, alcohol, or other drug use or violence. Recent national studies have indicated that these types of information sources are quite influential in guiding school districts' decisions to adopt evidence-based substance abuse prevention programs (Hallfors, Pankrantz, & Hartman, 2007; Rohrbach, Ringwalt, Ennett, & Vincus, 2005).
While scientific information has been made available to help organizations select effective substance use prevention programs, less technical assistance has been directed toward increasing organizations' understanding of the necessary conditions for successful implementation of evidence-based programs. The lack of guidance in this area is probably related to two factors. First, there are substantial gaps in our scientific knowledge about the replication, dissemination, and wide-scale implementation of evidence-based prevention programs (Bowen et al., 2009; Elliott & Mihalic, 2004; Glasgow, Lichtenstein, & Marcus, 2003, National Research Council and Institute of Medicine, 2009; Woolf, 2008. In general, we know more far more about prevention programs when they are implemented under conditions controlled by researchers (i.e., efficacy trials) than when they are delivered under real-world conditions (i.e., effectiveness trials) (c.f., Flay, 1986). Second, many organizations have substantial deficits in their infrastructure for evidence-based prevention program delivery. For example, it is common that organizations neither commit the necessary resources to ensure successful program implementation, nor do they provide adequate training and technical assistance for program implementers or establish effective systems for program monitoring and evaluation (Elliott & Mihalic, 2004).
In order to enhance the dissemination and implementation of evidence-based prevention programs, we need to devote more attention to studying the characteristics of organizations that adopt evidence-based prevention programs, and increasing our understanding about what happens after organizations initially put programs into use (Kerner, Emmonds, & Rimer, 2005; Schoenwald & Hoagwood, 2001). In one of the few studies that have examined factors affecting adoption of evidence-based substance abuse prevention programs in school settings, Rohrbach and colleagues found that adoption was positively associated with a larger student enrollment, urbanicity, more administrative effort devoted to coordinating prevention programming, and organizational innovativeness (Rohrbach et al., 2005). A recent review of the literature on the actual use of behavioral health promotion programs in real-world settings showed that a number of organizational-level factors are associated with successful program implementation, including organizational climate (e.g., willingness to try new approaches, shared vision, and positive morale), effective leadership, and organizational practices that allow shared decision-making and open communication (Durlak & DuPre, 2008). Importantly, research that applies a marketing perspective and examines who adopts evidence-based prevention programs, how they are implemented, and how acceptable they are to users is likely to accelerate dissemination, in part because it helps to shift the focus toward the needs and priorities of program consumers (Rotheram-Borus & Duan, 2003; Sandler et al., 2005).
In the present paper, we describe the results of an exploratory study of organizations that were early adopters of Project Towards No Drug Abuse (TND), an evidence-based substance abuse prevention program that targets high school-aged populations (Sussman, Dent, & Stacy, 2002). The present study examined: (1) the types of organizations that adopted Project TND during the first three years of its public dissemination; (2) the reasons why organizations adopted the program; and (3) how the organizations implemented the program; and (4) whether adoption decisions and program implementation differed by type of adopting organization. The primary goal of the study was to obtain information that would guide the development of hypotheses to be tested in future research.
Project TND is an interactive curriculum for high school-aged youth that incorporates motivation, skills and decision-making material targeting the use of cigarettes, alcohol, marijuana and other illegal drugs, and violence-related behaviors (Sussman, Earleywine, Wills, Cody, Biglan, Dent, & Newcomb, 2004; Sussman, Rohrbach, & Mihalic, 2004). The program has been evaluated in five randomized trials which have demonstrated a reduction in the use of alcohol, tobacco, marijuana, and other illegal drugs at one-year follow-up (Dent, 2001; Sussman, Dent, Stacy, & Craig, 1998; Sussman et al., 2002; Sun, Dent, Sussman, & Rohrbach, 2008; Sun, McCuller, & Dent, 2003; Valente et al., 2007), an impact on cigarette smoking, marijuana and other illegal drug use at two-years follow-up (Sussman et al., 2003), and an impact on illegal drug use at five-year follow-up (Sun, Skara, Sun, Dent, & Sussman, 2006).
The Project TND curriculum is comprised of 12 sessions, each lasting approximately 45 minutes. The program is designed to be implemented by a teacher, counselor, or prevention specialist in a classroom setting over a four-week period. It is considered a school-based approach since it was developed for, and tested in alternative and traditional high school settings. It is strongly recommended that program implementers participate in a one- or two-day certified training workshop prior to program delivery.
The research team that developed and evaluated Project TND began dissemination of the program in 2001, eight years after the start of the first experimental trial. Two events that took place in 2001 created demand for the program and provided an impetus for its widespread dissemination. The first event was the identification by the Substance Abuse and Mental Health Administration (SAMHSA) of Project TND as a “model” program for substance abuse prevention and the subsequent dissemination of program information by SAMHSA via written and face-to-face communications. The second event was the passage of the No Child Left Behind Act of 2001 (U.S. Department of Education, 2005), which mandated that schools use funds provided by the Safe and Drug Free Schools and Communities Act to support only those programs based on “scientifically-based research.”
In 2001, we began developing a database of information on all organizations that purchased Project TND curriculum materials. For the present study, our goal was to conduct a telephone interview with all of the organizations (n=201) that had purchased at least one teacher's manual and/or student workbook between 2001 and 2004. Within each organization, we sought to interview the staff member responsible for coordinating the implementation of Project TND.
Data were collected by telephone over a one-year period, beginning in October 2004. Prior to administering the phone survey, our trained research staff called each organization to identify the individual who coordinated Project TND training and program delivery. After a maximum of five call attempts, we were able to identify the target respondent in 185 of the 201 organizations. Next, we made a maximum of five attempts to reach the target respondent, obtain his/her informed consent, and conduct a telephone interview. After the fifth attempt, target respondents who were not reachable by phone were sent a paper-and-pencil version of the survey to the mailing address in the database, with instructions and a stamped return envelope.
Four of the target respondents declined to participate. We completed an interview or obtained a returned mail survey from 149 of the 185 target respondents (80.5% response rate). The average length of the telephone interview was 20 minutes. Among the 36 organizations that did not complete the survey, one half was comprised of school districts and one half included other types of organization. In our data analyses, we eliminated 29 organizations that reported they had purchased the program solely for preview purposes or had chosen not to implement it for other reasons, resulting in a final analytic sample of 120 organizations.
Survey items assessed background information on the organization, adoption decision making, and program implementation. Two forced-choice items assessed the type of organization (i.e., school district, school, health department, community-based organization, or other) and the population groups served by it (i.e., regular high school, alternative high school, juvenile probation, general youth population, adults, entire community, and/or other). We coded the geographic location of the organization based on categories used in the U.S. Census (U.S. Census Bureau, 2000). In order to describe the population density and urbanicity of the organizations' service areas, we obtained the U.S. Census rural-urban code for each community, and collapsed the codes into three categories: urban, suburban, or rural.
One open-ended item assessed how the organization first became aware of Project TND, and another open-ended item measured why they chose to adopt it. Program implementation items assessed the setting for the program (one forced-choice item), type of implementer (one open-ended item), training of implementer (one forced-choice item), sources of funding for the program (one forced-choice item), as well as the total number of program sessions taught, implementation schedule, adaptations made to the program, and perceived students' and implementers' responses to the program (one open-ended item each). Mutually exclusive categories were developed for responses to the open-ended items, and responses were coded into categories by the first and second authors. Averaged across the eight open-ended items, inter-coder agreement was 94%.
Table 1 presents data on characteristics of the organizations that participated in the survey. Nearly one-half of the participants (48%) were community-based, non-profit organizations. Approximately one-third (37%) were local educational agencies (i.e., schools, school districts, or consortia of districts). The majority of organizations (53–57%) served youth enrolled in traditional and/or alternative schools. One-third of the organizations (33%) targeted youths on probation. The majority of respondents were located in the southern or western region of the United States. Two-thirds of the organizations were located in urban areas of the country.
Table 2 shows the respondents' initial source of information about Project TND and primary reason for adopting the program. Responses are compared for two categories of organizations, schools (including schools, school districts, and groups of school districts) and non-school organizations (including community-based non-profits, health/mental health departments, and other types of organizations). For the sample of organizations as a whole, the two most commonly endorsed reasons for adopting the TND program were its evidence base (43%) and because it appeared appropriate for the population group served by the agency (34%). For nearly two-thirds of the organizations, the primary initial source of information about Project TND was a list of evidence-based programs provided either by SAMHSA or their state or county government agency. The tests for differences between school and non-school organizations on reasons for adoption and program information sources were not statistically significant.
Overall, half of the organizations reported they implemented the program in regular high school settings and half implemented in alternative high schools (Table 3). The setting for the program did not differ for school and non-school organizations. In schools, classroom teachers were more likely to deliver the program than other types of implementers, whereas in non-school organizations, prevention specialists and counselors were more likely to deliver the program (χ2 =36.5, p <.0001). Overall, less than half of the organizations had obtained the certified training that is strongly recommended by the dissemination organization (University of Southern California). The most common source of funds for the program in schools was the federal Safe and Drug Free Schools and Communities Act (34%), whereas local funds (e.g., county grants) were more common sources of support for non-school organizations (χ2 =29.3, p <.0001).
Data on fidelity of program implementation is presented in Table 4. Overall, 73% of organizations reported that they delivered all of the program sessions. Non-school organizations were more likely to deliver the full program than were schools (χ2 =9.87, p <.05). Only 43% of organizations followed the recommended program delivery schedule of two to three lessons per week. Non-school organizations were more likely than schools to spread program delivery over a longer period of time (χ2 =6.4, p <.10). Schools were more likely than non-school organizations to report they had added materials such as videos and games while delivering the curriculum (χ2 = 3.4, p <.10). Overall, the perceived response to the program was positive among the majority of students and implementers (72% and 80%, respectively), and these impressions did not differ by the type of organization in which the program was implemented.
The present study examined adoption decision making and implementation fidelity among schools and community organizations that adopted Project TND, an evidence-based substance abuse prevention program. We began national dissemination of Project TND soon after the initiation of policies mandating that schools and communities that used federal funds to support substance abuse prevention programs were required to use programs with proven effectiveness. During our early dissemination efforts, one of the primary sources of information about the program was SAMHSA, which had developed a list of “model” programs for substance abuse prevention that was distributed widely via the internet and other communications1. In order to be identified as a “model, effective, or promising” program by SAMHSA, the program needed to directly or indirectly address risk and protective factors for substance abuse, violence, and other problem behaviors, and present evaluation evidence showing positive effects on behavioral outcomes. The SAMHSA “model program” list described Project TND as a “school-based” prevention program that had been evaluated in both regular and alternative high schools, and which is appropriate for implementation in “virtually any school or school district.” (SAMHSA, 2005).
In this context, we were somewhat surprised to find that the majority of Project TND adopters were non-school organizations, such as community-based non-profit agencies and county health or mental health departments. Although about half of the non-school organizations reported that they implemented the program in regular and/or alternative high school settings, more than half reported they delivered the program in community and/or juvenile justice centers. These findings have several implications for the dissemination of school-based substance abuse prevention programs. First, they suggest that a prevention program like TND may be transportable and acceptable to a variety of organizations that conduct prevention programming intended for school-aged youth, including both schools and community-based organizations. In future research, it would be useful to examine how different types of organizations develop the capacity to enable program implementation, and to evaluate the effectiveness of substance abuse prevention programs when they are implemented outside of school settings.
Second, our findings suggest that at least in some parts of the country, it may be easier for non-school organizations, relative to schools, to adopt and implement prevention programming for high school-aged youth. One may speculate that non-school organizations are in a better position to procure federal and state resources to afford training and delivery of publicly marketed programs. In addition, there may be more barriers to prevention program implementation by schools relative to other types of organizations. For example, decisions about which prevention curricula will be used in classrooms are often made at the school district level without the direct involvement of teachers, and these decisions may or may not lead to actual program implementation. Furthermore, schools in the United States are experiencing substantial turbulence due to reductions in funding, restructuring efforts, an emphasis on improving academic outcomes, and other factors. In such a context, schools may resist implementing special programs like substance abuse prevention curricula, for which they are not held accountable (Rohrbach et al., 1996). Thus, it is possible that our findings reflect a trend toward increased school-community partnerships that are established for the purpose of improving the reach and implementation of evidence-based prevention programming (e.g., Spoth and Greenberg, 2005).
One important finding of this study is that a substantial proportion of organizations that we interviewed adopted Project TND because it is evidence-based, which suggests that organizations may be placing more emphasis than in the past on evaluation research as a criterion for program adoption. This finding is encouraging, given that the intent of the reauthorized Safe and Drug Free Schools Act was to ensure that federal funds are spent on effective, rather than ineffective or unevaluated substance abuse prevention programming. For both school and non-school organizations, the SAMHSA “model programs” list was the most prevalent primary source of information about effective substance abuse prevention programs. Because federal agencies like SAMHSA are poised to play a critical role in dissemination of information about prevention programs, it is important that the information that they disseminate reflects a consensus from the field about the standards of evidence for effectiveness of programs. Unfortunately, the prevention field has not yet reached consensus about criteria for evaluating evidence on the effectiveness of specific prevention programs to determine which programs are ready for widespread dissemination (Elliot, 2008).
One of the challenges involved in wide-scale dissemination of evidence-based prevention programs is determining how to maximize fidelity of program implementation. Research on a broad range of educational programs and mental health interventions shows that when efficacious interventions are translated to use in the real world, users often modify them to suit their needs or improve the fit of the intervention with local needs (Rogers, 2005). Effectiveness trials have shown considerable variability in dosage and quality of implementation when programs are applied to real-world situations (Rohrbach et al., 2006). Consistent with these studies, we found variability in implementation fidelity among Project TND early adopters on the fidelity indicators that we assessed. For example, about one-fourth of organizations reported they implemented less than the complete set of program sessions, only 43% followed the recommended program implementation schedule, and only 44% obtained the certified training for program implementers, even though it is strongly recommended by the program developers. More than half of the early adopters made adaptations to the program by adding other curriculum materials. In the prevention field overall, little is known about how local adaptations of evidence-based programs affect program outcomes. The twin concerns of fidelity and acceptability lead to one of the conundrums facing the process of translating evidence-based programs to real-world settings: how to build in adaptability so that programs may be more acceptable, yet avoid jeopardizing program outcomes. Future research on Project TND and other evidence-based programs should examine the frequency and types of adaptations that are made and their relationship to program outcomes. In particular, as suggested by the findings of the current study, studies should address multiple dimensions of implementation fidelity including not only fidelity to the content of the program, but also fidelity to other elements of the program design, such as staffing, training of staff, setting, and target population.
One limitation of this study is that the findings are generalizable to only organizations like those that we were able to survey. It is likely that the organizations we were unable to reach (26% of program purchasers) had never implemented the program, or had tried it out and did not sustain implementation. Thus, despite achieving a good rate of response to the survey, the findings may indicate greater levels of program acceptance than would be shown if we had been able to reach all program purchasers. A second limitation is that we were not able to collect data on the program outcomes achieved by adopting organizations, which would have provided the opportunity to examine the impacts of Project TND when it is implemented in real-world, non-research settings. Third, the study did not assess many of the organization capacity factors that may be related to adoption and implementation of evidence-based programs, such as organizational work climate, decision-making structures, communication patterns, norms regarding change, and managerial support (Durlak & DuPre, 2008). In order to enhance implementation of evidence-based prevention programs, there is a need for more research on factors related to organizational capacity to deliver and support the programs, as well as factors related to the program provider (e.g., self-efficacy and skill proficiency), the program itself (e.g., compatibility and fit), and the community context in which it is delivered (e.g., politics and policy).
In conclusion, studies such as the present one have important implications for the development of strategies for widespread dissemination of evidence-based prevention programs. In order to increase the impact of effective substance abuse prevention programs, dissemination systems should aim to increase program reach into the target population, program adoption by target institutions or settings, consistent delivery of the program, and maintenance of the program effects over time (Glasgow et al., 2003). Research that draws upon a marketing perspective, examining factors such as who adopts programs, the processes of their decision-making about which interventions to use and how they will be implemented, and how evidence-based programs are received, contribute to the goal of accelerating program impact by addressing on the needs of the target populations as well as target institutions.
This study was funded by a grant from the National Institute on Drug Abuse (#R01-DA-16090).
1As of 2007, SAMSHA no longer distinguishes programs as “model,” “effective,” or “promising,” and instead simply provides scoring information from multiple dimensions for every program reviewed. (See the SAMHSA's National Registry of Evidence-based Programs and Practices, available at http://www.nrepp.samhsa.gov).