Challenges in Replicating Interventions

doi:10.1016/j.jadohealth.2006.09.005

J Adolesc Health. Author manuscript; available in PMC 2008 June 1.

Published in final edited form as:

J Adolesc Health. 2007 June; 40(6): 514–520.

Published online 2006 November 29. doi: 10.1016/j.jadohealth.2006.09.005

PMCID: PMC1976555

NIHMSID: NIHMS24170

Challenges in Replicating Interventions

Stephanie G Bell, MS., Susan F Newcomer, Ph.D, Christine Bachrach, Ph.D., Elaine Borawski, Ph.D., John L Jemmott, Ph.D., Diane Morrison, Ph.D., Bonita Stanton, M.D., Susan Tortolero, Ph.D., and Richard Zimmerman, Ph.D.

Stephanie G Bell, AHRQ;

Contributor Information.

Corresponding author: Susan F Newcomer, NICHD, Building 61E, room 8B7G, Bethesda MD 20892 301-435-6981 ; Email: newcomes/at/mail.nih.gov

The publisher's final edited version of this article is available at J Adolesc Health

See other articles in PMC that cite the published article.

Publisher's Disclaimer

Abstract

Purpose

This paper describes and reflects on an effort to document, through a set of six interventions, the process of adapting effective youth risk behavior interventions for new settings. It provides insights into how this might best be accomplished. It discusses six studies funded by NIH starting in 1999. The studies were funded in response to a Request for Applications [RFA] to replicate HIV prevention interventions for youth. Researchers were to select an HIV risk reduction intervention program shown to be effective in one adolescent population and to replicate it in a new community or different adolescent population. This was to be done while systematically documenting those processes and aspects of the intervention hypothesized to be critical to the development of community-based, culturally sensitive programs. The replication was to assess the variations necessary to gain cooperation, implement a locally feasible and meaningful intervention, and evaluate the outcomes in the new setting.

Methods

This paper lays out the rationale for this initiative and describes the goals and the approaches to adaptation of the funded researchers.

Results

The paper discusses issues relevant to all interventions, those unique to replication and to these replications in particular. It then reflects on the processes and the consequences of the adaptations. It does not address the further challenges in taking a successful intervention “to scale.”

Conclusions

Replications of effective interventions face all of the challenges of implementation design plus additional challenges of balancing fidelity to the original intervention and sensitivity to the needs of new populations.

Introduction

Interventions that are successful in promoting behavior change are those that are theory-based and specifically aimed at that behavior [1]. They are also hypothesized to be ″community-based,″ [2] and ″culturally sensitive,″ reflecting the needs, values, and cultures of the targeted individuals

Yet there is also the expectation that interventions should be replicable and able to be “scaled up” for much wider use [3,4). Issues relating to the tensions between specificity and generalizability seem to occur across a wide range of innovations [5]. ″Replication″ implies fidelity to the original, while ″community-based and culturally sensitive″ implies variation. Continuing creation of new approaches may be wasteful of scarce resources, but ‘cookbook” replications are also fraught with hazard. Therein is a challenge.

Despite consensus regarding the need for “core” elements in interventions (6), it is often unclear which elements [in design, methods of instruction, or curriculum content] are essential. A 2001 study examined a number of sexuality education interventions and found design [not curricular content] commonalities in the successful ones. Those included having a strong theoretical base, training the teachers well, delivering a clear message, providing accurate information, addressing social pressures, using teaching methods that involve students, and lasting a sufficient amount of time [7]. Yet one study of the adaptation process of Stanton’s Focus on Kids program found thirty versions of the original intervention, but few were evaluated. [8]. A nationally representative 1998 survey of the implementation of prevention programs in schools found that the quality of implementation is, in general, poor [9] and that the average prevention program included only 71% of recommended content and used half of the best-practices methods recommended.

Two more recent studies [10,11] examined curriculum content, and included several of the studies discussed in this paper Shortening programs, eliminating the condom demonstrations or implementing community-designed studies in school classrooms had a deleterious effect on program effectiveness. The common curricular contents of successful programs included: how to talk with parents, basic information about risk and avoidance of risk, personal and peer values about sexual behavior, perceptions of abstinence, condom use and sexual behaviors, and clear messages about values. Issues relating to assuring fidelity of implementation are also important in programs and in replications, though are not specifically addressed here. See [12].

Here we document and reflect on an effort to develop, through six separate intervention studies, experience with the process of adapting effective adolescent HIV prevention interventions into new settings, either of populations [e.g. age, ethnicity, race] or locations [e.g. urban, rural, school type, after school program]. The work began in December of 1998 when the NIH issued a Request for Applications [RFA] to replicate community based HIV interventions for youth. Researchers were to select an HIV intervention program that had been shown to be effective in one adolescent population and to replicate it in a new community or population in the U.S. This was to be done while documenting those processes and aspects of the intervention hypothesized to be critical to the implementation of community-based, culturally sensitive programs. The chosen intervention must have been reported or accepted for publication in a refereed journal, and have demonstrated statistically significant behavior change related to at least one of these risk factors: sexual initiation, numbers or types of partners, the frequency of sexual intercourse and/or the consistent and correct use of condoms.

Here we lay out the background for this initiative, and describe the goals and the approaches to adaptation of the grantees. We discuss issues relevant to all interventions, those unique to replication and to these replications in particular. We then reflect on the processes and the consequences of the adaptations.

The theoretical basis for the need to vary interventions to achieve cultural sensitivity and community acceptance is widely accepted, but has been systematically developed and articulated less frequently [13]. The formal theories most often used in the design of intervention programs usually contain a rationale as to why a given approach is expected to work but may not address issues relating to a particular cultural context. If the potential for behavior change is influenced not only by intra-individual psychological factors but also by characteristics of the social and cultural context, such theories may provide an incomplete basis for the design of effective interventions [14). Calls for such an approach are common as are those that call for tests outside of randomized controlled trials, more relevant to “real world” conditions. These calls continue. [15,16]. On the other hand, from a marketing perspective, there are some suggestions that culture-based arguments are effective in producing initial, cursory responses, but that such arguments fail to be persuasive when the individual has time to deliberate on a decision. [17]

The Replication Projects

The six studies selected by NIH peer review for funding all provided evidence in their applications of community acceptance and willingness to cooperate in the intervention selected by the Principal Investigator. Each study took a unique approach to the problems inherent in adaptation. Table 1 provides a quick reference point and thumbnail sketches of the six projects follow.

Table 1

Original Program Target and Replication Target Audiences

Borawski replicated Jemmott’s Be Proud! Be Responsible! (BPBR) among older adolescents in five urban and suburban high schools in the middle west. [18]. The primary results paper from this study is in preparation.

Jemmott replicated his Making a Difference abstinence-based curriculum and his Making Proud Choices safer sex curriculum, with 6^th and 7^th-grade African American and Latino youth. [19] The primary results paper from this study is in preparation.

Morrison replicated Focus on Kids, [20] in the ethnically mixed Seattle, Washington area. This study compared the intervention to an attention control group that focused on career exploration. The primary results paper from this intervention is in press [21]

Stanton replicated Focus on Kids. [20] Originally developed for inner city, African American youth, the program was replicated among white, rural youth. This study used a three-part trial comparing two approaches to intervention replication. The primary outcome papers from this study are [22,23].

Tortolero replicated Safer Choices, [24,25,26] a school-based intervention in alternative schools. The primary results paper is in preparation.

Zimmerman replicated Reducing the Risk, an intervention originally designed for pregnancy prevention [27] among adolescents in rural areas and small towns in an economically depressed Appalachian region. The primary results paper is [28}

General Issues in Program Implementation

Although this group of studies was initiated to address challenges relevant to program adaptation, each had also to address the common concerns of intervention programs. These issues have been extensively discussed (see, specifically, 6,10 and the papers from each investigator about their programs]. The issues varied across locations. Briefly, the components are: community support, program fidelity and staff selection and training.

A variety of different tools were used to establish community and school support for their programs. For example, Dr. Zimmerman’s team partnered with local health departments, which were already teaching in area high schools. . As required by state law, for one project, in the event of a positive STD finding, adolescents - and not parents -- should be notified. School board members felt very strongly that parents should be notified. While this was not legally possible, it took a considerable amount of effort to assure school board members that the research team would work with adolescents to ensure they had the needed treatment.

A second element, critical for evaluation accuracy, is monitoring to be sure that the program is implemented as designed (e.g., program fidelity]. Each project was able to do this. The projects utilized well-tested field methods in this process.

A third linked element is staff selection and training. Experienced facilitators may be accustomed to having a voice in the design and implementation of programs and to customizing the interventions based upon their understanding of the audience. Although the original programs replicated in this initiative all used facilitators hired by the research groups, some of the replication studies trained on-site teachers and nurses to deliver the curriculum.

Challenges in the Adaptation Process

There are excellent articles on the technology transfer processes necessary to get a proven product into the field. See the special issue of the Journal of AIDS Education and Prevention in 2000 [volume 12, supplement A]. However, those studies looked for “fit” between the original program and the new venue, rather than changing the intervention to match the community. One [6] did call for a careful examination of the “core elements” of a program when replicating it, though the author did not suggest what those core elements might be [for that, see 11]. There are many reasons to look for the “best fitting” intervention rather than working on adaptation but there are challenges in defining “fit.” [See 29l. Here we discuss the processes followed by these NIH funded researchers.

Geography and existing community structures

Adolescents’ lives differ depending on the type of area in which they live. Rural and suburban youth are much more geographically dispersed than are urban youth. This dispersion complicates efforts to conduct programs outside the regular school day. Without transportation, students may not be able to participate in after school or weekend programs when the venue is far from home. Some programs provided transportation for youth.

Geography can also affect adolescents’ social lives. In rural areas where there are no local parks or malls at which to hang out, much of the local activity occurs at school. References to the mall or to local hangouts may be alien to suburban or rural youth, and, references to football games to urban youth. This can have implications for program effects because less diffusion of new knowledge and values occurs when students do not interact outside the program setting.

Programs in new places may differ in their needs for timing, number, and length of program sessions. They need to fit in to ongoing organizational structures and schedules. One project developed a more intense program model to counteract frequent student absences and high drop-out rates. Modifying the curriculum to run in 45-minute sessions, twice weekly, for eight to ten weeks increased the likelihood of exposing students to the whole program. Another project created alternative length versions of the curriculum to accommodate the schedules of summer programs and other community activities. All attempted to maintain content coverage and ensure that all core program elements were covered consistently with dose equivalence.

Community Cultures and Support

Obtaining local support for a program is crucial to successful implementation, even when local stakeholders have provided theoretical support for a known program contingent upon funding, as happened with the Stanton program. While the purpose of the replication projects was to adapt interventions in culturally sensitive ways, the very fact of approaching a new community with an HIV prevention intervention and the concomitant funding set up the potential for “mismatch”. Several of these replication projects engaged their communities’ support without difficulty, while others did encounter problems. Three types of mismatch occurred.

First, not all communities were open to HIV prevention for youth, although all had agreed in advance to participate in the research. This was less the case in alternative schools, where administrators and teachers understood the need for HIV prevention programming among their students and welcomed the research team than in a suburban setting. Areas where schools and community agencies were not already open to and/or providing adolescent sexuality education posed a challenge for investigators, however. One project worked hard to engage their community, but was unable to utilize a number of program elements from the chosen intervention. [22]

A second issue was the openness of communities to the advice of “outside experts.” For several projects, the original program’s proven effectiveness was an important selling point. Many schools were happy to receive pre-developed, tested curriculum materials, and school administrators valued both the provision of HIV prevention programming for their students as well as training for their teachers and nurses. However, in some rural settings the fact that the intervention had been designed by “outsiders for outsiders” continued to be a significant deterrent to its utilization.

A third issue was existing or changing community priorities. In one location, the community was just not interested or excited about the program. They felt that other needs were more pressing and relevant to their youth, most specifically, issues of alcohol use.

Community Cultures and Program Content

Clearly, using family or communication examples that clash with cultural norms is counterproductive. Different norms exist around socializing, dating, and sexual behavior among different ethnic and cultural groups. Concepts and practices related to family relations, religion, and work also may differ across and within communities. These differences all invite some degree of program adaptation. Community and youth advisory boards can be essential in determining program areas where change may be necessary - as well as in assuring buy-in to the program. The projects took different approaches to adapting their programs, and the degree and levels of adaptation varied. Each approach is summarized below, and examples of the specific changes made are described.

Approaches

Dr. Jemmott’s team adapted his own curricula to Latino culture. They took into account how Latino culture supports HIV/STD protective and risk behavior. They sought to retain the core elements of the curricula and to incorporate Latino community and adolescent input throughout the process. This was accomplished through extensive review of existing research, preliminary studies of the relation of cultural values to sexual behavior, the use of focus groups with adolescents and parents of adolescents, pilot tests of the interventions, and debriefing of pilot intervention facilitators and adolescent participants.

To develop the cultural adaptation of her own program, Dr. Stanton’s team utilized community input. This team was attempting to contrast two versions of the replication [one of which was to remain “true” to the original intervention except for minor alterations and the other was a more open process to change. However, because many communities were not comfortable with some of the “standard” elements, a number of activities, particularly those dealing with condoms, were removed from the unchanged program, perhaps contributing to the weakened intervention effect in this new setting.

In adapting Safer Choices for an alternative school population, Dr. Tortolero’s team utilized intervention mapping, [30] which considers the theoretical foundations most likely to influence behavior change and the likely variation needed to address cultural or other relevant issues for the new population. They found that involving members of the original development team provided important insights on which elements of the program should be retained or revised for the new population.

In adapting the ETR-developed program to meet the needs of rural teens, Dr. Zimmerman and his team reviewed the literature and assessed each element of the curriculum. Those elements deemed essential included behavior-related knowledge, self-efficacy, personalization of risk and adverse effects, and peer norms. Input from the program designer was invaluable and helped the team remain focused on important content issues when developing new modes of presentation.

Dr. Morrison’s team was committed to keeping all elements of the Stanton program. They utilized focus groups, community input, and project staff experience to change only those elements that were inappropriate or unworkable. Feedback from the original project staff was valuable in supporting team decisions to retain elements intact and was supportive of most changes.

Finally, Dr. Borawski’s team decided prior to the study that they would not change the Jemmott curriculum in order to examine its transferability to students from different backgrounds. Because the community was then engaged in a heated debate on sex education many schools were reluctant to include one of the more extensive condom discussions in the curriculum. Removing this one activity allowed the team to retain all other activities involving discussion, demonstration and role playing with condoms.

Modifications in the Programs

There were a number of areas in which investigators made program changes to improve consonance with community cultures. Each program had unique needs, which included:

Preferred communication styles

One project removed labels from the source program which suggested that indirect ways of communicating were generally incorrect or inferior. For traditional Asian Americans and Asian immigrants, less direct styles of communication are the norm and appropriate.

Machismo

One project framed HIV risk and protective behavior through an initial activity that introduced participants to attitudes and behaviors that were consistent with the many positive aspects of Latino cultural values. For example, participants were introduced to the positive aspects of the cultural norm of machismo. Such behaviors included the responsibility to protect one’s family and partner. Thus, condom use was viewed as a behavior that was consistent with this dominant value. Throughout the curriculum, preventive behavior was framed in a manner that was consistent with these values.

Family Composition & Family Structure

One program modified a family tree activity, which was designed to get youth buy-in by creating a family system with which youth participants could identify. Their adaptation was based on two types of family systems, one in which parents were born in the United States and one in which parents were immigrants. Many details were then supplied by the group, which allowed this multi-cultural audience to invent a family reflective of their own community norms.

Community Culture and Program Implementation

Cultural differences among target populations also affected the strategies used in implementing programs. For example, one project team working in a multicultural setting learned that youth were not correctly describing the program to their parents, so they hired consent gatherers, fluent in the parents’ language, to explain the program. These staff members were also helpful in tracking down families for follow up screening.

In another project, research staff as well as facilitators and project assistants who worked on previous research studies with the investigative team were provided with an overview of cultural characteristics and norms for communication common among Latinos. In addition, it was necessary to ensure that key staff were bilingual. This was especially important in recruitment and retention follow-up efforts.

One project included African immigrant youth, a number of whom were Muslim. To include these teens, project staff needed to learn about cultural and religious practices of their families and incorporate their perspective into the intervention. This included adjusting the schedule to accommodate religious holidays and not serving meat (e.g., pepperoni pizza) as a snack.

Adapting Programs to Address Emergent Needs

Another challenge that led to changes in the original programs was the discovery of specific problems within the new target populations. In many cases, programs were adapted to address those needs. Examples include:

Addressing relationship power issues

In its needs assessment, one project learned that many girls dated older men and the age gap made it more difficult to negotiate sexual relationships and condom use; a high prevalence of dating violence existed; and the language their population used to describe sexual relations and the opposite sex underscored a level of disrespect and depersonalization associated with sexual relationships. To address these issues, new strategies were added on recognizing healthy and unhealthy relationships, and an additional lesson was included on avoiding or exiting high-risk situations.

Addressing future-orientation, self-efficacy, and attitudes

One project found a number of additional self-esteem related risk factors among their population, so they developed a video, “Making Choices” to address determinants such as self-efficacy, power in relationships, future orientation, perceived risk, and attitudes by using modeling and increasing the opportunity for vicarious learning experiences.

Addressing social/sexual experiences & HIV knowledge

One project’s baseline data suggested that its suburban population was less sexually experienced and knowledgeable than those in the original trial, although of the same chronological age. The investigators adapted the curricula by incorporating more basic information about puberty, adolescent development, and sexuality and by incorporating more scripted role-plays, which are more appropriate for a less sexually experienced population. Another population was much more experienced with a wider range of risk behaviors, than the original group, so the curriculum was broadened.

Narrowing or expanding the intervention focus

One project’s original intervention also focused on risks other than HIV, including substance use, drug sales, and gang involvement, but these were not major issues for the study participants. So those portions of the curriculum were removed. Another project faced the opposite issue. Their study population had many more risk factors than addressed in the original intervention and required additional lessons to meet their needs. In a third, the community did not feel that alcohol abuse, an issue of great concern, was adequately addressed in the source intervention, and that other sexual risks were over-emphasized. So the adapted intervention included alcohol use issues.

Conclusions

Many elements must come together for a prevention program to be replicated and to produce behavior change in a different environment. In replicating their programs, investigators encountered both structural and cultural challenges. Some changes made to the original programs were theory-driven, to bring the examples closer to home, such as in vignettes selecting names of youth from the local community or making references to single as well as two parent families. Some changes were driven by local constraints and realities, within the theoretical bounds of the intervention. Some were driven by the discovery of unmet needs in the target population. All investigators, even those who had originally developed the program being replicated, struggled with the process of adaptation. There is some suggestion in the literature on program innovation that “additions” to a program do not eviscerate its effectiveness, but that “adaptations” or “alterations” may [31].

When a curriculum has a number of elements, combined in a variety of ways, it becomes difficult to evaluate which elements are actually essential to the success of a program. The “core” curricular elements [e.g. a condom use demonstration] are not and perhaps could not be presented or evaluated as stand-alone interventions. Measurement of such elements in the absence of multiple trials with random assignment to varying conditions is close to impossible. Thus, the impact of the various specific changes made in these interventions may range from negligible to serious. However, only one project, so far, has found measurable behavioral change. That project, Dr. Zimmerman’s replication of Reducing the Risk, shared many of the same features as the other programs - community and facilitator support, adaptation to the study population, and measured fidelity to the core elements of the original program. Some of the projects are still analyzing their outcome data. Three of the programs [Stanton, Morrison, Borawski] have not found significant changes in behavior, although Stanton’s program did find positive changes in perceptions and intentions. Borawski’s and Morrison’s results have not been published, so these findings are as yet tentative.

There may be a number of issues beyond the effectiveness of the program that need to be considered. For example, moving a program that was successful in a population with very high rates of sexual initiation and that found a strong impact on condom use, [20] to another, Seattle, where sexual initiation rates were much lower, resulted in no measurable impact of the program on condom use. The proportion of youth who initiated sex in either intervention or control groups was simply too small for a statistically significant change. The program was replicated, but upon reflection the focus of the original intervention program had little program relevance in the new situation. The Tortolero program, which had to ramp up its focus on risk reduction, is another case where the original match was less than ideal, but changes were possible. This reflects back on the need to improve the match of program with audience. It would seem that cultural sensitivity may have some influence at the margins, but that intervention delivery, training of facilitators and inclusion of essential curricular components continues to be more powerful influences upon resulting behavior change.

In the best of situations, risk reduction interventions have small positive impact on sustained behavior change. [cf 32]. Research has shown fairly clearly the curricular and programmatic elements necessary for such change. It is not clear whether studies isolating single curricular elements would be able to determine the “essential” ones for such change, given the interactions of one with another and with other aspects of the target audience.

Furthermore, the larger, relatively immutable, social context influences individual behaviors. In the US, media and popular culture may play a role. [26, 33, 34]. An ecological perspective builds from the individual through social relationships to living conditions, neighborhoods, institutions and policies [35] Researchers have listed a range of these extra-individual factors relevant to HIV/AIDS, including but not limited to neighborhood deprivation, the acquisition of social capital, and peoples’ social and sexual networks [see 36,37]. This issue has been carefully discussed in the arena of early childcare [38]. One could argue for the need for HIV/AIDS researchers and program people to develop a broader perspective. We as health providers, educators and researchers must consider how reasonable is it to expect cognitive individual-level interventions to have large and lasting effects. Nonetheless it is essential to continue to work producing small but widespread change over time, as these can build to normative shifts. Consider the changes in seat belt use or smoking.

Acknowledgments

This revised manuscript, “Challenges in Replicating Interventions’” is being submitted to the Journal of Adolescent Health by Susan F Newcomer, NICHD Program Officer, on behalf of the six Principal Investigators who were awarded funding through an NIH-issued Request for Applications, and Stephanie Bell, who was a Presidential Management Scholar at NICHD when the paper was initiated. Ms. Bell wrote the first several drafts. Each investigator has reviewed and commented and added to this paper and all are aware of this submission

Each research grant was approved by the Institutional Review Board of the university to which the Principal Investigator belongs, as well as having passed the NIH peer review process. This manuscript was reviewed by each P.I. as well as by the Office of the Director of NICHD.

The studies upon which this paper is based were funded by two NIH institutes, NICHD and NIMH, through the following grants:

Borawski R01-HD038456

Jemmott R01-HD039109

Morrison R01-HD038420

Stanton R01-MH061761

Tortolero: R01-HD038457

Zimmerman R01-MH061187

Footnotes

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Stephanie G Bell, AHRQ.

Susan F Newcomer, NIH.

Christine Bachrach, NIH.

Elaine Borawski, Case Western Reserve University.

John L Jemmott, University of Pennsylvania.

Diane Morrison, University of Washington.

Bonita Stanton, Wayne State University.

Susan Tortolero, University of Texas Medical Center Houston.

Richard Zimmerman, University of Kentucky.

Bibliography

1. National Cancer Institute Theory at a Glance: A Guide for Health Promotion Practice. USDHHS/NIH; 2005. http://www.nci.nih.gov/cancer_information/cancer_literature/

2. Schwab M, Syme SL. On paradigms, community participation, and the future of public health. Am J Public Health. 1997;87:2049–2051. [PubMed]

3. DesJarlais DC, Padian NS, Winkelstein W. Targeted HIV-prevention programs. N Eng J Med. 1994;331:1451–1453.

4. Frost J, Forrest JD. Understanding the impact of teenage pregnancy prevention programs. Family Planning Perspectives. 1995;27(5):188–195. [PubMed]

5. Rogers Everett. Diffusion of Innovations 1995. 4th Edition Free Press; New York:

6. Kelly JA, Heckman TC, Stevenson LY, Williams PN, Ertl T, Hays RB, Leonard NR, O’Donnell L, Terry MA, Sogolow ED, Neumann MS. Transfer of Research-based HIV Prevention Interventions to Community Service Providers: Fidelity and Adaptation. Journal of AIDS Education and Prevention. 2000;12(Supplement A):87–98.

7. Kirby D No easy answers: Research findings on programs to reduce teen pregnancy 1997. The National Campaign to Reduce Teen Pregnancy; Washington DC: http://www.teenpregnancy.org/

8. Galbraith Jennifer An Examination of the re-Invention process of a health promotion program: The changes and evolution of “Focus on Kids” HIV prevention program 2004. Dissertation submitted in partial fulfillment of Ph.D. degree to University of MarylandCollege Park MD.

9. Gottfredson DC, Gottfredson GD. Quality of School-based Prevention Programs: Results from a National Survey. J Research on Crime and Delinquency. 2002;39(1):3–35.

10. Kirby D. Emerging answers: Research findings on programs to reduce teen pregnancy 2001. The National Campaign to Reduce Teen Pregnancy; Washington DC: http://www.teenpregnancy.org/

11. Kirby D, Laris BA, Lori Rolleri. Impact of Sex and HIV Education Programs on Sexual Behaviors of Youth in Developing and Less Developed Countries. Family Health International Research Triangle Park NC; 2005.

12. Ringwalt CL, Ennett ST, Johnson R, et al. Factors associated with fidelity to substance use prevention curriculum guides. Health Education and Behavior. 2003;30:375–391.

13. Federal Register. 2000 December 22;65(247):80865–80879.

14. Sonenstein FL. Using self-reports to measure program impact. Children and Youth Services Review. 1997;19(7):567–585.

15. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. AJPH. 2003;93(8):1261–1267.

16. Manhart LE, Holmes KK Population Level and multi-level interventions for preventing sexually transmitted infections: What has worked? Journal of Infectious Diseases. 2005;101:S7–S24. [PubMed]

17. Briley Dl, Aaker J When does Culture Matter? Effects of Personal Knowledge on the Correction of Culture-based JudgmentsForthcoming in Journal of Marketing Research . [currently Stanford Graduate School of Business Working Paper 1911]

18. Jemmott JB, Jemmott LS, Fong GT Abstinence and safer sex HIV risk-reduction interventions for African-American adolescents. JAMA. 1998;279:1529–1536. [PubMed] [the Be Proud Be Responsible curriculum]

19. Jemmott JB, Jemmott LS. Interventions for adolescents in community settings. In: DiClemente Ralph, Peterson JL., editors. Preventing AIDS: Theories and Methods of Behavioral Interventions 1994. Plenum Press; NY:

20. Stanton BF, Fang X, Li X A randomized, controlled effectiveness trial of an AIDS prevention program for low-income African-American youths. Archives of Pediatric and Adolescent Medicine. 1997;150:363–372. [The Focus on Kids curriculum

21. Morrison DM, Hoppe MJ, Wells EA, Gillmore MR, Beadnell BA, Wilsdon A, Higa D, Casey E Replicating a teen HIV/STD preventive intervention in a multi-cultural city. 2006. Unpublished manuscript.

22. Stanton BF, Guo J, Cottrell L, et al. The complex business of adapting effective interventions to new populations: an Urban to Rural Transfer. Journal of Adolescent Health. 2005;37:252–5. [PubMed]

23. Stanton B, Harris C, Cottrell L, Li X, Gibson C, Guo J, Pack R, Galbraith J, Pendleton S, Wu Y, Burns J, Cole M, Marshall S. Trial of an urban adolescent sexual risk reduction intervention for rural youth: a promising but imperfect fit. 2006;38:55.e25–55.e36.

24. Parcel GS, Luttman D Evaluation of a sex education course for young adolescents. Family Relations. 1981;30(1):55–60. [the Safer Choices Curriculum]

25. Basen-Engquist K, Parcel, et al. The Safer Choices Project: Methodological issues in school-based health promotion intervention research. Journal of School Health. 1997;67(9):365–371. [PubMed]

26. Coyle K, Basen-Engquist K, Kirby D, Parcel G, Banspach S, Collins J, et al. Safer Choices: Reducing teen pregnancy HIV and STDs. Public Health Reports. 2001;116:82–93. [PMC free article] [PubMed]

27. Kirby D, Barthe R, et al. Reducing the Risk: Impact of a new curriculum on sexual risk-taking. Family Planning Perspectives. 1991;23(6):253–263. [PubMed] [The Reducing the Risk Curriculum]

28. Zimmerman R, Cupp PK, Hansen GL, Donohew RL, Roberts AJ, Abner E, Dekhtyar O The Effects of a School-based HIV and Pregancy Prevention Program in Rural Kentucky. In press

29. Solomon J, Card JJ Understanding, Selecting, and Replicating Effective Teen Pregnancy Prevention Programs. National Campaign to Prevent Teen Pregnancy; 2004. www.teenpgrenancy.org/works/pdf/MakingTheList.pdf

30. Tortolero SR, Markham C, Parcel G, et al. Using intervention mapping to adapt an effective HIV, sexually transmitted disease and pregnancy prevention program for high-risk minority youth. Health Promotion Practice. 2005;6(3):286–298. [PubMed]

31. Mayer JP, Blakely CH, Davidson WS., II Social program innovation and dissemination: A study of organizational change processes. Policy Studies Review. 1986 November 6;(2):273–286.

32. Brückner H, Bearman P. After the promise: the STD consequences of adolescent virginity pledges. Journal of Adolescent Health. 2005;36:271–278. [PubMed]

33. Collins RL, Elliott ML, Berry SH, Kanouse DE, Kunkel D, Hunter SB, Miu A. 2004. Watching Sex on Television Predicts Adolescent Initiation of Sexual Behavior. Pediatrics. 2004;114(3):280–289.

34. Darroch JE, Frost JJ, Singh S, et al. Teenage Sexual and Reproductive Behavior in Developed Countries: Can More Progress Be Made November2001. The Alan Guttmacher Institute; Occasional Report 3.

35. Smedley BD, Symes SL, editors. Institute of Medicine: Promoting Health: Strategies from Social and Behavioral Research. National Academies Press; Washington: 2000.

36. Rhodes T, Singer M, Bourgois P, Freidman SR, Strathdee SA The Social Structural Production of HIV Risk among Injecting Drug Users. Social Science and Medicine. 2005 In press.

37. Noar S, Zimmerman RS Health behavior theory and cumulative knowledge regarding health behaviors: Are we moving in the right direction? Health Education Research(in press)1–12.12.

38. Yoshikawa H, Rosman EA, Hsueh JA. Resolving paradoxical criteria for the expansion and replication of early childhood care and education programs. Early Childhood Research Quarterly. 2002;17:3–27.