The results of the present study indicate that both trained high school teachers and outside program specialists achieved effects on three of the five immediate outcomes of Project TND. Furthermore, classroom teachers implemented the program with levels of fidelity that were nearly as high as those demonstrated by the program specialists. Similarly, students in teacher-led and specialist-led classrooms provided comparable posttest ratings of the program overall and the quality of program implementation.
One of the strengths of this study is its design, which allowed for comparison of program delivery by two types of implementers while controlling for the potentially con founding effects of the school setting. Previous studies have not randomly assigned both implementer types within the same schools (e.g., Cameron et al., 1999
; McNeal et al., 2004
). The second strength of our study is the use of observer assessments of program fidelity. To date, only a few prevention studies have used observational methods to collect fidelity data (e.g., Botvin, Baker, Dusenbury, Botvin, & Diaz, 1995
; Harrington, Giles, Hoyle, Feeney, & Yungbluth, 2001
; Rohrbach et al., 1993
), even though observations are thought to be more objective, valid, and reliable than implementer self-reports (Dane & Schneider, 1998
; Dusenbury et al., 2003
; Hansen & McNeal, 1999
). Using multiple raters to assess key dimensions of fidelity, we demonstrated good-to-excellent inter-rater reliability for all variables.
The study focused on implementation fidelity, program-specific knowledge, drug use intentions and beliefs, and students’ responsiveness as immediate program outcomes. While the results suggest that classroom teachers can and will implement Project TND with fidelity and achieve effects on immediate program outcomes, it is important to demonstrate that teachers will achieve effects on student drug use. It was disappointing that neither program specialists nor teachers produced a main effect on drug use intentions at immediate posttest, and it is possible that neither program condition will demonstrate an effect on drug use at the one-year follow-up. At present we are collecting one-year follow-up data, and in future papers we will examine the effectiveness of the program overall, as well as the comparability of program effects achieved by the two types of program implementers.
One limitation of the study that should be noted is that the immediate outcome variables we examined are hypothesized to mediate program effects, but to date we have not conducted studies that identify specific mediators of Project TND. Of the immediate outcome variables we examined, addiction concern, self-control skills, and drug use intentions have been shown to predict adolescent substance use in previous studies (Rohrbach et al., 2005
; Sussman & Dent, 1996
; Sussman et al., 2003
), suggesting that these may be signifi-cant mediators. In addition, student acceptance of prevention programs has been positively associated with implementation fidelity (e.g., Rohrbach et al., 1993
). However, more research is needed to determine whether these immediate outcomes are mediators of Project TND. In our analyses of the one-year outcomes of the effectiveness trial, we will examine the relations of these items to the motivation-skills-decision-making program model (Sussman et al., 2004
) and as mediators of changes in student substance use.
A second limitation of the study is that classroom teachers were selected by their school principal on the basis of their enthusiasm for the program and willingness to implement it. Further, teachers received generous compensation for participating in the training and completing self-report background surveys. These study conditions differ from the “real world” of prevention programming in schools, in that teachers rarely receive extra compensation for implementing prevention programs and they may be asked to implement a new program regardless of their level of enthusiasm for it (Rohrbach et al., 1996
). The teachers in the present study may have been much more motivated than what one might expect to encounter in high school settings. Thus, the results of the study may be generalizable only to school settings in which substantial incentives for program implementation are provided. However, we view this study as fitting along a continuum of research on Project TND, from efficacy to dissemination. The study is what Flay (1986)
has called a “treatment effectiveness” trial, in which the optimized delivery of evidence-based programming in real-world settings is carefully assessed. One might argue that the use of payments to teachers in a study of this type was justified to optimize program delivery. In the future, we plan to conduct “implementation effectiveness” and “dissemination” (Flay, 1986
) studies of Project TND, which will carefully assess the relatively natural delivery of the program and the conditions that facilitate or impede its widespread use.
In conclusion, the study results indicate that there was little difference in fidelity and immediate program effectiveness between regular classroom teachers and outside specialists. The results suggest that motivated, trained classroom teachers can implement evidence-based prevention programs with fidelity and produce immediate effects. However, more randomized studies are needed to determine if proven prevention programs achieve longer-term program effects when implemented in typical school contexts by teachers, or other providers different from those in efficacy research.