|Home | About | Journals | Submit | Contact Us | Français|
This paper presents the results of an effectiveness trial of Project Towards No Drug Abuse [TND], in which we compared program delivery by regular classroom teachers and program specialists within the same high schools. Within 18 schools that were randomly assigned to the program or control conditions, health classrooms were assigned to program delivery by teachers or (outside) specialists. Classroom sessions were observed by pairs of observers to assess three domains of implementation fidelity: adherence, classroom process, and perceived student acceptance of the program. Pre- and immediate posttest survey data were collected from 2331 students. Of the four composite indexes of implementation fidelity that were examined, only one (quality of delivery) showed a difference between specialists and teachers, with marginally higher ratings of specialists (p < .10). Both teachers and program specialists achieved effects on three of the five immediate outcome measures, including program-specific knowledge, addiction concern, and social self-control. Students’ posttest ratings of the program overall and the quality of program delivery failed to reveal differences between the teacher- and specialist-led classrooms. These results suggest that motivated, trained classroom teachers can implement evidence-based prevention programs with fidelity and achieve immediate effects.
There is now substantial empirical evidence indicating that a number of school-based programs are effective in preventing or reducing substance abuse among adolescents (Tobler & Stratton, 1997; Tobler, Roona, Ochshorn, Marshall, Streke, & Stackpole, 2000). The body of evidence for substance abuse prevention has been generated primarily through efficacy trials, which employ randomized controlled designs and other rigorous methods to determine whether specific approaches reduce substance use and abuse (Clarke, 1995; Flay, 1986; Green & Glasgow, 2006; Glasgow, Lichtenstein, & Marcus, 2003). Efficacy research, however, often has limited application to “real-world” settings (Glasgow et al., 2003). To determine how efficacious programs will work in typical school contexts, effectiveness trials must be conducted (Flay, 1986; Greenwald & Cullen, 1985).
Two research questions that are pertinent to effectiveness and dissemination studies are: (1) Who should deliver the prevention program? and (2) How much and what type of training is required for providers to implement the program successfully? In efficacy trials, researchers carefully select, train, and supervise program implementers in order to ensure standardization and high quality of program delivery (e.g., Hansen, Graham, Wolkenstein, & Rohrbach, 1991). Typically, the implementers are health education specialists, mental health specialists (e.g., counselors or clinicians), or certified teachers who are employed solely to deliver the program being evaluated rather than having multiple competing responsibilities (Green & Glasgow, 2006). In contrast, generally in effectiveness trials programs are delivered by classroom teachers or other appropriate school staff (e.g., nurses or counselors) who have many other responsibilities, and who volunteer or are selected by their school administrators to participate in the study (e.g., Basen-Enquist, O'Hara-Tompkins, Lovato, Lewis, Parcel, & Gingiss, 1994; Botvin, Baker, Dusenbury, Tortu, & Botvin, 1990; Rohrbach, Graham, & Hansen, 1993).
Another difference between efficacy trials and typical school contexts relates to the training provided to implementers prior to, and during program delivery. Typically, in efficacy trials the training approach is more extensive, including instruction on the theoretical rationale for the program, demonstration and practice of the skills required to teach the program, and ongoing coaching or technical assistance (Joyce & Showers, 1980; Tortu & Botvin, 1989). Sometimes, it also involves observation of program delivery by experienced specialists prior to implementation (e.g., Hansen et al., 1991; Rohrbach, English, Hansen, & Johnson, 2000). When prevention programs are transported to typical school settings, one barrier commonly encountered is that teachers have little time available for training (Rohrbach, D'Onofrio, Backer, & Montgomery, 1996). Usually, teacher training is limited to one- or two-day workshops that present an overview of the program theory, content, and teaching methods, but provide few or no opportunities for practice (e.g., Basen-Enquist et al., 1994; Perry, Murray, & Griffin, 1990; Rohrbach, Graham, & Hansen, 1993; Tortu & Botvin, 1989).
A critical issue in effectiveness research is the extent to which regular classroom teachers will implement evidence-based prevention programs with fidelity to the program design. The importance of program fidelity has been demonstrated (Dusenbury, Brannigan, Falco, & Hansen, 2003); low fidelity results in smaller or no program effects on behavioral outcomes (Botvin et al., 1990; Botvin, Baker, James-Ortiz, Botvin, & Kerner, 1992) and program mediators (Resnicow et al., 1998; Rohrbach et al., 1993). However, little is known about the most effective models for improving teachers’ implementation fidelity. The few studies that have addressed this question have suggested that workshops are no more effective than other teacher training modalities, such as videos, in increasing implementation fidelity (Botvin et al., 1990; Basen-Enquist et al., 1994).
As public polices such as the federal Safe and Drug Free Schools and Community Act (U.S. Department of Education, 2005) move us toward broad dissemination of evidence-based substance abuse prevention programs, there is a need for more research on all aspects of program implementation (Dusenbury et al., 2003; Domitrovich & Greenberg, 2000; Rohrbach, Grana, Sussman, & Valente, 2006; Schoenwald & Hoagwood, 2001). Specifically, studies that examine whether proven prevention programs are effective when implemented in “real world” settings by providers who differ from those in the efficacy trials are important for at least two reasons. First, there is a paucity of studies on this topic. We found only two studies that systematically manipulated the type of implementer of a school-based substance abuse prevention program (Cameron et al., 1999; McNeal, Hansen, Harrington, & Giles, 2004). The study by McNeal and colleagues showed larger effects for classroom teachers, relative to outside program specialists, on substance use outcomes and mediating variables (McNeal et al., 2004). Cameron and colleagues found that teachers’ and school nurses’ effects on smoking rates varied by school risk level, with no difference between provider types in high-risk schools and larger effects for nurses in low-risk schools (Cameron et al., 1999). Meta-analysis is another method that has been used to compare the relative effectiveness of different types of prevention program providers (e.g., teachers, peers, police, and mental health clinicians). Two studies have shown no differences by provider type in program effect size (Gottfredson & Wilson, 2003; Tobler & Stratton, 1997). However, the utility of meta-analytic comparisons is limited by confounding factors such as interactions between program modality and type of provider, and combinations of provider types in the same study (Gottfredson & Wilson, 2003). A second reason to conduct research on prevention program providers is that the findings of such studies may increase our understanding of why prevention programs succeed or fail when implemented on a large scale (e.g., St. Pierre, Osgood, Mincemoyer, Kaltreider, & Kauh, 2005).
The present paper describes the first effectiveness trial of Project Towards No Drug Abuse [TND], a high school-based program that has been shown to be effective in reducing adolescent substance use (Dent, Sussman, McCuller, & Stacy, 2001; Sun, Skara, Sun, Dent, & Sussman, 2006; Sussman, Dent, & Stacy, 2002). The aim of the study was to compare program delivery within the same schools by regular classroom teachers and (outside) program specialists.
Nine school districts from two counties in southern California participated in this study. Eligible districts contained at least one regular high school (RHS) and one continuation high school (CHS)1, each with an enrollment between 50 and 2000 students. We selected one RHS and one CHS from each district, yielding a total sample of 18 schools. Using a randomized blocking procedure, these schools were assigned to one of three conditions: cognitive-perception-information-only curriculum, combined cognitive-perception-information + behavioral skills curriculum (“combined”), or standard care (control), which resulted in a sample of six schools per condition. Schools were blocked by estimates of student drug use prevalence, ethnic composition, total enrollment, standardized achievement test scores, and school type (RHS or CHS). Nine RHS-CHS pairs were matched into three sets of three using a linear composite of factor scores (Graham, Flay, Johnson, Hansen, Grossman, & Sobel, 1984), and randomly assigned within matched sets to the three conditions.
Project staff collaborated with the administrator of each program school to select a health teacher who was willing to participate in training and implement the program, and a second teacher in whose classroom the program would be implemented by an outside program specialist. In each program school, eight health classes were randomly selected to receive the program (four for delivery by the health teacher and four by the program specialist) and complete the surveys. In each control school, four health classes were randomly selected for participation in the surveys only. Students in all experimental conditions were given a questionnaire assessment at pretest and immediately after the program implementation period (posttest).
Project TND is a 12-session high school-based substance abuse prevention program that incorporates motivation, skills and decision-making components (Sussman et al., 2004). The present study examined the efficacy of a new curriculum (cognitive-perception-information-only) relative to the complete (combined) curriculum that had been evaluated in the third Project TND experimental trial (Sussman et al., 2002). For a detailed description of the differences between the two curricula, see Skara, Rohrbach, Sun, and Sussman, 2005.
Training of program specialists involved several components. First, they observed delivery of each Project TND session by an experienced specialist in a typical high school classroom setting. Second, the specialist trainees practiced delivery of the program during pilot testing. Third, both the specialists and the classroom teachers recruited from participating schools attended a one and one-half day training session that reviewed the theoretical underpinnings and evidence base of the curriculum, provided detailed instruction on each lesson followed by practice with feedback, and included an overview of classroom control techniques. The classroom teachers participated in the workshop training session only.
At each program school, delivery of the curricula (both versions) took place over a four-week period. Dates of delivery (October through June) were balanced across conditions.
Each regular classroom teacher received $1000 compensation for his/her participation in the training and assistance in collecting parental consent forms.
Program implementers included 13 regular classroom teachers and five program specialists. The majority of implementers were female (72%), and 56% were white, 17 % were Hispanic, 17% were African American, and 10% were Asian. Their mean age was 44 years (SD = 10). Slightly more than one-fourth (28%) had a Masters degree. On average, the implementers had five years (SD = 5) of prevention program implementation experience.
All students in the randomly selected classrooms were given parental consent forms to take home for signature, indicating parental approval or refusal of student participation in the study surveys. Subjects also assented to their involvement. The homes of those students who failed to return a consent form were called by project staff to request verbal parental consent for survey participation.
In total, 3908 high school students were enrolled in the selected classes. Of these, 2735 students (70% of the enrollment roster) were recruited to participate in the study. Reasons for non-enrollment in the study included chronic absenteeism, either the parent or the student declined participation, or the student was absent on testing days (80%, 5%, and 15% of those not enrolled in the study, respectively).
Of the students who completed pretest surveys, 2331 (85.2% of those recruited) also completed immediate post-program surveys. The sample of 2331 students constitutes our analysis sample. Student subjects ranged in age from 13 to 19 years, with a mean age of 15 (SD = 1). The sample was 52% male; 18% white, 62% Hispanic, 8% Asian, 8% African American, and 3% other ethnicity. One-quarter of the students were enrolled in continuation high schools.
Our approach to assessing fidelity of implementation emphasized program process, or the way in which the program was delivered (Mowbray, Holter, Teague, & Bybee, 2003). Our goal was to observe each program implementer delivering the same two highly interactive curriculum sessions to two separate classes in each school. In total, 117 classroom sessions were observed, including 90 sessions at which two observers were present and 27 sessions at which one observer was present. Analyses of inter-rater reliability are based on the 90 sessions that had paired observers. All other analyses of the fidelity data are based on the total number of observations (n = 207).
The 12 observers were graduate students and project data collectors who had participated in two training sessions on the use of the observation instrument. The first training involved review of the measures, practice in the use of the items after watching selected videotaped segments of in-school prevention program delivery, and discussion of the practice ratings until at least 90% inter-rater agreement was obtained. To prevent observer “drift,” we conducted a second training that involved additional viewing of videotaped program delivery, practice ratings, and feedback.
The observation instrument assessed three domains of implementation fidelity that have been identified in the literature: adherence, classroom process (i.e., quality of delivery), and perceived student acceptance of the curriculum (Dane & Schneider, 1998; Dusenbury et al., 2003). The specific fidelity items were adapted from previous studies (Dent, Sussman, Hennesy, Craig, Moss, & Stacy, 1998; Hansen et al, 1991; Rohrbach et al., 1993; Sussman, Dent, Burton, Stacy, & Flay, 1995; Sussman et al., 1997). An important feature of the observation instrument is that it included rating scales that specified behaviorally anchored criteria for the end- and mid-points on the scales (Mowbray et al., 2003).
The fidelity items were combined to create four indices. For each index, the items were standardized (mean = 0; std = 1) and averaged. The adherence index assessed whether the lesson was shortened, and adherence to the manual when delivering the introductory and core activities (3 items; α = 0.69). The classroom process index assessed the extent to which the objectives of the lesson were met and the implementer elicited student responses during the program activities, and how well the session was implemented overall (3 items; α = 0.73). The quality of delivery index averaged implementer enthusiasm, implementer confidence, the extent to which he/she treated students respectfully, and the pace of the lesson (4 items; α = 0.66). The perceived student acceptance index assessed how interested the students appeared to be, how much they seemed to like the implementer, and class control (3 items; α = 0.76).
Student self-report surveys were administered over one class period at pretest and immediate posttest. At both waves, packets that included the survey and instructions for its completion were left for students who were absent on testing days. If absentee students failed to return the posttest survey, they were contacted for telephone-based survey administration. Students were told that they were not expected to complete the full questionnaire. They were asked to complete as many items as they were able to in the one class period. At five minutes before the end of the posttest period, students in the program conditions were instructed to stop completing any other item and begin completing the program evaluation items appended to the end of the survey.
Student questionnaire measures were adapted from previous studies of Project TND (Sussman et al., 2002). Demographic characteristics included age (in years), gender, and ethnicity (coded as white, Hispanic, Black, Asian, Native American, or other).
We examined four theoretical mediators of Project TND that have been associated with substance use in previous studies (Rohrbach, Sussman, Dent, & Sun, 2005; Sussman & Dent, 1996; Sussman, McCuller, & Dent, 2003). Substance use intentions is an index of four items assessing how likely the student is to use cigarettes, alcohol, marijuana, and hard drugs (cocaine, amphetamines, heroin, etc.), respectively, in the next 12 months (α = 0.81). Addiction concern is an index of three items (Sussman & Dent, 1996) that assess students’ perceptions of the likelihood that they will become a drug abuser, addict, and alcoholic, respectively (α = 0.79). Social self-control is an average of ten items that assess self- control skills in social situations (α = 0.73; Sussman et al., 2003). Program-specific knowledge was measured with 27 items designed to assess learning of the content of the two separate curricula. Items were scored as correct or incorrect, summed, and converted to a percentage correct score for analysis.
In the process evaluation section of the posttest, students rated the program on nine positive adjectives (e.g., “enjoyable,” “believable,” “interesting”), which were averaged to comprise the program acceptance index (α = .90). Using measures parallel to those in the observation instrument, students rated the implementer's confidence, ability to elicit student participation, respectfulness toward students, ability to understand students, and likeability during the program lessons. These five items were averaged to comprise the evaluation of implementer index (α = 0.93). The overall program rating index averaged how much students liked each of the 12 curriculum lessons (α = 0.94). The classroom climate index averaged five items derived from Moos’ Classroom Environment scale (Moos, 1979) that assessed perceived attention to the program lessons by other students (α = 0.85).
Inter-rater reliability was calculated for each implementation fidelity item. For the continuous items, we calculated the intra-class correlation, or one minus the proportion of total variance due to observers nested within observed classroom sessions (Shrout & Fleiss, 1979). For the dichotomous variable, a weighted kappa was calculated (Cohen, 1968).
To test for a main effect of implementer type on fidelity and immediate student outcomes, we used a mixed-linear model (Murray & Hannan, 1990) as implemented in the SAS statistical package® (SAS Institute, 2004). In each of the models, implementer type (regular classroom teacher or program specialist) was considered as a fixed effect and class nested within school was considered as a random factor. In models using observer ratings, observer (nested within session) was also considered as a random effect. Models that examined the effect of implementer type on immediate program outcomes were conditional, including the pretest score on the respective variable as a co-variate.
Table 1 presents the mean, standard deviation, and inter-rater reliability for each variable that comprised the fidelity indexes. Inter-rater reliability ranged from .60 to .89, which is considered good to excellent (Nunally, 1978; Tinsley & Weiss, 1975).
The results of the multi-level models that tested the effect of implementer type on program fidelity are shown in Table 2. The quality of delivery rating for program specialists was marginally significantly higher than that for classroom teachers (p = .08). On the other fidelity index variables, ratings of specialists and teachers were not significantly different.
Table 3 shows that there was a significant main effect for experimental condition (i.e., both program groups vs. controls) on program-specific knowledge and social self-control (p's < .05), and a marginally significant effect on addiction concern (p < .10). There was no main effect for two immediate outcome variables, intentions regarding future drug use and motivation to improve.
Post-hoc comparisons indicated there were no differences between program specialist-led and teacher-led classrooms in pre- to post-test changes in knowledge, self-control skills, and addiction concern.
Table 4 compares the mean ratings of students in the two program conditions on four process evaluation composite variables. Both groups rated the program and the implementers fairly highly. Students in the program specialist-led classrooms perceived a more positive classroom climate for the program relative to students in the teacher-led classrooms (p < .05).
The results of the present study indicate that both trained high school teachers and outside program specialists achieved effects on three of the five immediate outcomes of Project TND. Furthermore, classroom teachers implemented the program with levels of fidelity that were nearly as high as those demonstrated by the program specialists. Similarly, students in teacher-led and specialist-led classrooms provided comparable posttest ratings of the program overall and the quality of program implementation.
One of the strengths of this study is its design, which allowed for comparison of program delivery by two types of implementers while controlling for the potentially con founding effects of the school setting. Previous studies have not randomly assigned both implementer types within the same schools (e.g., Cameron et al., 1999; McNeal et al., 2004). The second strength of our study is the use of observer assessments of program fidelity. To date, only a few prevention studies have used observational methods to collect fidelity data (e.g., Botvin, Baker, Dusenbury, Botvin, & Diaz, 1995; Harrington, Giles, Hoyle, Feeney, & Yungbluth, 2001; Rohrbach et al., 1993), even though observations are thought to be more objective, valid, and reliable than implementer self-reports (Dane & Schneider, 1998; Dusenbury et al., 2003; Hansen & McNeal, 1999). Using multiple raters to assess key dimensions of fidelity, we demonstrated good-to-excellent inter-rater reliability for all variables.
The study focused on implementation fidelity, program-specific knowledge, drug use intentions and beliefs, and students’ responsiveness as immediate program outcomes. While the results suggest that classroom teachers can and will implement Project TND with fidelity and achieve effects on immediate program outcomes, it is important to demonstrate that teachers will achieve effects on student drug use. It was disappointing that neither program specialists nor teachers produced a main effect on drug use intentions at immediate posttest, and it is possible that neither program condition will demonstrate an effect on drug use at the one-year follow-up. At present we are collecting one-year follow-up data, and in future papers we will examine the effectiveness of the program overall, as well as the comparability of program effects achieved by the two types of program implementers.
One limitation of the study that should be noted is that the immediate outcome variables we examined are hypothesized to mediate program effects, but to date we have not conducted studies that identify specific mediators of Project TND. Of the immediate outcome variables we examined, addiction concern, self-control skills, and drug use intentions have been shown to predict adolescent substance use in previous studies (Rohrbach et al., 2005; Sussman & Dent, 1996; Sussman et al., 2003), suggesting that these may be signifi-cant mediators. In addition, student acceptance of prevention programs has been positively associated with implementation fidelity (e.g., Rohrbach et al., 1993). However, more research is needed to determine whether these immediate outcomes are mediators of Project TND. In our analyses of the one-year outcomes of the effectiveness trial, we will examine the relations of these items to the motivation-skills-decision-making program model (Sussman et al., 2004) and as mediators of changes in student substance use.
A second limitation of the study is that classroom teachers were selected by their school principal on the basis of their enthusiasm for the program and willingness to implement it. Further, teachers received generous compensation for participating in the training and completing self-report background surveys. These study conditions differ from the “real world” of prevention programming in schools, in that teachers rarely receive extra compensation for implementing prevention programs and they may be asked to implement a new program regardless of their level of enthusiasm for it (Rohrbach et al., 1996). The teachers in the present study may have been much more motivated than what one might expect to encounter in high school settings. Thus, the results of the study may be generalizable only to school settings in which substantial incentives for program implementation are provided. However, we view this study as fitting along a continuum of research on Project TND, from efficacy to dissemination. The study is what Flay (1986) has called a “treatment effectiveness” trial, in which the optimized delivery of evidence-based programming in real-world settings is carefully assessed. One might argue that the use of payments to teachers in a study of this type was justified to optimize program delivery. In the future, we plan to conduct “implementation effectiveness” and “dissemination” (Flay, 1986) studies of Project TND, which will carefully assess the relatively natural delivery of the program and the conditions that facilitate or impede its widespread use.
In conclusion, the study results indicate that there was little difference in fidelity and immediate program effectiveness between regular classroom teachers and outside specialists. The results suggest that motivated, trained classroom teachers can implement evidence-based prevention programs with fidelity and produce immediate effects. However, more randomized studies are needed to determine if proven prevention programs achieve longer-term program effects when implemented in typical school contexts by teachers, or other providers different from those in efficacy research.
This study was funded by grants from the National Institute on Drug Abuse (Grant numbers R01-DA13814 and R01-DA-16090). We wish to thank Jason McCuller for his project management.
1In California, alternative high schools are referred to as “continuation” high schools. Students in continuation high schools have transferred out of the regular school system due to functional problems (e.g., lack of credits, drug use). These youth are at risk for dropout, but they have transferred to an alternative school to fulfill a California mandate that all youth receive at least part-time education until they are 18 years of age.