PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of herLink to Publisher's site
 
Health Educ Res. 2009 December; 24(6): 941–948.
Published online 2009 October 6. doi:  10.1093/her/cyp053
PMCID: PMC2777948

Implementation fidelity of packaged teen smoking cessation treatments delivered in community-based settings

Abstract

Efficacious ‘packaged’ teen smoking cessation treatment programs, those developed by national organizations, are widely disseminated to local communities to help teens quit smoking. The implementation fidelity of these programs in community settings has not been documented. The efficacy of these programs could be lessened if they are not implemented as intended. Data from Helping Young Smokers Quit describe the frequency and types of modifications made to packaged teen cessation treatment programs for community delivery. A national sample of 591 community-based teen tobacco cessation treatment programs was profiled and 59% used a single packaged treatment program. Bivariate analyses found that 63% of program administrators reported implementing their program as planned; 37% modified their selected program. The most frequently reported modifications were made to the length and format of the program. Of those who modified their programs, >90% reported multiple program modifications (e.g. length and content). Administrators modified their programs to accommodate implementation barriers, such as time constraints and low participant enrollment, and to address the needs of participants with multiple risk behaviors that are co-morbid with tobacco use.

Introduction

Cigarette use among teens remains a public health concern. Based on 2006 national estimates, 2.6 million (10.4%) teens aged 12– 17 years old were current smokers [1]. Among high school seniors who reported smoking in the past month (21.6%), 56% were daily smokers and 27% smoked half of a pack or more every day [2]. Providing effective evidence-based smoking cessation treatments remains a priority for assisting teens in quitting smoking [3]. A growing number of packaged teen smoking cessation treatments (defined as those that are developed, evaluated and disseminated nationally by external organizations) are available to help teens quit smoking (e.g. [46]. Two packaged cessation programs, the American Lung Association's Not-On-Tobacco (N-O-T) program and Project EX, have been recommended as model programs by the Substance Abuse and Mental Health Services Administration for their effectiveness in short-term smoking cessation and cigarette smoking reduction among teens [7].

Recently, the Youth Tobacco Cessation Collaborative, a panel that seeks to understand the most effective cessation strategies to assist teens with quitting smoking, recommended that the implementation fidelity and delivery of teen smoking cessation treatments be examined [8, 9]. Though some packaged cessation programs demonstrated feasibility when examined under optimal conditions in evaluation studies [10], the efficacy of these programs could be diminished if they are not implemented as planned in community-based settings [11]. In these settings, packaged cessation programs are administered by non-research personnel and factors, such as participant recruitment, time constraints and cooperation from providers, may not be controlled for and may lead to significant changes in program delivery.

The ability to promote cessation and reduce smoking among teens, in part, depends on the implementation fidelity of efficacious packaged programs. Using data from the Helping Young Smokers Quit (HYSQ) initiative, this paper addresses three key questions regarding the implementation fidelity of packaged teen smoking cessation programs by community-based organizations: How often are packaged programs modified for community-based delivery? Are some programs modified more often than others? What types of modifications are made? Our analysis encompasses modifications in program length, format and content (cf. [11]. Tobacco use and smoking cessation outcomes were not assessed in these data; we were unable to determine the association between implementation fidelity and teen smoking outcomes. However, this paper contributes to the literature by expanding our knowledge about the delivery of efficacious treatment options for teen smoking in community-based settings.

Methods

Overview of HYSQ

HYSQ is a multiphase, national initiative that addresses the critical need to disseminate effective developmentally appropriate cessation programs for teen smokers. The first phase of HYSQ identified and surveyed a national sample of existing community-based youth cessation programs across the United States. Data for this paper come from that survey. Additional details about the design of the study and sampling procedures can be found in Curry et al. [3].

Sample

A two-staged sampling design was used to obtain the sample of existing youth smoking cessation programs across the United States. The first sampling frame consisted of 2453 US counties stratified by four criteria (urbanization, socioeconomic status, young adult smoking prevalence and state-level tobacco control expenditures). A sample of 408 counties was selected from the first sampling frame. The second sampling frame consisted of teen smoking cessation program administrators from the 408 counties who were identified by a snowball sampling procedure. Program administrators were those who were responsible for the overall supervision of the program but may or may not have implemented or provided the program to participants. A total of 1347 program administrators were identified [3].

Program eligibility

Among the 1347 program administrators, 1275 were reached by phone and screened to determine program eligibility for inclusion in the study. Eligible programs were defined as any teen cessation program not part of a research initiative that was established at least 6 months prior to the HYSQ evaluation and provided direct tobacco cessation services (e.g. in person) to persons aged 12–24 years. Based on these criteria, 756 programs from the 408 counties were eligible to be included in the study.

Survey procedures

Administrators of eligible programs were asked to participate in a 45-min telephone interview that included closed- and open-ended questions. The telephone interview was administered by the University of Illinois Survey Research Laboratory. Program administrators received a paper copy of the survey prior to the telephone interview. A copy of this survey can be obtained at http://helpingyoungsmokersquit.org/hysq_phase1.

Of the 756 eligible programs, administrators from 591 teen smoking cessation programs completed the survey (78.2%). Reasons for not completing the survey included: respondents were unavailable to complete the survey, respondents refused to take the survey and/or respondents could not be reached after multiple attempts. Of the 591 surveys, one was eliminated because of excessive missing data; the final number of completed surveys was 590.

Program inclusion criteria

Of the 590 eligible programs with completed surveys, 75 reported developing their own cessation treatments internally, 141 reported using both internally developed and packaged cessation treatments (referred to as combined treatment programs), 25 reported using materials from multiple packaged cessation treatments and 349 reported using a single packaged teen cessation treatment program. This paper reports on data from the 349 sites that used a single packaged cessation treatment program. We focus on these sites because we were interested in modifications to programs that were selected to be offered ‘as developed’. This approach allowed us to examine specific types of modifications made under conditions of planned adoption of existing treatment programs.

Measures

The telephone survey completed by program administrators consisted of questions about the general community context where the program operated, the setting of the organization sponsoring the program, the delivery of the program, and the content and evaluation of the program. No outcome measures (e.g. percentage of students who quit smoking) were included in this survey.

Program administrators were asked to specify the name of the packaged cessation treatment program used and the name of the external organization that developed the treatment program. Administrators also rated the importance of the following criteria in their decision to use the treatment program: cost, research evidence that the treatment program had worked, previous relationship, if any, with the organization that developed the treatment program, ease of adoption, recommendation to use the treatment program by other colleagues, convincing presentation from the developers or program marketers and recommendations from experts in youth tobacco cessation. Modification of the treatment program was assessed by first asking how close to the specifications of the external organization was it implemented. Respondents indicating anything other than very closely were asked a series of open- and closed-ended questions about how the treatment program was modified in terms of overall length of the program, length of the individual sessions, format and content.

Analysis

Responses were categorized to how closely the sites adhered to their implementation to the packaged treatment program specifications. Treatment programs for which respondents reported implementing ‘very closely’ to the specifications were classified as ‘not modified’; all other treatment programs were classified as ‘modified’. Bivariate comparisons were made between the modified and not modified treatment programs using chi-square and t-tests. Analyses were conducted using SAS version 9.1.

For purposes of this paper, we elected to include all packaged treatment programs in the sample for analysis. Though several different packaged treatment programs were used, prior analyses suggest that the format and core content of the programs used by the participants in our sample were quite homogenous [3].

Results

Overview of program characteristics

Consistent with previous reports [3], the majority of treatment was administered in group format in school-based settings. Programs averaged just over 2 months in duration and averaged nine contacts, with each contact averaging just over an hour and a quarter. The top three factors influencing the decision to select a packaged treatment program included: research evidence that the program had worked (reported by 73%), ease of adoption (reported by 61%) and recommendations from experts in youth tobacco cessation (reported by 52%).

Frequency and type of program modifications to single treatment programs

The Table I summarizes the frequency and type of program modifications for those sites using a single treatment program and sites using combined treatment programs. Overall, 63% of program administrators using single treatment programs reported adhering very closely to the program specifications and 37% modified their selected treatment program for community delivery. The most frequently reported modifications made by the 129 modified programs were to the length of the program and to the format of the program.

Table I.
Types and combinations of program modifications reported

Modifications to program length

Of the 129 modified programs in our sample, >50% either shortened the number of weeks that the program was offered or shortened the length of the individual program sessions. Sixty-four administrators offered comments about modifications to the program length. Among those, the most commonly reported reason for shortening the program was lack of time (reported by 12.5%). Some administrators specifically mentioned shortening their program fit into their allotted classroom time (reported by 37.5%). This sometimes resulted in program content modifications as administrators needed to reduce the amount of material presented to their participants. In other instances, programs maintained all the content. Example comments offered in the surveys were: ‘I broke it into smaller sections so that each section was shorter’ and ‘They are getting the same information in fewer sessions—a school driven request’.

Modifications to program format

Almost 45% of the administrators from the 129 modified programs made changes to the format of their program. Of those, 55 respondents offered comments about modifications to program format. The most frequently reported comments included: changing the modality of counseling sessions from group to individual counseling and changing gender composition of the group from single sex to coed groups (Table I). Though some changed the modality of the counseling sessions because of low participant enrollment (reported by 6.6%), the majority of those providing comments reported offering individual face-to-face counseling sessions instead of group-based counseling to tailor to the individual needs of their participants (reported by 20%). For example, an administrator said:

We did a lot of individual face-to-face [counseling], more than originally prescribed. Because participants often had a lot of things going on that weren't appropriate for a group setting.

Those that used a gender-specific teen smoking cessation program reportedly changed the composition of the groups from single sex to coed groups because of low participant enrollment. A respondent noted:

It's a gender-specific program. We've not had enough [participant] response to justify having two different groups. There hasn't been enough interest to have staff officers, one for girls and one for boys.

Modifications to program content

Changes in program length and format sometimes resulted in modifications to program content (e.g. selecting portions of the program, providing supplementary materials). However, modifications to the content of the treatment program were the least frequently reported by administrators. Of those administrators who modified their treatment program's content, 54 responded to open-ended questions about these modifications. The most frequently reported content modification was the inclusion of additional of materials or topics to enhance the program (Table I). Comments included:

Sometimes we used different handouts, but with the same information presented in a more eye catching way; [we] often try to incorporate other media rather than just video or games to add more variety to messages.

Some of the content from the other smoking cessation program—added that to our program. Added an ‘Up in smoke’ video from the Tobacco Education Group/Tobacco Awareness Program (TEG/TAP) and from the Minnesota program—added a session on media's influence.

Over a third of administrators also reported eliminating content from the program. Content was eliminated for various reasons, including time constraints and the material was perceived as not applicable, unimportant or redundant for teens.

Nearly, a quarter of administrators who provided open-ended responses reported modifying the content of the program to address other adolescent-specific issues beyond tobacco use. For example, respondents said:

Generally [we] moved from cessation (because the [participants] aren't quitting) to substance abuse, conflict resolution, rather than giving ways to keep quit.

Most of the kids that smoke cigarettes also smoke marijuana. Program addressed other issues including marijuana and family programs.

Modifications in combination

As the data in the table illustrate, once a program was modified, it was usually modified in more than one way, with 93% of modified programs reporting modifications in combination. Over a third of the sample modified both the length and the format of their selected programs. Approximately, a quarter of the sample modified the length and content of the program. A minority of modified programs had changes in length, format and content combined.

Modifications to the American Lung Association N-O-T program

When asked to name the specific packaged program they used, just over half of the administrators using single treatment programs (51.3%) reported using the American Lung Association's Not-On-Tobacco (N-O-T) program. The two other packaged programs named most often were the Tobacco Education Group/Tobacco Awareness Program (TEG/TAP, 16.3%) and the ‘American Cancer Society’ program (7.4%). A total of 87 other individual programs, mostly consisting of local or state-based initiatives, comprised the remaining 25%. Since the N-O-T program was the most often used packaged treatment program, we examined whether it was modified more or less often than other packaged programs. Overall, modifications were reported for 30% of the N-O-T program compared with 44% of other programs (P < 0.01). Thus, the N-O-T program was modified less often than other programs. When the program was modified, there were no significant differences in the types of changes reported for the N-O-T program compared with other programs.

Frequency of program modifications including combined treatment programs

If the 141 combined treatment programs are included as ‘modified’ programs, then the percentage of modified programs increases to 55% (270/490). A post hoc analysis of these 141 responses was conducted to characterize the modifications made to combined treatment programs. We did not include responses for the 25 administrators that used multiple packaged programs in our analysis because the data indicated that they were conceptually different from those that used combined treatment programs. Data suggest that although these administrators used multiple packaged treatment programs, they were more likely to follow the specifications from each treatment than those who used combined treatment programs.

Detailed information on the nature of program modifications was only obtained from administrators who described these programs as modified (n = 91). Among those 91 programs, the most frequently reported modifications were made to the length and content of the program. Additionally, combined treatment programs were more likely to report combinations of modifications compared with those using single programs (P < 0.001).

Discussion

The effectiveness of evidence-based teen smoking cessation treatment programs implemented in community-based settings may require fidelity to the specifications of the program developer. Though some treatment programs have demonstrated efficacy and feasibility when evaluated under ‘ideal’ conditions, less attention has been paid to the delivery of these programs in community settings, such as classrooms and schools [12]. This paper describes the fidelity of implementation of packaged teen smoking cessation treatment programs as reported by administrators in a national sample of community-based organizations not affiliated with research programs. Our study found that while the majority of the sites using single treatment programs reported closely following the specifications of the program developers, 37% modified their program for community delivery, indicating that treatment as designed is not always treatment as delivered. Over half of packaged programs were modified if the 141 sites that used the combined treatment programs (characterized as using both internally developed materials and single treatment programs) are considered.

Factors that are commonly controlled for in research, such as time constraints and low participant enrollment, often become barriers for non-research personnel implementing packaged programs in community settings. Modifications reported in our survey appeared to be in response to these barriers. For instance, administrators that faced competing time demands from their schools reduced the length of their program. Some programs changed from group to individualized counseling to meet the specific needs of their participants or because of low enrollment. The content of some programs was altered either by eliminating topics that were not perceived as relevant or by covering additional topics such as marijuana or alcohol use. Modifications such as these may address the needs of teens who were dealing with adolescent problem behaviors that co-occur with tobacco use. Notably of those administrators who changed their program, nearly all reported multiple modifications (e.g. length and content) to their program, suggesting the presence of several barriers and the necessity to adapt program delivery in an attempt to meet their local needs.

Despite these reported adaptations, the modified treatment programs in our sample met some of the current standards for teen cessation programs. Over a third of the modified treatment programs were ≥6 weeks in duration and the majority of programs were offered in school-based settings. These components are consistent with current evidence-based recommendations that cessation treatment programming should consist of at least five sessions and be delivered in a school-based context [13]. Our findings also suggest that nationally recommended evidence-based packaged programs are adapted less often for community delivery. The American Lung Association's N-O-T program was the most often used packaged treatment program in our sample. The program has had significant impact on both quitting and smoking reduction among teens [4, 10, 14] and is described as a ‘user-friendly’ program in school settings [15]. Our survey found that the N-O-T program was modified less often than other packaged programs. Horn and Dino [16] state in a publication addressing procedures for ensuring implementation fidelity in the N-O-T program that ‘…to increase the adherence and fidelity to the program, all N-O-T facilitators must be trained and certified to use the program’. Perhaps, extensive training accounts for fewer modifications made to the N-O-T program.

Facilitator training has been associated with implementation fidelity in tobacco and other drug use prevention studies [17, 18] and should perhaps be adopted by other program developers to ensure that packaged teen smoking cessation programs are implemented with fidelity. As noted by Rohrbach et al. [19], facilitator training should include ‘instruction on the theoretical rationale for the program, demonstration and practice of the skills required to teach the program, and ongoing coaching or technical assistance’. Other recommendations to establish implementation fidelity include offering facilitators technical assistance during program delivery. Related program developers may wish to incorporate assessments of fidelity to encourage facilitators to gauge the degree that program materials are implemented based on the training they received. Program developers should also consider collaborating and consulting with existing on-the-ground program facilitators to understand their experiences in implementing smoking cessation packaged programs with teen participants. This may aid in identifying areas where the program can be adapted to address the participants’ and facilitators’ needs while still maintaining program efficacy.

While maintaining program efficacy is certainly important, emphasis should also be placed on assessing program effectiveness. Effectiveness evaluations are needed to understand the generalizability of teen smoking cessation treatments. Findings from these studies can tell us if the treatments can be delivered as planned and yield expected results when they are implemented in a variety of settings where barriers, such as time constraints and low participant enrollment, are present. These studies can also inform us about the timing of dissemination of teen smoking cessation treatments by highlighting what modifications may be required to deliver the program in real-world settings before it is widely disseminated [12]. Efficacy and effectiveness evaluations have been conducted for the N-O-T program [10] and efficacy evaluations for Project EX-4 (a classroom-based derivative of the Project EX program) [20]. Treatment programs developed to address teen tobacco use are continuously emerging. As these programs are found to be efficacious, effectiveness evaluations should be encouraged before they are widely disseminated.

Although the current study extends our knowledge of implementation fidelity of packaged teen smoking cessation programs, there are limitations. The data in our study are self-reported from program administrators. We were unable to independently verify the accuracy of their reports. Some evidence suggests that non-research personnel may not have sufficient understanding of the program to accurately respond to implementation fidelity assessments and are biased in their reports [21, 22]. Future studies may wish to use additional data collection methods, such as direct behavioral observations, to verify information provided from program personnel. No outcome measures were included in our data; hence, we were unable to examine the association between fidelity of implementation and teen smoking cessation outcomes.

In sum, the findings from this study suggest that a sizable percentage of packaged teen smoking cessation programs are modified for community-based delivery. Program administrators who modified their packaged programs appeared to do so to address their local needs, as well as the needs of their participants. An important question that stems from our findings is, ‘what level of program adaptation, if any, is acceptable?’ Deviations from the recommended programming could impact the nature of the intervention and have been shown to adversely affect outcome in some drug abuse prevention studies [11]. While collection of treatment outcome data was beyond the scope of this survey, further studies that are designed to evaluate the effect of implementation fidelity on smoking and cessation outcomes for teens are needed. These studies should also be designed to examine the association between modifications made to the program and teen smoking cessation outcomes to understand what level of program modification results in loss of effectiveness or what level of adherence is needed for optimal outcome.

Funding

Robert Wood Johnson Foundation (04-6472); National Cancer Institute and the Centers for Disease Control and Prevention through the US Department of Health and Human Services, National Institutes of Health, National Cancer Institute (contract 233-02-0087); National Cancer Institute (2R25CA057699).

Conflict of interest statement

None declared.

References

1. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. Results from the 2006 National Survey on Drug Use and Health: National Findings. Available at: http://www.oas.samhsa.gov/NSDUH/2K6NSDUH/2K6results.cfm. Accessed: 21 July 2009.
2. Johnston L, O'Malley P, Bachman J, et al. 2007. Data: Trends in 645 Prevalence of Use of Cigarettes for Eighth, Tenth, and Twelfth Graders (Online) Available at: http://www.monitoringthefuture.org/data/07data.html#2007data-cigs. Accessed: 2 April 2008.
3. Curry S, Emery S, Sporer A, et al. A national survey of youth tobacco cessation programs for youth. Am J Public Health. 2007;97:171–7. [PubMed]
4. Dino GA, Horn KA, Goldcamp J, et al. Statewide demonstration of not on tobacco: a gender-sensitive teen smoking cessation program. J Sch Nurs. 2001;17:90–7. [PubMed]
5. Sussman S, Dent CW, Lichtman KL. Project EX: outcomes of a teen smoking cessation program. Addict Behav. 2001;26:425–38. [PubMed]
6. Coleman-Wallace D, Lee J, Monotgomery S, et al. Evaluation of developmentally appropriate programs for adolescent tobacco cessation. J Sch Health. 1999;69:314–9. [PubMed]
7. Substance Abuse and Mental Health Services Administration. Effective Substance Abuse and Mental Health Programs. Available at: http://modelprograms.samhsa.gov/. Accessed: 8 May 2008.
8. Maule CO, Moyer CA, Lovato CY. Application of a better practices framework to review youth tobacco use cessation. Am J Health Behav. 2003;27(Suppl. 2):S132–43. [PubMed]
9. Backinger CL, McDonald P, Ossip-Klein DJ, et al. Improving the future of youth smoking cessation. Am J Health Behav. 2003;27(Suppl. 2):S170–84. [PubMed]
10. Horn K, Dino G, Goldcamp J, et al. The impact of Not On Tobacco on teen smoking cessation: end-of-program evaluation results. J Adolesc Res. 2005;20:640–61.
11. Dusenbury L, Branningan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse prevention in school-based settings. Health Educ. 2003;18:237–56. [PubMed]
12. Schoenwald SK, Hoagwood K. Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatric Services. 2001;52:1190–7. [PubMed]
13. Sussman S, Sun P, Dent C. A meta-analysis of teen cigarette smoking cessation. Health Psychol. 2006;25:549–57. [PubMed]
14. Dino G, Horn K, Meit H. A pilot study of Not On Tobacco: a stop smoking programme for adolescents. Health Educ. 1998;6:230–41.
15. Horn K, Dino G, Gao X, et al. Feasibility evaluation of Not On Tobacco: the American Lung Association's new stop smoking programme for adolescents. Health Educ. 1999;5:190–206.
16. Horn K, Dino G. A comparative review of the effectiveness trial of N-O-T in Alabama: guideposts for future research, 1998–2003. Am J Health Educ. 2009;10:47–55.
17. Basen-Engquist K, O'Hara-Tompkins N, Lovato C, et al. The effect of two types of teacher training on implementation of Smart Choices: a tobacco prevention curriculum. J Sch Health. 1994;64:57–66. [PubMed]
18. Botvin G, Baker E, Dusenbury L, et al. Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: results of a 3-year study. J Consult Clin Psychol. 1990;58:437–46. [PubMed]
19. Rohrbach L, Graham J, Hansen W. Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993;22:237–60. [PubMed]
20. Sussman S, Miyano J, Rohrbach L, et al. Six-month and one-year effects of Project EX-4: a classroom-based smoking prevention and cessation intervention program. Addict Behav. 2007;32:3005–14. [PubMed]
21. Skara S, Rohrbach L, Sun P, et al. An evaluation of the fidelity of implementation of a school-based drug abuse prevention program: project Toward No Drug Abuse (TND) J Drug Educ. 2005;35:305–29. [PubMed]
22. Hansen WB, McNeal RB., Jr Drug education practice: results of an observational study. Health Educ Res. 1999;14:85–97. [PubMed]

Articles from Health Education Research are provided here courtesy of Oxford University Press