PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
School Ment Health. Author manuscript; available in PMC 2010 May 4.
Published in final edited form as:
School Ment Health. 2009 September 1; 1(3): 103–106.
doi:  10.1007/s12310-009-9013-x
PMCID: PMC2863023
NIHMSID: NIHMS193688

Intervention Integrity: New Paradigms and Applications

This special series seeks to summarize the status of integrity monitoring in school-based intervention research and practice. The four articles included in this issue highlight several innovative research projects to design, implement, and evaluate integrity across diverse settings, and outline future research directions and practical implications for school-based professionals. Although the concept of understanding the degree to which an intervention has been implemented as planned has long been recognized as critical within the prevention and intervention literature (Basch, Sliepcevich, Gold, Duncan & Kolbe, 1985), research and conceptual models related to treatment integrity have surprisingly lagged behind. For instance, a series of recent reviews of the literature suggests that treatment integrity is only recorded in a small minority of studies, ranging from 3.5% to 28% of studies reviewed (e.g., Gresham, MacMillan, Beebe-Frankenberger, & Bocian, 2000; McIntyre, Gresham, DiGennaro, & Reed, 2007; Perepletchikova, Treat, & Kazdin, 2007).

Intervention integrity is an important construct to understand and evaluate in the school-based mental health literature given its possible positive association with intervention success (e.g., Gresham, 1989; Peterson, Homer, & Wonderlich, 1982). Thus, effective programs implemented with a high degree of fidelity are expected to produce the most consistent and positive effects. Further, systematically monitoring the integrity with which treatments are implemented provides insight into interventions that do not produce positive effects. Specifically, intervention integrity data can determine if a program was ineffective because it was not a strong program (e.g., poorly conceptualized/designed) or because it was implemented poorly (e.g., the main components of the program were not implemented as intended). Monitoring how programs are carried out can also provide information on components that are critical to intervention success (Perepletchikova, & Kazdin, 2005) and the feasibility of implementing the intervention (Peterson & McConnell, 1993). Finally, given the recent reauthorization of the Individuals with Disabilities Education Improvement Act in 2004, increasingly school staff are being called upon to utilize a response to intervention (RTI) framework in identifying children with learning disabilities (Klotz, & Canter, 2006). Within the RTI movement, there is a clear mandate that the school provide continuous monitoring of evidence-based interventions to determine treatment adherence and success for individual students. For all of these reasons, it is critical to develop comprehensive, practical, and community-responsive treatment integrity procedures within the school-based mental health literature base.

Historical Context of Treatment Integrity

Over the last 25 years, clinical research trials have emphasized the importance of using treatment manuals, and implementing consistent practices, trainings, and supervision (e.g., Drozd, & Goldfried, 1996; Leff, Power, Manz, Costigan, & Nabors, 2001; Luborsky, & DeRubeis, 1984). Traditionally, treatment integrity or fidelity has referred to therapist adherence to the intervention manual (Gresham, Gansle, Noell, Cohen, et al., 1993). In other words, has a therapist or program implementer covered the main or crucial steps stipulated for a particular intervention session? This is often referred to as procedural or content integrity. While procedural integrity is an important part of integrity monitoring, this current special series draws upon recent research suggesting that there are a number of other important ways in which to conceptualize integrity that expand upon this definition and provide future directions to the field of prevention and intervention science. For one, intervention integrity has expanded to include measurement of therapist ability to effectively carry out the treatment, i.e., therapist competence. Thus, the extent to which the therapist effectively manages child behavior, responds to participant questions, and encourages active involvement is considered critical to intervention success. Intervention integrity has also been expanded to address the issue of treatment differentiation, such that evaluating the extent to which different program components are implemented allows researchers to compare the relative impact of different treatment components on participant outcome. Finally, recent literature on treatment fidelity includes participant response to the intervention, suggesting that participants who enjoy and are engaged with the intervention are more likely to experience a positive impact (Dane, & Schneider, 1998).

Controversies and Conceptual Models

As a field, there are a number of areas in which research has helped to further our understanding and appreciation of integrity monitoring over the past few decades. These areas include (a) the recognition that there must be “flexibility” within the delivery of manualized treatments (Kendell, & Beidas, 2007); (b) that models for treatment integrity must include not just procedural or content integrity, but also indices for determining the quality of treatment delivery and measures of participant engagement (Perepletchikova, & Kazdin, 2005); and (c) that we consider partnership-based approaches to help ensure the cultural sensitivity and participant engagement aspects of the interventions (Power et al., 2005).

Flexibility in Implementation Procedures

Over the last 10 years, the core ingredients for determining whether a program is efficacious (proven) or possibly efficacious (promising) have been clearly articulated (Chambless, & Hollen, 1998). Programs are designated as efficacious if they employ an experimental group design (e.g., random assignment procedures), have well-documented and clearly delineated treatment procedures including treatment manuals, contain uniform therapist training procedures, employ multi-method outcome assessments, have demonstrated longitudinal effects, and have an independent replication by another team of researchers. If all but the independent replication has occurred, then the program is possibly efficacious. In recent years, there has also been growing recognition that treatment adherence or procedural integrity is also critical to identifying successful programs. Specifically, providers can enhance program development by delineating and measuring the core or critical components of each intervention session. Parallel with this is the recognition that program facilitators can effectively develop their own “voice” or style, and that these features can be understood and monitored in terms of quality indicators (e.g., Kendall, & Beidas, 2007).

Multi-Component, Multi-Method Indices of Intervention Integrity

As the definition of intervention integrity has expanded, the need for utilizing multiple methods and multiple reporters has been highlighted. For example, integrity monitoring systems should include both participant and therapist reports on key program processes. Including participant report will not only serve as a check on the reliability of therapist perceptions, but will also provide unique insight into critical intrapsychic factors, such as participant engagement. In addition, for interventions with more than one participant, integrity monitoring systems should include both individual and group-level indices. Doing so will allow for a more nuanced assessment of the mediating effect of program integrity (e.g., evaluation of differential intervention impact for participants with high vs low engagement). Finally, diversity in evaluation methodology will also provide a more accurate understanding of key intervention components. As such, intervention integrity should be monitored using multiple methods, including self-report, observer-report, direct observations, and permanent products.

In their article entitled, “An analysis of teacher investment in the context of a family-school intervention for children with ADHD,” Power, Soffer, Mautone, Jones, and Clarke, (2009) described the reliability and validity of the Teacher Investment Questionnaire (TIQ), an instrument designed to assess participant (i.e., teacher) investment in intervention delivery. This measure contributes to the knowledge base related to intervention integrity in that it was designed to assess the important yet relatively under-developed construct of “participant engagement” in intervention delivery.

Partnership-Based Approaches to Integrity Monitoring

Traditionally, the development of assessment tools places the emphasis upon the research team to identify critical constructs, develop and fine-tune items or subscales, and to establish the psychometric properties of the tools generated. These approaches are generally top-down and are directed by researchers without the involvement of research participants (see Nastasi et al., 2000; Power et al., 2005). While these certainly are strengths to these traditional approaches, there also are some limitations which include that the resultant measures may not be reflective of the social context and/or may not be meaningful for the local community (Leff et al., 2006). Recently, Power et al. (2005) extended the notion of using partnership-based approaches to the development and validation of intervention implementation procedures. This allows for a much better understanding of what aspects of treatment are applied and effective, how participant engagement impacts treatment delivery, and a recognition that deviations from treatment protocols provide an ample opportunity to better understand treatment feasibility.

In their article entitled, “Using participatory action research to design an intervention integrity system in the urban schools,” Gullan, Feinberg, Freedman, Jawad, and Leff, (2009) described partnership-based procedures used to develop and refine an integrity monitoring system designed to evaluate multiple dimensions of implementation integrity, namely, the extent and quality of intervention delivery and participant responsiveness. This research is one of the few studies to examine the effect of implementation fidelity on participant outcomes, and it may be the only study to have so extensively explored the mediating effect of integrity measures on intervention effects.

Supports for Monitoring Integrity

Monitoring intervention integrity is a best practice and in certain instances a mandated requirement (e.g., Iowa Department of Education, 2007). Implementing interventions with (e.g., Noell et al., 2005) and collecting implementation integrity data consistently (e.g., Burns, Peters, & Noell, 2008; Cochrane, & Laux, 2008), however, can be challenging. For example, efforts to increase the acceptability and feasibility of interventions and integrity measurement in time- and resource-limited school settings are often at odds with proper monitoring of intervention implementation (Evans, Schultz, & Serpell, 2008). Two articles in this special issue describe supports for families to facilitate the implementation of interventions (Swanger-Gagne, Garbacz, & Sheridan, 2009) and supports for school staff to monitor implementation integrity (Brown, & Rahn-Blakeslee, 2009).

In their study entitled, “Intervention implementation integrity within conjoint behavioral consultation: Strategies for working with families,” Swanger-Gagne et al. (2009) described procedures that consultants used to support parents’ implementation of home-based interventions in the context of a conjoint behavioral consultation model. Those authors presented data from a large study that described parents’ implementation integrity assessed as per two methodologies: self-report and permanent products. They also presented a case study that described in rich detail the types of supports one family in the “at-risk” group received. This study extends the literature by initiating a line of research assessing the integrity of interventions being implemented within the home setting with a particular focus on families whose life circumstances may make it particularly difficult to implement interventions consistently.

In their article entitled, “Training school-based practitioners to collect intervention integrity data,” Brown and Rahn-Blakeslee (2009) described the Heartland Area Education Agency’s model of assessing intervention integrity, which includes providing professional development and supports to help school staff monitor intervention integrity and use these data when making educational decisions in an RTI framework. The article provides specific, practical information about the logistics and content of staff training and monitoring tools that can be broadly applied in school settings.

Summary

While the field of intervention and prevention science has in recent years seen an increase in the recognition and importance of providing empirically supported and well-defined prevention and intervention programs, researchers are now advocating for the need for multi-faceted intervention implementation monitoring systems. This special series highlights new methods and innovative approaches for accomplishing this task, while identifying challenges in this new area of research. We hope that the special series will expand the current definitions of treatment integrity, identify gaps in the literature-base, and outline practice implications and future research directions for the monitoring of interventions across diverse settings.

Acknowledgments

The first author has received, two NIMH grants, R34MH072982 and R01MH075787, for the purpose of this research. This research was made possible, in part, by the support of the School District of Philadelphia. Opinions contained in this report reflect those of the authors and do not necessarily reflect those of the School District of Philadelphia.

References

  • Basch CE, Sliepcevich EM, Gold RS, Duncan DF, Kolbe LJ. Avoiding type II errors in health education program evaluations: A case study. Health Education Quarterly. 1985;12(4):315–331. [PubMed]
  • Brown S, Rahn-Blakeslee Training school-based practitioners to collect intervention integrity data: One agency’s model. School Mental Health. 2009 doi: 10.1007/s12310-009-9014-9. [Cross Ref]
  • Burns MK, Peters R, Noell GH. Using performance feedback to enhance implementation fidelity of the problem-solving team process. Journal of School Psychology. 2008;46:537–550. [PubMed]
  • Chambless DL, Hollon SD. Defining empirically supported therapies. Journal of Consulting and Clinical Psychology. 1998;66:7–18. [PubMed]
  • Cochrane WS, Laux JM. A survey investigating school psychologists’ measurement of treatment integrity in school-based interventions and their beliefs about its importance. Psychology in the Schools. 2008;45:499–507.
  • Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review. 1998;18:23–45. [PubMed]
  • Drozd JF, Goldfried MR. A critical evaluation of the state-of-the-art in psychotherapy outcome research. Psychotherapy: Theory, Research, Practice, Training. 1996;23(2):171–180.
  • Evans SW, Schultz BK, Serpell ZN. Balancing feasibility and treatment adherence in school mental health research. Advances in School Mental Health Promotion. 2008;1(1):28–38.
  • Gresham FM. Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review. 1989;18:37–50.
  • Gresham FM, Gansle KA, Noell GH, Cohen S, et al. Treatment integrity of school-based behavioral intervention studies: 1980–1990. School Psychology Review. 1993;22(2):254–272.
  • Gresham FM, MacMillan D, Beebe-Frankenberger M, Bocian K. Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research & Practice. 2000;15:198–205.
  • Gullan RL, Feinberg BE, Freedman MA, Jawad A, Leff SS. Using participatory action research to design an intervention integrity system in the urban schools. School Mental Health. 2009 doi: 10.1007/s12310-009-9006-9. [PMC free article] [PubMed] [Cross Ref]
  • Iowa Department of Education. Iowa administrative rules of special education. Des Moines, IA: Author; 2007.
  • Kendall PC, Beidas RS. Smoothing the trail for dissemination of evidence-based practices for youth: Flexibility within fidelity. Professional Psychology: Research and Practice. 2007;38(1):13–20.
  • Klotz MB, Canter A. Response to intervention (RTI): A primer for parents. May, 2006. www.nasponline.org/resources/handouts/rtiprimer.pdf.
  • Leff SS, Crick NR, Angelucci J, Haye K, Jawad AF, Grossman M, et al. Social cognition in context: Validating a cartoon-based attributional measure for urban girls. Child Development. 2006;77:1351–1358. [PubMed]
  • Leff SS, Power TJ, Manz PH, Costigan TE, Nabors LA. School-based aggression prevention programs for young children: Current status and implications for violence prevention. School Psychology Review. 2001;30:343–360.
  • Luborsky L, DeRubeis RJ. The use of psychotherapy treatment manuals: A small revolution in psychotherapy research style. Clinical Psychology Review. 1984;4:5–14.
  • McIntyre L, Gresham F, DiGennaro F, Reed D. Treatment integrity of school-based interventions with children in the journal of applied behavioral analysis 1991–2005. Journal of Applied Behavioral Analysis. 2007;40:659–672. [PMC free article] [PubMed]
  • Nastasi BK, Varjas K, Schensul SL, Silva KT, Schensul JJ, Ratnayake P. The participatory intervention model: A framework for conceptualizing and promoting intervention acceptability. School Psychology Quarterly. 2000;15:207–232.
  • Noell GH, Witt JC, Slider NJ, Connell JE, Gatti SL, Williams KL, et al. Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review. 2005;34:87–106.
  • Perepletchikova F, Kazdin AE. Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice. 2005;12(4):365–383.
  • Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology. 2007;75(6):829–841. [PubMed]
  • Peterson CA, McConnell SR. Factors affecting the impact of social skills interventions in early childhood special education. Topics in Early Childhood Special Education. 1993;13:38–56.
  • Peterson L, Homer A, Wonderlich S. The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis. 1982;15:477–492. [PMC free article] [PubMed]
  • Power TJ, Blom-Hoffman J, Clarke AT, Riley-Tillman TC, Kelleher C, Manz PH. Reconceptualizing intervention integrity: A partnership-based framework for liking research with practice. Psychology in the Schools. 2005;42:495–507.
  • Power TJ, Soffer SL, Mautone JA, Jones HA, Clarke AT. Assessing teacher engagement in a family-school intervention for children with ADHD. School Mental Health. 2009 doi: 10.1007/s12310-009-9005-x. [PMC free article] [PubMed] [Cross Ref]
  • Swanger-Gagne MS, Garbacz SA, Sheridan SM. Intervention implementation integrity within a behavioral consultation model: Strategies for working with families at risk. School Mental Health. 2009 doi: 10.1007/s12310-009-9012-y. [Cross Ref]