Results of item response analyses of the 16 EBP items (15 items with the adult sample in which we did not include the item for developmentally appropriate treatment) demonstrated that a reliable measure of the extent to which juvenile and adult correctional facilities, and community treatment agencies serving offenders, have adopted various research-supported treatment practices suggested by the literature could be developed using Rasch modeling. The literature has made a distinction between EBP adoption and implementation, with adoption characterized by a facility’s use of EBPs and implementation characterized by the extent to which eligible recipients receive them (Roman and Johnson, 2002
). We have focused on EBP adoption in this study. Future studies incorporating measures of treatment participation, utilization, and fidelity are needed to study EBP implementation.
The observed patterns of EBP use were consistent with the Rasch model, reflecting both the difficulty of using the EBPs across programs as well as the extent to which the programs were using them. These findings advance the field of the assessment of EBPs, particularly with substance abusing offenders, which has been limited by either focusing on individual treatment practices or by assessing an array of practices using an inventory approach.
Inventory approaches to EBP assessment implicitly assume that all EBPs are equally important to the persons responsible for adopting innovative treatment practices. In contrast, and consistent with Knudsen et al. (2007)
, these data suggest that clusters of EBPs tend to occur together. This adoption pattern suggests that facilities using more EBPs may have overcome key resource-related and philosophical barriers to EBP use such that additional EBPs may be adopted with less difficulty. As such, these findings are consistent with Rogers’ (2003)
diffusion of innovation theory, which posits that new innovations are likely to be adopted when they are consistent with previously introduced technologies. While Knudsen et al. tested a clustering hypothesis with respect to medication use in private sector treatment facilities, the current study tests the idea with respect to psychosocial interventions adopted in criminal justice settings.
The fact that EBPs clustered together introduces new hypotheses concerning EBP adoption behavior in criminal justice settings. Namely, the findings suggest that innovators do not necessarily implement one EBP at a time but that they instead may implement certain practices together. For example, drug testing and systems integration tended to be equally likely to be used in both adult and juvenile samples. One process that may explain these seemingly distinct activites occurring together is that committing to do good drug testing necessitates that a program deal with other agencies in determining how to arrive at valid, clinically useful results, deciding with whom to share the results, etc. Similarly, the use of engagement techniques, assessment of treatment outcomes, and a planned 90 day duration of treatment have approximately equal difficulty levels. This pattern of results may suggest that programs serious about assessing the impact of their treatment practices take additional measures to ensure that offenders are also appropriately engaged in treatment given the consistency in the literature linking engagement and treatment outcome. Such programs may also want to ensure that treatment is of sufficient intensity and duration to produce the desired impact on treatment outcomes. Because EBPs tended to cluster together these results may assist researchers, clinicians, and program administrators plan for and prioritize new EBP adoption by starting with “easier” treatment practices or strategically targeting EBPs with a similar difficulty levels to EBPs facilities are currently using.
The Rasch measure we derived also shows evidence of concurrent validity because it was significantly associated with numerous organizational variables, namely the extent to which the facilities carried out joint activities, particularly with non-criminal justice focused organizations, internal support for new programs, training opportunities, management emphasis on the quality of treatment, and administrator commitment to the organization. As such, these findings are consistent with other organizational and health services research focused on predictors of innovation, including EBP adoption (Glisson, 2002
; Knudsen and Roman, 2004
; Rogers, 2003
; Wejnert, 2002
). With a few exceptions, the pattern of relationships was similar to what Henderson et al. (2007)
and Friedmann et al. (2007)
found in previous studies using the same data source, and those exceptions pointed to the benefits of Rasch modeling. First, the relationships were typically stronger using the Rasch measure, through smaller standard errors, which suggests that the Rasch measure produces estimates with improved precision. A second difference is that more of the training and resources variables were significant in these models. Namely, resources in the adult sample and physical facilities, resources, and funding in the juvenile sample significantly predicted EBP use. Predictably, training and quality facilities were positively correlated with EBP use; however, funding and other resources showed a negative association with EBP use. It may be that training and facilities comprise an infrastructure important to EBP use, while additional, more general resources play a superfluous role. In our experience, programs that adopt and implement EBPs are those that find a way to do so in spite of funding and resource limitations (within reasonable limits); therefore, these negative relationships may be associated with other administrator attitudes (e.g., hopelessness, comfort with the status quo) that were not assessed in this survey.
In comparison to the findings reported in Friedmann et al. (2007)
and Henderson et al. (2007)
, in which simple, summary counts of number of EBPs present served as the dependent measures, results using the Rasch-developed measure showed few substantive differences. However, the Rasch measure has stronger measurement properties in that it takes into account both the likelihood that programs are using the various EBPs (i.e., item difficulty in an IRT sense) and programs’ patterns of EBP use. And, the Rasch-developed technique allows us to examine patterns or clustering of EBPs in a manner that is not obvious from the counts of EBPs. In addition, because the Rasch model is a linear model producing latent trait estimates on a true interval scale, we can be certain that our Rasch-derived EBP measure has these same measurement properties. In essence, although the concurrent validity of the Rasch measure is not superior to the summary counts we reported previously (at least in this particular application), the construct validity of the Rasch measure is stronger given its superior measurement properties.
In terms of substantive interpretations of the results, it is of some concern, though not necessarily surprising, that developmental appropriateness and research-supported treatment modalities (i.e., CBT, therapeutic community, or other manualized treatment) were the least used EBPs. Research on empirically supported substance abuse treatments has indicated that treatment type does matter, and that with respect to adolescents, developmental concerns must be taken into account to maximize treatment gains (Dennis et al., 2003
; Liddle and Rowe, 2006
). However, a large body of research has indicated that transporting empirically supported treatments to naturalistic settings is plagued by many difficulties (Institute of Medicine, 1998
). Findings from the current study suggest that EBPs that do not necessarily require changing the service delivery infrastructure, or are driven by legislation and/or accreditation (e.g., incentives, qualified staff, use of standardized substance abuse assessment tools, comprehensive services) are used fairly frequently. However, EBPs requiring “deep structure” modifications of service delivery or the content of programming are less likely to be used. Therefore, we are encouraged by recent studies (Baer et al., 2007
; Liddle, et al., 2006
) suggesting that empirically supported treatments can be successfully transported to naturalistic settings, and policy movements supporting studies designed to disseminate research-supported treatment models and investigate their implementation and sustainability (e.g., NIDA’s Clinical Trials Network, http://www.nida.nih.gov/ctn
; National Institutes of Health, 2006
; Oregon’s passage of the Evidence-Based Practices legislation, etc.). Similar studies are greatly needed in criminal justice settings.
The current study is limited in certain respects. First, the items were selected from a broad survey assessing many different constructs. Although one goal of the survey was to assess EBPs, this was only one of several goals. As such, certain items needed to be defined by establishing threshold performances and combining items. It is possible that more parsimonious items could be derived for future investigations. Second, DIF between the corrections administrators and treatment directors was evidenced for two items, family involvement in treatment, and comprehensive services. Some researchers have recommended removing items with DIF, as they may reflect bias in how groups interpreted and responded to the items. We have chosen to retain them in the current study given the strong empirical base for family involvement in treatment (Henggeler et al., 1995
) and comprehensive services (Etheridge et al., 2001
). Our assumption is that rather than reflecting item bias, the DIF results from real differences in respondent perceptions between the two populations. Corrections settings face many more logistical difficulties providing family services than community settings (Henderson et al., 2007
), and conversely because offenders can not access services from other facilities when incarcerated and secure institutions often have legal mandates to provide services, corrections settings are more likely to provide offenders with comprehensive services (Taxman et al., 2007b
; Young et al., 2007
). Third, the response rate for juvenile facilities was substantially lower than the response rate for adult facilities. Although Taxman et al. (2007a)
found no evidence of response bias among the juvenile respondents, it is nevertheless possible that we may have found different results if a larger number of respondents working with juvenile offenders had completed the survey. Fourth, this is a cross sectional survey and longitudinal data would benefit a greater understanding of EBP use. Fifth, the data are limited to self-reports of program administrators, and therefore, there is no way of verifying their use of EBPs or examining the quality or fidelity with which they are used.
Despite these limitations, the current study also possesses noteworthy strengths. Foremost among these is the fact that the parent study obtained nationally representative estimates of substance abuse treatment practices in juvenile and adult correctional and community settings (Taxman 2007a
). Second, to our knowledge, this is the first application of IRT methods to the study of EBPs, and our findings suggest there is promise in conducting similar studies in the future. Such studies will help advance further stages of inquiry, namely assessing predictors of EBPs, and potentially setting the foundation for future interventions aimed at modifying the organizational context of corrections agencies so that they may be implemented more effectively. However, consensus on what constitutes evidence-based practice, and on measures for assessing its use, are necessary first steps in developing effective interventions.