Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Addiction. Author manuscript; available in PMC 2007 December 20.
Published in final edited form as:
PMCID: PMC2148498

A vision of the next generation of behavioral therapies research in the addictions*


Whither, or wither, empirically supported therapies? Increasingly rigorous research in behavioral therapies has yielded a large number of effective therapies, but comparatively little work, demonstrating that integrating empirically supported therapies (ESTs) into standard practice results in meaningful improvements in patient outcomes. Methodology and strategies for evaluating ESTs and their effectiveness in clinical practice is a fairly recent innovation, and a host of unanswered questions remain regarding issues such as selection among different ESTs and what type of ESTs should be emphasized in dissemination efforts, what type of clinicians should be trained in what type of ESTs, the most effective training strategies for various types of clinicians, the need for ongoing supervision to maintain minimum levels of treatment fidelity and skill. In this review, we call for broader use of new research strategies and methods relevant to dissemination of ESTs; these may include adaptive designs, identification of mechanisms of action to foster greater emphasis on effective change principles, training and adoption trials, as well as novel implementation strategies including computer-assisted therapy and computer-assisted training.

Keywords: Clinician training, computerized therapies, dissemination, empirically supported therapies, research to practice, stage model


By almost any measure, treatment efficacy research in the addictions has been a brilliant success. Multiple well-conducted clinical trials have yielded impressive empirical support for a number of well-defined behavioral therapies [15], that to a large extent parallel the progress of behavioral therapies research for the larger range of psychiatric disorders [68]. The enhanced focus on using rigorous methodology to identify effective therapies not only yielded numerous ‘empirically supported therapies’ (ESTs), but also led in part to current efforts to improve the quality of psychological treatments for mental disorders. These arose first in the United Kingdom, with the broader movement toward evidence-based medicine [9], and then in the United States, with the efforts of groups such as the Task Force on Promotion and Dissemination of Psychological Procedures of Division 12 (Clinical Psychology) of the American Psychological Association [7,1012] that described standards for defining ESTs (or closely related concepts such as ‘empirically validated therapies’, ‘empirically based treatments’, ‘science-based approaches’, and so on), and which classified multiple interventions in terms of their being ‘well-established/efficacious and specific’, ‘probably efficacious’ or ‘promising’. The criteria used to determine whether an intervention qualifies as ‘empirically supported’ consists typically of (a) whether there are at least two prospective, independent, randomized clinical trials in which the treatment has been compared to a well-defined control or comparison condition, (b) those trials have adequate statistical power, (c) the patient groups are defined using explicit inclusion/exclusion criteria, (d) the intervention is described in a manual or another clearly defined method useable as the basis for training, (e) treatment fidelity is monitored or treatment integrity is evaluated and (f) outcome assessments are well-validated and blinded as to treatment condition [68]. Using these standards, there are now at least 10 behavioral therapies that are classified as ‘efficacious’ or ‘probably efficacious’ for the addictions and well over 100 ESTs for a range of other mental disorders, many of which co-occur with addictions in adults [7].

The proliferation of ESTs and the efforts to define and label them have been controversial and sparked important discourse among researchers and clinicians (see [7,1318]) revolving around the questions, ‘Now that we have all these ESTs, what should we do with them?’ and ‘What do we need to know about the efficacy, value, and transferability of ESTs into clinical practice?’. In this paper, we will first consider some of the implications and questions that rise from the burgeoning number of ESTs, specifically for the field of addictions treatment research. Next, we will review briefly some of the strengths and weaknesses of current methods for moving ESTs into clinical practice. Finally, we will articulate a series of research questions for training, treatment programs and treatment research that follow from these implications.

The theoretical and practical questions regarding the identification of ESTs and how they should affect clinical practice are particularly pressing in the addictions, where the gap between research knowledge and clinical practice is both persistent and formidable [1921]. Despite the increasing availability of number of effective treatments in the past 20 years, the majority of treatment programs in the United States remain grounded in traditional counseling models that have largely not been evaluated rigorously [19,22]. In addition, many drug abuse treatment programs persist in their use of interventions and strategies that have been demonstrated to be ineffective (e.g. acupuncture for drug dependence) [23] and even some that may be harmful to some populations (e.g. group treatments for antisocial adolescents) [24]. It is also common practice for programs to offer high-cost services with little long-term effectiveness (e.g. opioid detoxification without follow-up treatment) [25,26], presumably because these services remain covered by third-party payors. Moreover, the quality of clinician training is variable and rarely includes meaningful training in ESTs; rates of clinician turnover within programs remain unacceptably high. Finally, objective assessment of clinician/ program performance and patient outcomes remain quite rare in clinical practice [2729]. That is, while it is increasingly common in general and specialty medical settings for consumers to have access to data on physician performance and rates of successful outcomes, even the most basic information on retention and substance abuse outcomes is often not even collected systematically, much less available to prospective patients as a basis for making an informed choice about a particular clinical program.

It is also becoming clear that establishing the efficacy of a treatment approach via clinical trials by no means ensures it is used in clinical practice. Although rigorously conducted efficacy studies have validated a broad range of ESTs, they have addressed very few of the key questions regarding dissemination of those approaches. For example, how should clinicians choose among the various ESTs? Multiple large-scale randomized trials have identified which treatments ‘work’ but have provided very little guidance regarding the relative superiority of the treatment alternatives [11]. Luborsky’s meta-analysis of the studies that have compared active treatments suggested a fairly small effect size (0.20) between different therapies [30,31], indicating that while ESTs have been demonstrated to be effective relative to control therapies, research has yet to identify meaningful outcome differences across various treatments. Similarly, the available literature on treatment-matching and the identification of patient predictors of outcome has provided relatively little clinically meaningful guidance regarding the type of individuals who are most (or least) likely to respond to specific ESTs (see [13,32,33]).

Beyond the lack of data demonstrating the relative superiority of any specific treatment and the lack of data regarding powerful patient moderators of outcome, clinical research has also been largely silent on the question: ‘Are ESTs actually superior to standard clinical practice; that is, do they work in the real world as well as the ivory tower?’. Until recently, there were very few studies that compared a specific EST to standard clinical practice or a ‘clinician’s choice’ condition [15,34,35]. Although clinical trials conducted in community settings do generally replicate the findings of those conducted in more highly controlled research settings [3537], research has addressed infrequently the issue of the extent to which introducing ESTs into clinical practice will result in significant or clinically meaningful improvements in outcome [13,15,38]. Without such data, it has been and will remain difficult to generate enthusiasm among policy makers or clinicians to alter their standard practice.

In the United States, the National Institute on Drug Abuse’s Clinical Trials Network (CTN) [39] is making important progress on this question by conducting a series of multi-site randomized clinical trials evaluating the benefit of integrating or adding ESTs to standard care. While retaining key design features such as random assignment to treatment conditions, objective assessment of outcome by blind evaluators, assessment of treatment integrity and fidelity, CTN trials are addressing the question: ‘do ESTs work in the real world?’ by conducting these trials in community-based clinics (rather than research or university settings), broadening inclusion/ exclusion criteria so that samples are as heterogeneous as possible (e.g. ‘taking all comers’ rather than treating highly selected samples), with delivery of study treatments by clinicians drawn from the staff of the clinics (as opposed to doctoral level clinicians selected for their skill or allegiance to a particular model) [40]. To date, the results of completed CTN trials suggest that, even in the context of high variability across sites, programs, clinicians and participants, ESTs such as contingency management (CM) can improve outcomes when added to standard treatment for stimulant dependence [41,42], motivational interviewing (MI) can improve retention when integrated into the early phase of out-patient substance abuse treatment [43,44] and detoxification using buprenorphine is a more effective detoxification strategy than is clonidine [45]. Moreover, the CTN trials providing extensive ‘hands-on’ experience in using novel therapies to a large number of clinics, as well as offering training and supervision in ESTs to a broad range of ‘real-world’ clinicians [46]. Data from these trials also suggest that clinicians drawn from the staff of CTN clinics can learn to implement ESTs such as MI, when provided with didactic training and ongoing monitoring of treatment fidelity via regular supervision based on review of session tapes. What is not yet clear is the extent to which trained clinicians will retain their new skills and continue to use them in clinical practice once the studies are completed and their performance is no longer monitored.

Another area largely unaddressed by existing efficacy study is the question: ‘what should happen when a patient doesn’t respond to an EST?’. Although there are now a number of viable alternative options for treating smoking (including buproprion, nicotine replacement and behavioral strategies), opioid dependence (including naltrexone, buprenorphine and methadone) and alcohol dependence (naltrexone, disulfiram and many behavioral approaches), there are few empirically based guidelines regarding, for example, strategies for opioid-dependent individuals who do not respond to methadone maintenance, cocaine users who do not respond to clinical management or smokers who do not respond to buproprion. Adaptive treatment designs, which evaluate systematically the relative benefit of augmenting, discontinuing or switching treatment strategies for individuals who do not respond to, or who relapse following, treatment are a potent strategy for addressing these very practical issues regarding clinicians’ response to poor patient outcomes in response to ESTs [29,47]. Large-scale efforts to develop treatment guidelines, adaptive treatment strategies and algorithms has begun in areas where multiple effective treatment alternatives exist, such as depression [4850], but have only recently been applied to the addictions [51,52].


The issues discussed in the section above suggest that immediate deployment of a massive army of trainers in the 100 or more ESTs to all clinicians world-wide will not be feasible, helpful or particularly effective in improving the quality of addictions treatment. Even if we assume that the integration of ESTs will improve standard care, there are remarkably few data on such fundamental questions as (1) ‘which of the available ESTs are likely to foster the greatest actual improvements in practice?’ and (2) ‘how best to train clinicians and to sustain the effects of training?’. Designation of a therapy as an ‘EST’ does not automatically also assure it of the designation ‘Easily Sustainable Treatment’: issues such as cost, ease of trainability, acceptability and attractiveness to patients can easily derail even a very powerful treatment from being used in clinical practice.

The Stage Model of Behavioral Therapies Research, developed by the National Institute on Drug Abuse in the United States, anticipated many of these issues and offers a heuristic model for thinking through research strategies and questions regarding how the availability of ESTs should alter clinical practice. The model articulates three progressive stages in the development and evaluation of behavioral therapies that roughly parallel those for the development of pharmacologic therapies [53]. Stage I consists of initial development of and pilot/feasibility testing for new and untested treatments [54]. Stage II consists principally of randomized controlled clinical trials to evaluate the efficacy of treatments that have shown promise or efficacy in initial studies (i.e. essentially the process that has produced ESTs and generated the criteria by which they are judged). Stage III is a more novel area that is intended to address issues of transportability into clinical practice of treatments whose efficacy has been demonstrated in prior Stage II efficacy trials. Although the methodology of Stage III research is still developing [55], we have argued it should address a range of research questions, including: (1) efficacy and generalizability (i.e. ‘will this treatment retain its efficacy when implemented by different practitioners in different settings, and with a broader range of patients?’, ‘is Treatment X truly more effective than current practice?’); (2) implementation issues (i.e. ‘what kinds of training, by what kinds of trainers, are necessary for what kinds of clinicians to learn a new technique?’); (3) cost-effectiveness issues (i.e. ‘compared with the costs of learning and implementing this treatment, what are the savings, particularly in comparison to the existing standard of care?’); and (4) consumer/marketing issues (i.e. ‘how acceptable is a new treatment to both clinicians, patients, and payors outside of research settings?’). Because we have discussed previously issues related to cost-effectiveness and consumer/marketing issues for addictions treatments [40], in the sections below we will focus primarily on issues of generalizability of ESTs and implementation/training issues.


Given the lack of clear superiority for any single EST and the sheer impracticability of all clinics offering all available ESTs, a reasonable start would be to identify a reasonable ‘first’-line therapy that can be offered to the majority of individuals entering treatment for a given primary disorder. The current default first-line therapy in the United States (if immediate detoxification is not needed and no pharmacological alternative exists) is group drug abuse counseling, usually with referral to self-help such as Alcoholics Anonymous meetings. This practice has largely been determined on the basis of cost and convenience considerations, as there is little research on the effectiveness of ‘group counseling’ (although we might extrapolate a relatively weak effect because unstructured group counseling is a frequently used control or comparison condition against ESTs that have been compared) [56]. Hence, identification of a reasonable ‘first-line’ treatment that is inexpensive, broadly applicable to a range of patients and settings, and has some data supporting its effectiveness is needed. Possible candidates for an empirically supported first-line approach for out-patient addictions treatment include MI, cognitive behavioral therapy (CBT) or combined MI/CBT, the last of which has been demonstrated to be effective in several substance using samples [5759] and may have cost and feasibility advantages relative to contingency management [60] or family/ couples therapy. Following an initial trial of the first-line approach, objective benchmarking outcomes (retention, urine toxicology screen results) could be monitored regularly to determine whether more intensive and costly interventions are indicated (e.g. pharmacotherapy or contingency management). The choice of an appropriate first-, second- or third-line EST could also be determined empirically, using the adaptive treatment strategy approach described above.


The dominant current strategy for training clinicians in the United States in ESTs is through brief workshops offered through federally supported addiction technology transfer centers (ATTCs) or single-state agencies, typically with continuing education credits offered as an incentive. The system thus capitalizes on self-selection, as clinicians are free to attend as many (or as few) EST workshops as they wish and there is no system in place for assuring minimal standards of competence for virtually any ESTs (e.g. through formal certification processes). Thus, there is increasing pressure on agencies and clinics to state that they offer ESTs and some incentives for clinicians to be exposed to ESTs, but no system or standards for ensuring that empirically supported behavioral therapies are delivered with even minimal levels of adherence or competence.

Broader use of any ESTs in clinical practice will inevitably raise a number of issues relative to the training of clinicians. First, who should be trained in ESTs (students, current practitioners in specialty programs, clinicians in private practice)? Students in social work, psychology, psychiatry and various addiction training programs who are just entering the field are an important target for training efforts, but systematic training in ESTs remains rare and uneven even in these programs [61,62]. Clinicians already working in specialty substance abuse treatment programs are a crucial target of EST training efforts. Although many clinicians claim extensive use of ESTs in clinical practice, independent review of audio-tapes of ‘standard treatment’ in sites participating in a recent CTN protocol indicated very infrequent use of any interventions associated with ESTs, including skills training, outcome planning or referral to self-help. Finally, although students/trainees and clinicians working in specialty programs are logical and reasonably accessible target audiences for training efforts, it should also be recognized that a very large number of clinicians who deliver services to substance users do not practice in specialty clinics or with highly specialized populations where the number of ESTs might be manageable.

Next, how much and what kind of training will be required for the various types of clinicians? The standard training approach of brief workshops has been shown to be of limited effectiveness in imparting key skills and competence to experienced clinicians and appears, at best, to provide exposure to (rather than mastery of), novel approaches [21,6366]. For example, Miller and colleagues demonstrated that workshop training was insufficient to impart key MI skills to experienced drug abuse clinicians [67,68]. In a similar study that randomized experienced clinicians to different methods for learning CBT [69], only the training condition that offered intensive didactic training plus performance-based supervision and feedback enabled the clinicians to reach adherence and competence standards that would be required for a CBT efficacy trial. However, as discussed in more detail below, this type of training is expensive, time-consuming and the absence for several days of clinical staff can be extremely disruptive to small clinics. Thus, research on the effectiveness of alternative training models is also needed, such as ‘train the trainer’ models, where one key clinical leader from a program staff receives elaborate instruction, not only in a specific EST but also in how to train other clinicians in the approach and how to monitor and supervise it as well. Ongoing research by Steve Martino in our group is directly contrasting the effectiveness of a ‘train the trainer’ approach to direct training of all staff by a single expert trainer.

Can all clinicians be trained? It is not yet clear that all clinicians, particularly those strongly wedded to a particular clinical approach, can attain competence in all clinical approaches, and the available anecdotal data suggests that efforts to train ‘all comers’ to competence may not be successful. Those training studies that have included a wide range of clinician/volunteers to be trained in a particular EST have noted substantial variability in clinician competence, even after extensive training and feedback [6972]. Our data also suggest that a number of clinicians who had only ‘on-the-job’ training prior to participating in the training study lacked basic clinical skills, and it is not clear how these basic skills should be taught, or even whether they can be taught [69,73]. Similarly, previous multi-site trials of behavioral treatments have indicated that a substantial proportion of clinicians who begin protocol training either do not meet competence criteria and are not certified to participate in the trial, or leave training through a self-selection process [70,72,7476]. It is also not clear whether most clinicians can achieve competence in multiple ESTs and, even if they did so, whether they could maintain that competence in clinical practice. Thus, it may be a reasonable starting-point to simply require that clinicians demonstrate competence in a minimum of one EST, with reasonable exposure to other, newly emerging therapies as needed and as dictated by the clinical populations with whom they work. This strategy would at least facilitate broader availability of ESTs to a wider proportion of individuals with substance use problems.

How can clinicians’ competence in ESTs be maintained? The emerging data from training studies suggest that review, feedback, and coaching of therapists implementation of ESTs is crucial in developing competence in behavioral therapies [7780]. It is unlikely that competence could be maintained without some ongoing monitoring of a clinician’s implementation of a given EST, with ongoing provision of supervision and support from clinicians who have training and experience with the approach [66,8083]. However, clinical supervision based on objective standards or systems is virtually nonexistent in the United States, and is also not a compensated or reimbursable service. In fact, in our two studies of clinician training methods, the clinicians reported that they received an average on 1 hour of clinical supervision each week, and this was typically in the context of a staff meeting in which clinicians’ case-loads and the status of their charting and paperwork was reviewed. Moreover, given the high rate of turnover of clinicians in substance abuse treatment programs [84,85], extensive training, certification and supervision procedures may not be seen as cost-effective. Thus, in order to initiate and then sustain delivery of ESTs in practice, it will be important to demonstrate not only that ESTs result in measurably improved outcomes when added or integrated into standard clinical programs, but also that the provision of training and supervision in ESTs reduces staff turnover or reduces other costs. Hence, it will be essential that cost analyses be integrated into such trials in order to generate the data needed to persuade policy makers, third-party payors and program managers that adoption of ESTs is warranted [11,8688] as well as to counter financial and programmatic disincentives for adoption of ESTs [38,89].


Given the proliferation of ESTs, what ESTs should be emphasized in dissemination efforts? Some selection and prioritization will be required, as it is clearly unreasonable to demand that clinicians be familiar with, much less competent in, the more than 100 available ESTs. Similarly, it is unlikely that most training programs could have the capacity to offer training in all available treatments for all disorders. Hence, there is a need for some way to either combine similar approaches/strategies or to select among the ESTs, particularly at the level of training programs.

Regarding the former strategy, a number of investigators have called for focus upon and teaching of basic principles and strategies, rather than focus on ‘name brand’ therapies [13,15,17,90,91]. This approach emphasizes understanding of the principles and mechanisms of change inherent in effective treatments [9294]. A definitive list of change principles, although still elusive, would presumably be brief and arguably easier to use as a basis for training clinicians [13,95]. An example would be Beutler’s approach of using more directive techniques for externalizing disorders and more exploratory techniques for internalizing disorders [96,97].

In the addictions, while there are multiple variants and population-specific versions of many different treatment types, many of these fall under the broader categories with general principles/strategies such as cognitive–behavioral, behavioral/contingency management, interpersonal/psychodynamic and family approaches [1]. Trainees could first achieve general familiarity or basic competence with the broad principles and strategies of these core approaches and then subsequently master specific ESTs on the basis of need or interest.

This is generally the approach used currently, where trainees are exposed to a few basic skills in their formal didactic training, and then learn specific techniques in their clinical practica [61]. The available evidence also suggests that this approach is resulting in very little actual use of ESTs in practice. For example, in our CBT training study, although the clinicians claimed comparatively high levels of familiarity with CBT, their pretraining CBT adherence/competence ratings were strikingly low [69]. Similarly, in the CTN study of MI, a self-report survey of the clinicians delivering the standard treatment condition prior to the training indicated that the therapists reported regular use of techniques associated with 12-Step/disease model concepts, CBT and MI [98]. However, ratings based on session tapes indicated interventions such as referral to self-help meetings, skills training, use of assessment/feedback were almost nonexistent, as were use of basic counseling strategies such as program orientation, goal setting and treatment planning.

This approach also raises the issue of the robustness of ESTs; that is, the extent to which ESTs remain effective when practiced imprecisely or when used with clients who differ (in terms of psychiatric comorbidity, multiple drug use or complexity and range of co-occurring problems) from those who participated in the original efficacy studies [99,100]. For example, CM has been shown to be a particularly effective approach across a range of substance use disorders when delivered consistently, at adequately high reinforcer magnitude and when focusing on a single drug or behavior [2,101]. However, CM has also been shown to be highly sensitive to relatively small changes in reinforcement magnitude or reinforcement schedules [2,101,102]. Thus, because CM is more effective when applied to a single type of drug use rather than multiple substances or target behaviors, and large proportions of clinical populations use multiple substances, CM may be less effective in the clinical situations where it is most needed.

It is also unclear whether some of the major classes of psychotherapy are highly interchangeable in their applications to different problem areas. For example, CBT approaches, grounded in similar theoretical models of the relationships between behavior, affect and cognition, have also been shown to be effective in populations of individuals with depressive and anxiety disorders [6,8,103]. However, in practice, CBT models for substance use, depression and anxiety sharply differ from one another with regard to their relative emphasis on behavior versus cognition, use of specific strategies and exercises and types of extra-session practice exercises. Hence, it cannot be assumed that a clinician who is highly accomplished in CBT for depression can easily ‘pick up’ CBT for addiction without substantial training and supervised practice. In contrast, interpersonal [104] or motivational interviewing [105,106] approaches appear to afford considerable leeway and may not require extensive retraining to be applied to different types of individuals and disorders. A related question is the degree to which high levels of adherence and competence is associated with measurable differences in outcome: while therapist skill and adherence to manual guidelines have been linked to outcomes for several treatments [107109], in other areas the findings have been more mixed or even negative [110,111].

Regarding the approach of selecting and training in only a few ESTs, the ‘divide and conquer’ strategy might, for example, require training programs to focus on fewer ESTs, or require that trainees master and demonstrate only two or three specific strategies. These could be selected on the basis of ‘first-line’ approaches for two or three common disorders or, for specialty training programs, two or three ESTs within a specific class of disorders. This strategy allows less breadth in training but would have the advantage of facilitating clear standards for establishing competence and certification; in effect, producing clinicians who are very good at a few therapies rather than who are only moderately effective (or worse) in many.

For experienced therapists receiving training in ESTs within either specialty training or treatment programs (and thus have more familiarity with basic principles of change), offering specialty training in single ESTs may be a reasonable approach. We have used the strategy of training experienced clinicians in single ESTs with clinicians recruited in two ways: (1) preselected, experienced clinicians volunteering for efficacy studies and (2) clinicians serving in community programs chosen to participate in effectiveness studies. For example, we have conducted a series of studies evaluating different strategies for training clinician/volunteers in CBT, MI, CM or 12-Step facilitation (TSF). In these trials, clinicians working full-time in substance abuse treatment facilities are randomized to one of several training conditions. In our initial study of CBT training methods, 78 clinicians were assigned to one of three training conditions: review of the CBT manual only, access to a web-based training site (which included additional frequently asked questions and practice exercises) plus the manual, or a 3-day didactic seminar plus up to three sessions of supervision from a CBT expert trainer based on actual session tapes submitted by the participants. Outcomes included (1) between-group comparisons of the clinicians’ ability to demonstrate key CBT techniques based on structured role-plays administered before and after training, and (2) scores on a CBT knowledge quiz. The videotaped role-plays were scored by independent raters, blind to the participants’ training condition as well as time (e.g. pre- versus post-training) and on adherence/competence ratings of specific CBT techniques from the Yale Adherence and Competence Scale (YACs) [81]. Although all groups demonstrated improved adherence and competence scores over time, the only training condition that reached levels of skill consistent with those required of clinicians participating in our CBT efficacy trials was the seminar + supervision condition, with intermediate ratings for the web condition. The mean effect size for the seminar + supervision versus manual-only condition comparisons was consistent with a large effect (0.69), while the average effect size for the web versus manual-only condition contrasts was consistent with a medium-sized effect (0.30). In addition, the seminar + supervision condition was associated with significantly more clinicians reaching criterion levels for adequate fidelity than those assigned to the manual-only condition (54% versus 15%). Our subsequent training studies, which have evaluated different training strategies for CM, MI and TSF [112], have also used the YACs as a training, supervision and outcome tool to foster comparison across studies and provide an objective means of benchmarking therapist performance.

A notable finding from our series of training trials is the surprisingly positive performance of the computer-based training conditions [69,112]. Given the many dissemination challenges noted in this paper, the recent availability of distance learning methods, particularly computer and web-based training, is an exciting new model for training clinicians [113119] and may present a feasible and less costly strategy for training larger numbers of substance use clinicians than is possible through standard, face-to-face training strategies. Given the time constraints and variable educational background of substance use clinicians, computer-based training may offer a number of potential advantages, including flexibility in scheduling, allowing more clinicians to access training, increasing opportunities for practice, as well as increased flexibility and individualization in pace and material covered [120]. Moreover, the rate of staff turnover in many community treatment programs is quite high, with estimates ranging from 16% to 50% per year [22,121,122; P. Roman, personal communication, 2002]; this underlines the need for more rapid, flexible and inexpensive methods of training such as computer-assisted models.

Although comparatively few well-designed randomized trials comparing ‘live’ teaching versus computer-assisted training have been conducted, those that have been done suggested that computer-assisted training can (1) be as or more effective than traditional lecture format training and (2) improve skills as well as knowledge [123,124]. It should be noted that computer-based training might have potential disadvantages as well. Computers are still not universally available in treatment or training programs in the United States [27], and clinicians who are not familiar with computers may find it frustrating to use programs that are not highly user-friendly. The extent to which computer-assisted training programs can convey basic clinical skills such as the ability to form an alliance or maintain session structure is not clear (although it is not yet clear whether traditional face-to-face training can do this, either) [120,125127]. Thus, it is critical that computer-based training programs be carefully evaluated, with a range of clinicians and in comparison to standard methods of training, before they are widely disseminated.

As a second example of the ‘divide and conquer’ strategy, in the CTN studies [43,44] we have instituted a training model intended to foster both competence in specific ESTs for clinicians participating in these effectiveness studies as well as persistence of the approach in the clinic after the trials are completed. In this model, expert trainers provide training and supervision to two types of clinicians. First, based on training strategies used in previous multi-site trials [70,71,76], experienced clinicians working at the clinics are provided with roughly 2 days of training, which includes detailed review of the training manual plus extensive role-play exercises and feedback; this is followed by close supervision on video- or audio-taped training cases until pre-established standards of fidelity and competence are met for certification [77]. However, to foster greater durability of the approach in the clinic, the expert trainers also provide more intensive training to at least one clinician in a clinical leadership role at the site. The clinician/supervisor is trained to competence standards in the treatment but also receives additional instruction in strategies and standards for supervision of the EST and then also plays a role in monitoring adherence/competence of the site clinicians as the trial goes on. Thus, after the trial is completed, these clinical supervisors will have accrued sufficient experience to enable them to provide ongoing training and supervision in the approach to other clinicians at that setting [46].


In the sections above we have tried to highlight some of the complexities, challenges and unanswered questions regarding ESTs into standard clinical practice. Given the large number of issues related to training clinicians, the high relative cost of ESTs compared to the existing standard care models, the high levels of variability in delivery and outcome of ESTs across clinicians [128130], and the barriers and costs of introducing performance-based supervision into standard clinical practice, a more novel alternative strategy would be to deliver treatments directly to patients via computer-assisted training. Computer-assisted therapies offer a number of potential advantages, including offering more therapeutic time to patients, ease of time scheduling (as patients can access computer programs outside of appointment times or regular clinic hours), allowing more patients to access treatment, providing greater confidentiality, increasing opportunities for practice and fostering patients’ sense of mastery and control [131,132]. Particularly in cases where clinician training is difficult or time-consuming, computer-based therapies may make effective treatments more accessible to a much broader range of treatment than the current system is able to reach [133].

Computer-assisted therapies potentially also offer more consistent delivery of interventions to patients, particularly for comparatively complex approaches such as CBT where clinician fidelity and skill in implementing the treatment tends to be variable. Computer-assisted training, as an adjunct to standard treatment, offers a number of attractive features: these include offering more opportunities for patients to access treatment at low cost, relieving clinicians of some repetitive tasks, and offering an alternative method of skills training for overburdened clinicians who can rarely offer individualized time to patients.

Computer-assisted treatment, although still in comparatively early stages of development, has shown promise in several areas of mental health treatment, including simple phobias [134], obsessive-compulsive disorder [135], panic disorder [136], depression [137,138], attention deficit hyperactive disorder (ADHD) [139] and smoking [140]. We are currently piloting a computer-based version of CBT (‘CBT4CBT’), a user-friendly, multimedia version of our CBT manual [141]. The program includes six modules covering key basic CBT concepts (functional analyses, coping with craving, monitoring and changing cognitions), with multiple movie-like vignettes of realistic characters implementing these skills to change their drug use. Thus, patients can actually see examples of use of coping skills and implementation of CBT principles, rather than simply hearing a clinicians’ abstract oral presentation of those principles. It also includes interactive graphics and narration teaching of basic skills, interactive exercises, vignettes of effective versus ineffective use of skills, and detailed examples of practice exercises. We are currently piloting this program in a community-based treatment center to evaluate (1) its feasibility and acceptability to patients and clinicians and (2) its effectiveness as an addition to standard treatment. Clearly, computer-assisted treatment is not a total substitute for a live clinician and unlikely to be applicable to severely impaired patients or to longer-term, more exploratory therapies. However, given the efficiency and low cost of these approaches, there may be less need to show that computerized ESTs are superior to treatment as usual, as they are bound to be less costly than therapist-delivered treatments, even if they are equivalent or slightly inferior to standard care approaches.


Whither ESTs? Like freshly minted PhD recipients holding their diplomas, proud developers of ESTs for the addictions have passed one milestone only to face a series of other challenges to overcome before their accomplishments pay off. Unlike Food and Drug Administration (FDA)-approved medications that can be manufactured in bulk and delivered in pure form anywhere in the world, new behavioral treatments can be disseminated only through training therapist after therapist, with the hope that they will remain in practice and stay motivated to deliver the treatment. Before (or while) ESTs are put into mass production, several new lines of research are needed to address important questions about the value of making ESTs the standard of care and about the best ways to disseminate them. First and most fundamentally, more research is needed that verifies the superior efficacy and cost-effectiveness of ESTs over standard practice. Second, when multiple ESTs are available, studies with adaptive designs are needed to guide the choice of front-line treatments and to identify optimal sequences of treatments to deliver when initial results are poor. Third, a new subfield of ‘training effectiveness’ research is needed to address a series of questions about treatment dissemination that parallel those articulated by Paul [142,143] about behavioral therapies efficacy research: that is, given substantial cross-treatment differences in complexity and difficulty of delivery, therapist training research needs to answer: ‘which training is needed in which therapy for which trainee to treat which type of patient?’. Fourth, while behavioral treatments resist formulation and delivery with the precision of FDA-approved medications, computer-assisted therapist training and computer-delivered treatment hold considerable promise for bringing the efficiency and precision of EST dissemination to new levels.


Support was provided by NIDA grants K05-DA00457, K05-DA00089, P50-DA09241, and the US Department of Veterans Affairs VISN 1 Mental Illness Research, Education and Clinical Center (MIRECC).


*Presented as the Society Lecture at the Society for the Study of Addictions Annual Symposium, 17 November 2005.


1. Carroll KM, Onken LS. Behavioral therapies for drug abuse. Am J Psychiatry. 2005;162:1452–60. [PMC free article] [PubMed]
2. Griffith JD, Rowan-Szal GA, Roark RR, Simpson DD. Contingency management in outpatient methadone treatment: a meta-analysis. Drug Alcohol Depend. 2000;58:55–66. [PubMed]
3. Stanton MD, Shadish WR. Outcome, attrition, and family-couples treatment for drug abuse: a meta-analysis and review of the controlled, comparative studies. Psychol Bull. 1997;122:170–91. [PubMed]
4. Burke BL, Arkowitz H, Menchola M. The efficacy of motivational interviewing: a meta-analysis of controlled clinical trials. J Consult Clin Psychol. 2003;71:843–61. [PubMed]
5. Irvin JE, Bowers CA, Dunn ME, Wong MC. Efficacy of relapse prevention: a meta-analytic review. J Consult Clin Psychol. 1999;67:563–70. [PubMed]
6. Nathan PE, Gorman JM. A Guide to Treatments That Work. 2. New York: Oxford University Press; 2002.
7. Chambless DL, Ollendick TH. Empirically supported psychological interventions: controversies evidence. Annu Rev Psychol. 2001;52:685–716. [PubMed]
8. Roth A, Fonagy P. What Works for Whom? a Critical Review of the Psychotherapy Literature. 2. New York: Guilford Press; 2005.
9. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine. New York: Churchill Livingstone; 1997.
10. Chambless DL, Hollon SD. Defining empirically supported therapies. J Consult Clin Psychol. 1998;66:7–18. [PubMed]
11. Beutler LE. David and Goliath: when empirical and clinical standards of practice meet. Am Psychol. 2000;55:997–1007. [PubMed]
12. Sanderson WC. Why empirically supported psychological treatments are important. Behav Modif. 2003;27:290–9. [PubMed]
13. Follette WC, Beitz K. Adding a more rigorous scientific agenda to the empirically supported treatment movement. Behav Modif. 2003;27:369–86. [PubMed]
14. Rosen GM, Davidson GC. Psychology should list empirically suported principles of change (ESPs) and not credential trademarked therapies or other treatment packages. Behav Modif. 2003;27:300–12. [PubMed]
15. Westen D, Novotney CM, Thompson-Brenner H. The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials. Psychol Bull. 2004;130:631–63. [PubMed]
16. Garfield SL. Some problems associated with ‘validated’ forms of therapy. Clin Psychol Sci Pract. 1996;3:218–29.
17. Wampold BE, Bhati KS. Attending to the omissions: a historical examination of evidence-based practice movements. Profess Psychol Res Pract. 2004;35:563–70.
18. Herbert JD. The science and practice of empirically supported treatments. Behav Modif. 2003;27:412–30. [PubMed]
19. Institute of Medicine. Bridging the Gap Between Practice and Research: Forging Partnerships with Community-Based Drug and Alcohol Treatment. Washington, DC: National Academy Press; 1998. [PubMed]
20. Institute of Medicine. Crossing the Quality Chasm: a New Mental Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
21. Fals-Stewart W, Logsdon T, Birchler GR. Diffusion of an empirically supported treatment for substance abuse: an organizational autopsy of technology transfer success and failure. Clin Psychol Sci Pract. 2004;11:177–82.
22. McLellan AT, McKay JR. The treatment of addiction: what can research offer practice? In: Lamb S, Greenlick MR, McCarty D, editors. Bridging the gap between practice and research: forging partnerships with community based drug and alcohol treatment. Washington, DC: National Academy Press; 1998. pp. 147–85.
23. Margolin A, Kleber HD, Avants SK, et al. Acupuncture for the treatment of cocaine addiction: a randomized controlled trial. JAMA. 2002;287:55–63. [PubMed]
24. Dishion TJ, McCord J, Poulin F. When interventions harm: peer groups and problem behavior. Am Psychol. 1999;54:755–64. [PubMed]
25. Mattick RP, Breen C, Kimber J, Davoli M. Methadone maintenance therapy versus no opioid replacement therapy for opioid dependence. Cochrane Database Syst Rev. 2003;2:CD002209. [PubMed]
26. Masson CL, Barnett PG, Sees KL, Delucchi KL, Rosen A, Wong W, et al. Cost and cost-effectiveness of standard methadone maintenance treatment compared to enriched 180-day methadone detoxification. Addiction. 2004;99:718–26. [PubMed]
27. McLellan AT, Carise D, Kleber HD. Can the national addiction treatment infrastructure support the public’s demand for quality care? J Subst Abuse Treat. 2003;25:117–21. [PubMed]
28. Babor TF, Higgins-Biddle JC. Alcohol screening and brief intervention: dissemination strategies for medical practice and public health. Addiction. 2000;95:677–86. [PubMed]
29. Lambert ME, Whipple JL, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Pract. 2003;10:288–301.
30. Luborsky L, Rosenthal R, Diguer L, Andrusyna TP, Berman JS, Levitt JT, et al. The dodo bird verdict is alive and well—mostly. Clin Psychol Sci Pract. 2002;9:2–12.
31. Luborsky L, Singer B, Luborsky L. Comparative studies of psychotherapies: is it true that ‘Everyone has won all must have prizes’? Arch Gen Psychiatry. 1975;32:995–1008. [PubMed]
32. Project MATCH Research Group. Matching alcohol treatments to client heterogeneity: Project MATCH posttreatment drinking. J Stud Alcohol. 1997;58:7–29. [PubMed]
33. UK Alcohol Treatment Trial (UKATT) Research Team. Effectiveness of treatment for alcohol problems: findings of the randomised UK alcohol treatment trial (UKATT) BMJ. 2005;331:541. [PMC free article] [PubMed]
34. Morgenstern J, Blanchard KA, Morgan TJ, Labouvie E, Hayaki J. Testing the effectiveness of cognitive-behavioral treatment for substance abuse in a community setting: within treatment and posttreatment findings. J Consult Clin Psychol. 2001;69:1007–17. [PubMed]
35. Addis ME, Waltz J. Implicit and untested assumptions aout the role of psychotherapy treatment manuals in evidence-based mental health practice. Clin Psychol Sci Pract. 2002;9:421–4.
36. Shadish WR, Matt GE, Navarro AM, Siegle G, Crits-Christoph P, Hazelrigg MD, et al. Evidence that therapy works in clinically representative conditions. J Consult Clin Psychol. 1997;65:355–65. [PubMed]
37. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care. Am Psychol. 2006;61:671–89. [PubMed]
38. Addis ME. Methods for disseminating research products and increasing evidence-based practice: promises, obstacles, and future directions. Clin Psychol Sci Pract. 2002;9:367–78.
39. Hanson GR, Leshner AI, Tai B. Putting drug abuse research to work in real-life settings. J Subst Abuse Treat. 2002;23:69–70. [PubMed]
40. Carroll KM, Rounsaville BJ. Bridging the gap between research and practice in substance abuse treatment: a hybrid model linking efficacy and effectiveness research. Psychiatr Serv. 2003;54:333–9. [PMC free article] [PubMed]
41. Peirce JM, Petry NM, Stitzer ML, Blaine JD, Kellog S, Satterfield F, et al. Effects of lower-cost incentives on stimulant abstinence in methadone maintenance treatment: a National Drug Abuse Treatment Clinical Trials Network study. Arch Gen Psychiatry. 2006;63:201–8. [PubMed]
42. Petry NM, Peirce JM, Stitzer ML, Blaine JD, Roll JM, Cohen A, et al. Effect of prize-based incentives on outcomes in stimulant abusers in outpatient psychosocial treatment programs: a national drug abuse Clinical Trials Network study. Arch Gen Psychiatry. 2005;62:1148–56. [PubMed]
43. Carroll KM, Ball SA, Nich C, Martino S, Frankforter T, Farentinos C, et al. Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: a multisite effectiveness study. Drug Alcohol Depend. 2006;81:301–12. [PMC free article] [PubMed]
44. Ball SA, Martino S, Nich C, Frankforter TL, Van Horn DH, Crits-Christoph P, et al. Site matters: motivational enhancement therapy in community drug abuse clinics. J Consult Clin Psychol. in press. [PMC free article] [PubMed]
45. Ling W, Amass L, Shoptaw S, Annon JJ, Hillhouse M, Babcock D, et al. 2005 A multicenter randomized trial of buprenorphine-naloxone versus clonidine for opioid detoxification: findings from the National Institute on Drug Abuse Clinical Trials Network. Addiction. 2006;100:1090–100. [PMC free article] [PubMed]
46. Carroll KM, Farentinos C, Ball SA, Crits-Christoph P, Libby B, Morgenstern J, et al. MET meets the real world: design issues and clinical strategies in the Clinical Trials Network. J Subst Abuse Treat. 2002;23:73–80. [PMC free article] [PubMed]
47. Collins LM, Murphy SA, Bierman KL. A conceptual framework for adaptive preventive interventions. Prev Sci. 2004;5:185–96. [PMC free article] [PubMed]
48. Rush AJ, Trivedi MH, Wisniewski SR, Stewart JW, Nierenberg AA, Thase ME, et al. Bupropion-SR, sertra-line, or venlafaxine-XR after failure of SSRIs for depression. N Engl J Med. 2006;354:1231–42. [PubMed]
49. Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, et al. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25:119–42. [PubMed]
50. Trivedi MH, Fava M, Wisniewski SR, Thase ME, Quitkin F, Warden D, et al. Medication augmentation after the failure of SSRIs for depression. N Engl J Med. 2006;354:1243–52. [PubMed]
51. Murphy SA, Lynch KG, Oslin D, McKay JR, Tenhave T. Developing adaptive treatment strategies in substance abuse research. Drug Alcohol Depend. 2007;88(Suppl 2):S24–S39. [PMC free article] [PubMed]
52. O’Malley SS, Rounsaville BJ, Farren C, Namkoong K, Wu R, Robinson J, et al. Initial and maintenance naltrex-one treatment for alcohol dependence using primary care vs. specialty care: a nested sequence of three randomized trials. Arch Intern Med. 2003;163:1695–704. [PubMed]
53. Onken LS, Blaine JD, Battjes R. Behavioral therapy research: a conceptualization of a process. In: Hennegler SW, Amentos R, editors. Innovative approaches for difficult to treat populations. Washington, DC: American Psychiatric Press; 1997. pp. 477–85.
54. Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: getting started and moving on from Stage I. Clin Psychol Sci Pract. 2001;8:133–42.
55. Hohmann AA, Shear MK. Community-based intervention research: coping with the ‘noise’ of real life in study design. Am J Psychiatry. 2002;159:202–7. [PubMed]
56. Miller WR, Wilbourne PL. Mesa Grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addiction. 2002;97:265–77. [PubMed]
57. MTP Research Group. Brief treatments for cannabis dependence: findings from a randomized multisite trial. J Consult Clin Psychol. 2004;72:455–66. [PubMed]
58. Carroll KM, Easton CJ, Nich C, Hunkele KA, Neavins TM, Sinha R, et al. The use of contingency management and motivational/skills-building therapy to treat young adults with marijuana dependence. J Consult Clin Psychol. 2006;74:955–66. [PMC free article] [PubMed]
59. Anton RF, O’Malley SS, Ciraulo DA, Cisler RA, Couper D, Donovan DM, et al. Combined pharmacotherapies and behavioral interventions for alcohol dependence: the COMBINE study: a randomized controlled trial. JAMA. 2006;295:2003–17. [PubMed]
60. Olmstead T, Sindelar JL, Easton CJ, Carroll KM. The cost-effectiveness of four treatments for marijuana dependence. Addiction. in press. [PMC free article] [PubMed]
61. Weissman MM, Verdeli H, Gameroff MJ, Bledsoe SE, Betts K, Mufson L, et al. National survey of psychotherapy training in psychiatry, psychology, and social work. Arch Gen Psychiatry. 2006;63:925–34. [PubMed]
62. Crits-Christoph P, Frank E, Chambless DL, Brody C, Karp JF. Training in empirically validated therapies: what are clinical psychology students learning? Profess Psychol Res Pract. 1995;26:514–22.
63. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA. 1992;268:1111–17. [PubMed]
64. Gotham HJ. Diffusion of mental health and substance abuse treatments: development, dissemination, and implementation. Clin Psychol Sci Pract. 2004;11:160–76.
65. VandeCreek L, Knapp S, Brace K. Mandatory continuing education for licensed psychologists: its rationale and current implications. Profess Psychol Res Pract. 1990;21:135–40.
66. Barlow DH, Levitt JT, Bufka LF. The dissemination of empirically supported treatments: a view to the future. Behav Res Ther. 1999;37:S147–62. [PubMed]
67. Miller WR, Yahne CE, Moyers TB, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivation interviewing. J Consult Clin Psychol. 2004;72:1050–62. [PubMed]
68. Rubel EC, Sobell LC, Miller WR. Do continuing workshops improve participants skills? Effects of a motivational interviewing workshop on substance-abuse counselors’ skills and knowledge. Behav Ther. 2000;23:73–7.
69. Sholomskas D, Syracuse G, Ball SA, Nuro KF, Rounsaville BJ, Carroll KM. We don’t train in vain: a dissemination trial of three strategies for training clinicians in cognitive behavioral therapy. J Consult Clin Psychol. 2005;73 :106–15. [PMC free article] [PubMed]
70. Crits-Christoph P, Siqueland L, Chittams J, Barber JP, Beck AT, Frank A, et al. Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. J Consult Clin Psychol. 1998;66:484–92. [PubMed]
71. Weissman MM, Rounsaville BJ, Chevron E. Training psychotherapists to participate in psychotherapy outcome studies. Am J Psychiatry. 1982;139:1442–6. [PubMed]
72. Carroll KM, Connors GJ, Cooney NL, DiClemente CC, Donovan DM, Longabaugh RL, et al. Internal validity of Project MATCH treatments: discriminability and integrity. J Consult Clin Psychol. 1998;66:290–303. [PubMed]
73. Vakoch DA, Strupp HH. The evolution of psychotherapy training: reflections on manual-based learning and future activities. J Clin Psychol. 2000;56:309–18. [PubMed]
74. Miller WR, Moyers TB, Arciniega L, Ernst D, Force-himes A. Training, supervision and quality monitoring of the COMBINE Study behavioral interventions. J Stud Alcohol. 2005;15(Suppl):188–95. discussion 168–9. [PubMed]
75. Tober G, Godfrey C, Parrott S, Copello A, Farrin A, Hodgson R, et al. Setting standards for training and competence: the UK alcohol treatment trial. Alcohol Alcohol. 2005;40:413–8. [PubMed]
76. Carroll KM, Kadden R, Donovan D, Zweben A, Roun-saville BJ. Implementing treatment and protecting the validity of the independent variable in treatment matching studies. J Stud Alcohol. 1994;(Suppl 12):149–55. [PubMed]
77. Baer JS, Ball SA, Campbell BK, Miele GM, Schoener EP, Tracy K. Training and fidelity monitoring of behavioral interventions in multi-site addictions research. Drug Alcohol Depend. 2007;87:107–18. [PMC free article] [PubMed]
78. Andrzejewski ME, Kirby KC, Morral AR, Iguchi MY. Technology transfer through performance management: the effects of graphical feedback and positive reinforcement on drug treatment counselors’ behavior. Drug Alcohol Depend. 2001;63:179–86. [PubMed]
79. Martin GW, Herie MA, Turner BJ, Cunningham JA. A social marketing model for disseminating research-based treatments to addictions treatment providers. Addiction. 1998;93:1703–15. [PubMed]
80. Henggeler SW, Schoenwald SK, Liao JG, Letourneau EJ, Edwards DL. Transporting efficacious treatments to field settings: the link between supervisory practices and therapist fidelity in MST programs. J Clin Child Adolesc Psychol. 2002;31:155–67. [PubMed]
81. Carroll KM, Nich C, Sifry R, Nuro KF, Frankforter T, Ball SA, et al. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug Alcohol Depend. 2000;57:225–38. [PubMed]
82. Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. J Consult Clin Psychol. 1993;61 :620–30. [PubMed]
83. Calhoun KS, Moras K, Pilkonis PA, Rehm LP. Empirically supported treatments: implications for training. J Consult Clin Psychol. 1998;66:151–62. [PubMed]
84. McLellan AT, Carise D, Kleber HD. Can the national addiction treatment infrastructure support the public’s demand for quality care? J Subst Abuse Treat. 2003;25:117–21. [PubMed]
85. McLellan AT. What we need is a system: creating a responsive and effective substance abuse treatment system. In: Miller WR, Carroll KM, editors. Rethinking substance abuse: what the science shows and what we should do about it. New York: Guilford Press; 2006. pp. 275–92.
86. Olmstead TA, Sindelar JL, Petry NM. Cost-effectiveness of prize-based incentives for stimulant abusers in outpatient psychosocial treatment programs. Drug Alcohol Depend. 2007;87:175–82. [PMC free article] [PubMed]
87. Chiles JA, Lambert ME, Hatch AL. The impact of psychological interventions on medical cost offset: a meta-analytic review. Clin Psychol Sci Pract. 1999;6:204–20.
88. Yates BT. Toward the incorporation of costs, cost-effectiveness analysis, and cost–benefit analysis into clinical research. J Consult Clin Psychol. 1994;62:729–36. [PubMed]
89. Addis ME, Krasnow AD. A national survey of practicing psychologists’ attitudes toward psychotherapy treatment manuals. J Consult Clin Psychol. 2000;68:430–41. [PubMed]
90. Prochaska JO, DiClemente CC. Transtheoretical therapy: toward a more integrative model of change. Psychother Theory Res Pract. 1982;19:276–88.
91. Beutler LE, Moleiro C, Talebi H. How practitioners can systematically use empirical evidence in treatment selection. J Clin Psychol. 2002;58:1199–212. [PubMed]
92. Kazdin AE, Nock MK. Delineating mechanisms of change in child and adolescent therapy: methodological issues and research recommendations. J Child Psychol Psychiatry. 2003;44:1116–29. [PubMed]
93. O’Donohue W, Yeater EA. Individuating psychotherapies. Behav Modif. 2003;27:313–21. [PubMed]
94. Haaga DAF. Being systematic, and maybe even scientific, about planning psychotherapy research. Clin Psychol Sci Pract. 2004;11:387–0.
95. O’Donohue W, Buchanan JA, Fisher JE. Characteristics of empirically supported treatments. J Psychother Pract Res. 2000;9:69–74. [PMC free article] [PubMed]
96. Beutler LE, Clarkin JF, Bongar BM. Guidelines for the Systematic Treatment of Depressed Patients. New York: Oxford University Press; 2000.
97. Beutler LE, Harwood HJ. Prescriptive Psychotherapy: a Practical Guide to Systematic Treatment Selections. New York: Oxford University Press; 2000.
98. Ball SA, Bachrach K, DeCarlo J, Farentinos C, Keen M, McSherry T, et al. Characteristics of community clinicians trained to provide manual-guided therapy for substance abusers. J Subst Abuse Treat. 2002;23:309–18. [PubMed]
99. Berwick DM. Disseminating innovations in health care. JAMA. 2003;289:1969–75. [PubMed]
100. Wiltsey Stirman S, Crits-Christoph P, DeRubeis RJ. Achieving successful dissemination of empirically supported therapies: a synthesis of dissemination theory. Clin Psychol Sci Pract. 2004;11:343–59.
101. Petry NM. A comprehensive guide to the application of contigency management procedures in clinical settings. Drug Alcohol Depend. 2000;58:9–25. [PubMed]
102. Kirby KC, Marlowe DB, Festinger DS, Lamb RJ, Platt JJ. Schedule of voucher delivery influences initiation of cocaine abstinence. J Consult Clin Psychol. 1998;66:761–7. [PubMed]
103. DeRubeis RJ, Crits-Christoph P. Empirically supported individual and group psychological treatments for adult mental disorders. J Consult Clin Psychol. 1998;66:37–52. [PubMed]
104. Weissman MM, Markowitz JC. Interpersonal psychotherapy. current status. Arch Gen Psychiatry. 1994;51:599–606. [PubMed]
105. Rubak S, Sandbaek A, Lauritzen T, Christensen B. Motivational interviewing: a systematic review and meta-analysis. Br J Gen Pract. 2005;55:305–12. [PMC free article] [PubMed]
106. Burke BL, Arkowitz H, Menchola M. The efficacy of motivational interviewing: a meta-analysis of controlled clinical trials. J Consult Clin Psychol. 2003;71:843–61. [PubMed]
107. Henggeler SW, Melton GB, Brondino MJ, Scherer DG, Hanley JH. Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity. J Consult Clin Psychol. 1997;65:821–33. [PubMed]
108. Foley SH, O’Malley SS, Rounsaville BJ. The relationship between patient difficulty and therapist performance in interpersonal psychotherapy. J Affect Disord. 1987;12:207–17. [PubMed]
109. Barber JG, Crits-Christoph P, Luborsky L. Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. J Consult Clin Psychol. 1996;64:619–22. [PubMed]
110. Henry WP, Strupp HH, Butler SF, Schacht TE, Binder JL. Effects of training in time-limited dynamic psychotherapy: changes in therapist behavior. J Consult Clin Psychol. 1993;61:434–40. [PubMed]
111. Castonguay LG, Goldfried MR, Wiser S, Raue PJ, Hayes AM. Predicting the effect of cognitive therapy for depression: a study of unique and common factors. J Consult Clin Psychol. 1996;64:497–504. [PubMed]
112. Sholomskas DE, Carroll KM. One small step for manuals: computer-assisted training in twelve-step facilitation. J Stud Alcohol. 2006;67:939–45. [PMC free article] [PubMed]
113. Issenberg SB, McGaghie WC, Hart IR, Mayer JW, Felner JM, Petrusa ER, et al. Simulation technology for health care professional skills training and assessment. JAMA. 1999;282:861–6. [PubMed]
114. Todd KH, Braslow A, Brennan RT, Lowery DW, Cox RJ, Lipscomb LE, et al. Randomized, controlled trial of video self-instruction versus traditional CPR training. Ann Emerg Med. 1998;31:364–9. [PubMed]
115. Huang MP, Alessi NE. The internet and the future of psychiatry. Am J Psychiatry. 1996;153:861–9. [PubMed]
116. Piemme TE. Computer-assisted learning and evaluation in medicine. JAMA. 1988;260:367–72. [PubMed]
117. Jones I, Cookson J. Computer-assisted learning design for reflective practice supporting multiple learning styles for education and training in pre-hospital emergency care. Int J Train Dev. 2001;5:74–80.
118. Budman SH. Computer-mediated addiction services: tomorrow won’t look like today. Behav Healthcare Tomorrow. 2002;11:14–21. [PubMed]
119. Davis DD, Thompson O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education. JAMA. 1999;282:867–74. [PubMed]
120. Williams CD, Aubin S, Harkin P, Cottrell D. A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ. 2001;35:847–54. [PubMed]
121. Carise D, McLellan AT, Gifford LS, Kleber HD. 1999 Developing a national addiction treatment information system: an introduction to the Drug Evaluation Network System. J Subst Abuse Treat. 2002;17:67–77. [PubMed]
122. McLellan AT, Belding MA, McKay JR, Zanis D, Alterman AI. Can the outcomes research literature inform the search for quality indicators in substance abuse treatment? In: Edmunds M, Frank RG, Hogan M, McCarty D, Robinson-Beale R, Weisner C, editors. Managing managed care: quality improvement in behavioral health. Washington, DC: National Academy Press; 1996.
123. Keane DR, Norman GR, Vickers J. The inadequacy of recent research on computer-assisted instruction. Acad Med. 1991;66:44–8. [PubMed]
124. Clark RE. Dangers in evaluation of instructional media. Acad Med. 1992;12:819–20. [PubMed]
125. Vogel M, Wood DF. Love it or hate it? Medical students’ attitudes to computer-assisted learning. Med Educ. 2002;36 :214–15. [PubMed]
126. Cook DA. Learning and cognitive styles in web-based learning: theory, evidence, and application. Acad Med. 2005;80:266–78. [PubMed]
127. Lane JM, Addis ME. Pros and cons of educational technology as methods for disseminating evidence-based treatments. Clin Psychol Sci Pract. 2004;11:336–8.
128. Crits-Christoph P, Baranackie K, Kurcias J, Beck AT, Carroll KM, Perry K, et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res. 1991;1:81–91.
129. Carroll KM. Constrained, confounded, and confused: why we know so little about therapist effects. Addiction. 2001;96 :203–6. [PubMed]
130. Project MATCH Research Group. Therapist effects in three treatments for alcohol problems. Psychother Res. 1998;8:455–74.
131. Budman SH. Behavioral health care dot-com and beyond: computer-mediated communications in mental health and substance abuse treatment. Am Psychol. 2000;55:1290–300. [PubMed]
132. Marks I, Shaw S, Parkin R. Computer-aided treatments of mental health problems. Clin Psychol Sci Pract. 1998;5:151–70.
133. Norquist G, Regier DA. The epidemiology of psychiatric disorders and the de facto mental health care system. Annu Rev Med. 1996;47:473–9. [PubMed]
134. Kirkby KC. Computer-assisted treatment of phobias. Psychiatr Serv. 1996;47:139–40. [PubMed]
135. Baer L, Greist JH. An interactive computer-administered self-assessment and self-help program for behavior therapy. J Clin Psychiatry. 1997;58:23–8. [PubMed]
136. Newman MG, Kenardy J, Herman S, Taylor CB. Comparison of palmtop-computer assisted brief cognitive-behavioral treatment to cognitive-behavioral treatment for panic disorder. J Consult Clin Psychol. 1997;65:178–83. [PubMed]
137. Wright JH, Wright AS, Albano AM, Basco MR, Goldsmith LJ, Raffield T, et al. Computer-assisted cognitive therapy for depression: maintaining efficacy while reducing therapist time. Am J Psychiatry. 2005;162:1158–64. [PubMed]
138. Selmi PM, Klein MH, Greist JH, Sorrell SP, Erdman HP. Computer-administered CBT for depression. Am J Psychiatry. 1990;147:51–6. [PubMed]
139. Slate SE, Meyer TL, Burns WJ, Montgomery DD. Computerized cognitive training for severely emotionally disturbed children with ADHD. Behav Modif. 1998;22:415–20. [PubMed]
140. Shiffman S, Paty JA, Rohay JM, DiMarino ME, Gitchell JG. The efficacy of computer-tailored smoking cessation material as a supplement to nicotine patch therapy. Drug Alcohol Depend. 2001;64:35–46. [PubMed]
141. Carroll KM. A Cognitive-Behavioral Approach: Treating Cocaine Addiction. Rockville, MD: NIDA;
142. Paul GL. 1967 Strategy of outcome research in psychotherapy. J Clin Psychol. 1998;31:109–18. [PubMed]
143. Paul GL, Licht MH. Resurrection of the uniformity assumption myth and the fallacy of statistical absolutes in psychotherapy research. J Consult Clin Psychol. 1978;46:1531–4. [PubMed]