Training Primary Care Staff to Deliver a Computer-Assisted Cognitive-Behavioral Therapy Program for Anxiety Disorders

doi:10.1016/j.genhosppsych.2011.04.011

Gen Hosp Psychiatry. Author manuscript; available in PMC 2012 July 1.

Published in final edited form as:

Gen Hosp Psychiatry. 2011 Jul-Aug; 33(4): 336–342.

Published online 2011 June 8. doi: 10.1016/j.genhosppsych.2011.04.011

PMCID: PMC3139130

NIHMSID: NIHMS294602

Training Primary Care Staff to Deliver a Computer-Assisted Cognitive-Behavioral Therapy Program for Anxiety Disorders

Raphael D. Rose, Ph.D.,¹ Ariel J. Lang, Ph.D., M.P.H,^2,⁸ Stacy Shaw Welch, Ph.D.,³ Laura Campbell-Sills, Ph.D.,² Denise A. Chavira, Ph.D.,² Greer Sullivan, M.D.,⁴ Cathy Sherbourne, Ph.D.,⁵ Alexander Bystritsky, M.D.,⁶ Murray B. Stein, M.D., M.P.H.,⁷ Peter P. Roy-Byrne, M.D.,³ and Michelle G. Craske, Ph.D.^1,⁶

¹University of California, Los Angeles, Department of Psychology

²University of California, San Diego, Department of Psychiatry

³University of Washington at Harborview Medical Center and CHAMMP

⁴University of Arkansas for Medical Sciences

⁵The RAND Corporation

⁶University of California, Los Angeles, Department of Psychiatry and Biobehavioral Sciences

⁷University of California, San Diego, Department of Psychiatry and Department of Family & Preventive Medicine

⁸VA San Diego Healthcare System

Corresponding Author: Raphael D. Rose, Ph.D., Associate Research Psychologist, UCLA - Department of Psychology, 1285 Franz Hall - Box 951563, Los Angeles, CA 90095-1563, Phone/fax: 310-825-9048, Email: rose/at/psych.ucla.edu

The publisher's final edited version of this article is available at Gen Hosp Psychiatry

See other articles in PMC that cite the published article.

Publisher's Disclaimer

Abstract

Objectives

This paper describes the training model used with primary care staff to deliver an evidence-based computer-assisted cognitive-behavioral therapy (CBT) program for anxiety disorders within a collaborative care treatment delivery model.

Methods

We describe the training and proficiency evaluation procedures utilized in the CALM study, a large multi-site study of collaborative care for anxiety disorders in primary care. Training incorporated readings, didactic presentations, video demonstrations of CBT skills, role-plays, computer-assisted practice, CBT training cases, and ongoing group supervision provided by study psychologists.

Results

Proficiency training case data from 15 clinicians are presented. The anxiety clinical specialists (ACS) were highly proficient at delivering the CBT component of the CALM intervention. ACS also provided Likert-scale ratings and open-ended responses about their experiences with the training. Overall, the training was rated very positively, and was described as very thorough, indicating a high level of acceptability to clinicians. Recommendations for future training are described.

Conclusions

Primary care staff with none or minimal prior CBT experience can be trained to deliver a computer-assisted, evidence-based treatment for anxiety disorders. The implications for dissemination and transportability of evidenced-based interventions are discussed.

Keywords: Primary Care, Cognitive-behavioral therapy (CBT), anxiety disorders, training

There is a substantial need for treatments for anxiety disorders in primary care that are both effective and easy to disseminate. Anxiety disorders are the most common mental health problem seen in primary care, outnumbering even depressive disorders [1]. Additionally, anxiety is poorly recognized by Primary Care Physicians (PCPs) and when recognized, evidenced-based treatments such as cognitive behavioral therapy (CBT) are used infrequently and anxiolytic medications – though more common- are often used suboptimally [2-3]. When provided, treatment is often marked by poor adherence, which in turn leads to poor outcomes [4-5].

Prior effectiveness studies of primary care-based interventions for anxiety and depression [2, 6-10] have demonstrated superiority for both psychosocial and pharmacologic interventions compared to treatment as usual (TAU). However, these interventions are difficult to sustain and disseminate because of the heavy cost burden of implementation, particularly the costs of both initial training and ongoing support of external personnel. Therefore, a major component of our CALM (Coordinated Anxiety Learning and Management) study design was to create both a disseminable and effective anxiety disorder intervention for primary care [11]. A key dissemination component of our study was to train clinic staff (i.e., nurses, social workers), who work in primary care but who had little or no experience in evidence-based anxiety interventions to deliver CBT in an adherent and competent way.

CALM Study

CALM was a multi-site (Los Angeles, San Diego, Seattle, Little Rock) collaborative later-stage effectiveness study funded by the National Institute of Mental Health (NIMH). The overall aim was to test a novel model of service delivery for the four most prevalent anxiety disorders in primary care: panic disorder, generalized anxiety disorder, social anxiety disorder, posttraumatic stress disorder. The focus was on the delivery of evidence-based pharmacologic and cognitive-behavioral therapy (CBT) treatments embedded within a novel service delivery model [3, 11]. A major component of the CALM design was the use of a computer-assisted CBT program that facilitated the delivery of CBT by novice clinicians. This paper describes the training model used to prepare and then sustain novice clinicians in the delivery of CBT for multiple anxiety disorders.

Computer–Assisted Intervention

We created a computer-assisted CBT program called Calm Tools For Living. The program included standard, empirically-supported CBT strategies for anxiety disorders including self monitoring, psychoeducation, diaphragmatic breathing, cognitive restructuring, exposure to internal and external anxiety provoking stimuli, and relapse prevention. It contained both a unified treatment approach for all four targeted anxiety disorders as well as branching mechanisms to address unique aspects of each disorder [12].

Essentially both the anxiety clinical specialists (ACS) and the computer-based program combine to deliver the treatment to the patient with the ACS being guided by prompts in the program to demonstrate certain CBT skills. In this method of treatment delivery, the program serves as a both a resource of information and prompts for the ACS to review and/or demonstrate certain CBT techniques to the client. For example, in the diaphragmatic breathing (CALM Breathing) exercise, the computer provides background on the rationale for this skill (i.e., to help regulate physiology), which is reviewed by the ACS and/or read by the patient. The program also prompts the ACS to teach the skill to the client, provides step-by-step instructions on how to do it, and presents visual aids in the form of diagrams or figures. Additionally, the program contains videotaped demonstrations, created by study researchers that model the skill, so that the ACS and patient can watch them during a session.

This program was introduced via an initial didactic training and then reinforced with intermittent supervision by an expert psychologist. At the same time, the program itself served a vital training function since its ongoing use iteratively enhanced therapist competence and adherence to CBT with each patient treated (i.e. by following the structured treatment protocol within the program, therapists were more likely to adhere to the specific skills and activities of CBT and to deliver them in a competent manner, although the initial training also enhanced therapist understanding of CBT principles which also likely enhanced competence).

Dissemination and sustainability

The aims of the CALM training and treatment program were to be 1) brief and low in complexity (or breakable into steps), 2) usable by primary care clinicians who had little/no prior training in CBT or mental health in general 3) user-friendly across a variety of staff/disciplines (e.g., nurses, social workers), 4) applicable to the four targeted anxiety disorders, and 5) flexible enough to fit into existing clinic culture and routines.

Training Approaches

Training clinicians for efficacy trials using manualized treatment approaches typically follows an intensive formula that is difficult to replicate in primary care settings. Therapists who are experienced and committed to the treatment modality are usually recruited. Trainings are typically conducted through didactic presentations with extensive role-playing and practice of therapy skills, and completion of at least one supervised training case that demonstrates proficiency. Application of these training methods for “real-world” (effectiveness) research has rarely been attempted previously. Instead, studies examining training of clinicians have relied on dissemination of manuals [13-14].

There are relatively few studies addressing the means for effectively training primary care staff to deliver evidenced-based interventions [15-19] and to our knowledge only one study specifically addressed training primary care staff (i.e., general practitioners) to deliver evidence-based anxiety interventions [20]. Only two studies have compared different training strategies (e.g., computer-based training, didactics, manuals, or role-plays) with training real-world clinicians in evidence-based interventions [13, 21]. There is limited evidence to suggest that novice primary care staff or paraprofessionals can be effectively trained in evidence-based anxiety interventions in a widely scalable model that most primary care settings will likely adapt [22]. The computer-assisted CBT intervention in CALM may help address some of the challenges of training novice primary care staff to deliver evidenced-based interventions by providing the guidance and structure to do so. This may be especially relevant for training masters level clinicians (e.g. RNs, MSW, MFC, MA) as opposed to PhDs, particularly since they are less costly to hire and there are many more masters than PhD clinicians available in the US.

Methods

Subjects

Subjects were 15 primary care staff hired to work either part or full time as anxiety clinical specialists as part of the CALM study at one of the 13 primary care clinics among the four study sites; Los Angeles, San Diego, Seattle, and Little Rock, AK. As Table 1 shows, the clinicians who took part in this study, Anxiety Clinical Specialists (ACS), were nearly all Master's level clinicians, female, licensed, and experienced in working with a clinical population that includes medical as well as emotional problems. However, the ACS had relatively little psychotherapy training, and hardly any prior CBT training. ACS with these backgrounds were purposely selected because we thought it would be representative of the clinical staff who would actually be trained to do this kind of intervention in real world settings.

Table 1

ACS Demographics Information

Procedures

CALM CBT training was coordinated by the UCLA site, and training was conducted locally at each site by expert study psychologists and psychiatrists, with the exception of UAMS, where ACS were trained directly by UCLA personnel. Trainers were expert in CBT approaches for anxiety disorders. Training included readings, didactic seminar presentations, quizzes, videotape presentations, role-plays, assessment and therapy training cases, and ongoing in-person and telephone supervision [13]. The main focus of training was on active practice and rehearsal of CBT skills with corrective feedback from supervisors. Training was spaced out over 10 weeks with weekly or biweekly half day to full day didactic seminar presentations followed by skills practice and role-plays. This schedule was designed to allow time in between trainings for rehearsal of skills and accommodate scheduling needs of some of the clinics involved in the study.

Assigned readings included the CALM study treatment procedures and selected readings on CBT treatment of anxiety disorders. There were five half day didactic presentations by trainers covering the CALM study approach, an overview of CBT in general, CBT skills for each of the four targeted anxiety disorders, and basic information about each disorder (i.e., generalized anxiety disorder, panic disorder, social anxiety disorder, and posttraumatic stress disorder). An overview of motivational enhancement techniques, adapted for anxious patients, was also incorporated to address client motivation to engage in treatment, as most study participants in CALM were not actively seeking treatment for anxiety but rather were referred by their PCP. Cultural sensitivity training addressing issues related to minority participant recruitment, retention, and treatment was provided via videotaped presentation by an expert in this area. After each didactic presentation, ACS answered multiple-choice questions that were created by the trainers to assess ACS understanding of the assigned readings and didactic presentations. Corrective feedback was provided as needed.

ACS viewed video demonstrations that portrayed expert therapists (with mock patients) conducting core elements of the CBT treatment (i.e., psychoeducation, abdominal breathing, cognitive restructuring, hierarchy construction, and creating appropriate exposure exercises). These videos were incorporated into the actual CBT computer program so clients and ACS also could view them during the course of treatment. Additional videos of “bad” therapy techniques were made for training purposes only to show ACS what NOT to do (e.g., assigning an exposure exercise that was inappropriate or that encouraged using avoidance as a coping strategy).

ACS practiced CBT skills by role playing vignettes that described fictitious patient/therapist interactions. The vignettes were role-played in person or via telephone weekly for several weeks and were observed and rated for competence by study psychologists, who provided feedback and modeled desired skills.

ACS were trained on the computer-assisted CBT program utilizing a similar structure to the role-plays described above where they practiced the different modules of the program, alternated being patient/therapist and received feedback from study psychologists.

Lastly, ACS treated up to three training cases, each for 6 sessions delivered over 4-6 weeks. Training cases, met Diagnostic and Statistical Manual (4^th Edition) criteria for at least one of the four targeted anxiety disorders as determined by administration of the MINI International Neuropsychiatric Interview [23]. Anyone requiring additional treatment was referred out for further care. Expert psychologists evaluated a minimum of 3 sessions per training.

Once ACS were certified as proficient in the CALM CBT intervention, they received weekly supervision from study psychologists and psychiatrists on their caseload. Supervision typically lasted about 15-30 minutes per ACS and took place in a group format either via telephone or in person (whichever was more convenient). Group telephone supervision allows for the most efficient approach to supervision as it allows expert clinicians to not have to be on site and creates scheduling flexibility. CALM utilized an electronic web-based patient tracking system, available to all study personnel, which facilitated supervision by identifying which patients were not doing well and directing supervisor focus to these individuals [3, 24]. The web-based system provided up-to-date information on each patient enrolled in the study in terms of basic demographics, targeted anxiety disorder, duration of treatment, type of treatment (CBT or medication or both), number of sessions, progress notes, and weekly anxiety and depression symptom scores. This patient information was updated after each session by the ACS and helped supervisors formulate treatment recommendations.

Total training time to reach proficiency for ACS was approximately 45-50 hours. That was comprised of five half-day didactic presentations (20 hours), role-play practice in between presentations (5 hours), computer practice with the CALM CBT Program (8 hours), treating two or three training cases (approx 12 hours), and readings (3 hours). The duration of our training is slightly longer than that of a comparable training approach. [13]. They trained full-time community-based clinicians with a manual, didactic seminars, and supervised casework (Mean time = 33 hours) who had some familiarity with CBT. However, the practitioners in CALM were CBT novices in treating anxiety disorders in primary care, which led to the slightly longer training duration.

Measures

Quizzes

Assessment of ACS knowledge of readings and didactic presentations consisted of 57 multiple-choice questions divided by content area (e.g., psychoeducation, breathing retraining, cognitive restructuring etc.).

Role-Plays

Study psychologists evaluated ACS performance on role-plays of fictitious clinical vignettes created for the CALM CBT program. Role-plays assessed the degree to which ACS performed certain CBT skills (e.g., cognitive restructuring, assigning and reviewing exposure assignments). Ratings were based on a 7-point behaviorally anchored competence Likert scale (1 = Not at all; 4 = Somewhat; 7 = Full).

Training Case Proficiency

Expert evaluation of ACS performance consisted of review of audiotapes of the training sessions using a proficiency rating system based on the Yale Adherence Competence Scale (YACS) [25]. The proficiency rating system was comprised of a 7-point behaviorally anchored competency Likert scale (1 = low competency; 7 = high competency), which was used to rate therapy skills that were expected to be utilized in a particular therapy session (e.g., how well ACS educated their patient about anxiety disorders, or how well they instructed them to use CBT skills). It also contained a 7-point adherence scale used to rate how well the ACS adhered to the computer-assisted program (1 = low adherence; 7 = high adherence). Adequate proficiency was defined as an average score of 4 or higher collapsing across competency and adherence ratings for all sessions per training case. Once the ACS demonstrated proficiency on 2 training cases, he/she could begin seeing study participants. If an ACS was not deemed sufficiently proficient after 4 training cases, they would have been dropped from the study: This did not occur.

ACS Training Survey

ACS completed a survey about their training experiences midway through the study. Using a 1-7 point Likert scale where 1 was not at all useful and 7 was extremely useful, ACS rated how useful they found the different components of the training (i.e., didactic presentations, readings, role-plays, training cases, and ongoing supervision). ACS also were asked open-ended questions about what they liked most and least about the training, and for suggestions for improvements to the training program.

Results

Training Case Proficiency

As Table 2 shows, across all study sites the mean number of training cases was 2.13, SD 0.35. The mean proficiency score, based on 1-7 Likert scale across all ACS training cases, was 6.37, with a SD of .97.

Table 2

ACS Training Case Proficiency Outcome Data By License Type

Role-Play Competence

ACS mean score on role-plays demonstrating CBT skills based on expert psychologist ratings on a 1-7 Likert scale was 6.31 SD=2.21

Quizzes

The mean score on quizzes assessing ACS knowledge of readings and didactic presentations was 98%.

ACS Experience and Training Performance

Pearson product moment correlations were conducted to examine the relationships between ACS experience and training, and their performance on training cases. Previous years of clinical experience (which included any type of clinical work) was negatively correlated with ACS performance on training cases as rated by study psychologists; r = -.70, (p < .01). There was no significant relationship between ACS educational degree or type of license and ratings of their training cases.

ACS Training Survey

As Table 3 shows, ACS rated the training quite favorably

Table 3

ACS Training Usefulness Survey

ACS rated all of the training components as highly useful. The cross-cultural presentation was rated lowest which may stem from the fact that it was the only presentation conducted via “videotape” at all sites except for one. All other presentations were conducted live at each site by study personnel. They also rated the length and amount of information presented as appropriate. An examination of their open-ended responses indicated that ACS found that the in person and hands on training they received from their clinical supervisors was extremely helpful, as was weekly supervision, and the overall structure of the training. Some ACS thought the role-plays were very helpful, while others found the role-plays to be somewhat time-consuming and challenging. The biggest criticism ACS had regarding the training was its length; they thought that the training took too long and that it could have been made more compact by being less repetitive and decreasing the time between presentations and practice sessions.

Discussion and Conclusions

Although a number of evidenced-based psychotherapies, including CBT, are known to be clinically effective, many do not have access to them in part because widespread dissemination and implementation of these practices has been challenging. The goal of our work was to develop and evaluate a system for training primary care staff to deliver evidenced-based treatment for anxiety disorders. Such a system would provide an important mechanism for dissemination of evidence-based care for anxiety disorders. An additional aspect of the CALM study is that training occurred at four geographic sites covering a wide and diverse swatch of the US. Practitioners were located at 13 primary care clinics among these sites and the training and evaluation of the training was managed out of the UCLA (Los Angeles) site. Therefore, this training or a substantial part of it can be done at a distance. Based on study psychologists' ratings of the training cases, our training resulted in ACS who were competent at delivering CBT skills and adherent to the computer-assisted CBT program.

The training was generally viewed very positively and as being very useful, with the more active components of training (i.e., role-plays, and ongoing supervision) rated as most helpful. ACS's previous clinical experience was negatively associated with how well they performed on their training cases as rated by study psychologists. Perhaps ACS with greater clinical experience were more set in their ways and took a bit longer to adapt to the structure of the computer-assisted CBT intervention. It should be noted that despite the significant inverse relationship between prior clinical experience and proficiency ratings, all ACS were rated proficient on their training cases with the average proficiency rating being greater than 6 out of 7 points.

As [19] reported, training in evidenced-based interventions in psychology, psychiatry, and social work programs is lacking. Behavioral health professionals need access to training that is practical and applicable to real-world settings. The CALM training program addressed that void in evidenced-based training and successfully trained clinicians in the principles of CBT, not just in the use of a manual. Additionally, outcome data for the 503 subjects randomized to the CALM intervention shows that they did significantly better than those randomized to TAU (N=501). Furthermore 91% of those in the CALM intervention received CBT [3]. This suggests that our training led to effective outcomes.

There are some limitations to the data presented here. Bias may have been introduced because the ACS may have not felt fully comfortable to rate the training negatively given their close interactions with the study psychologists. While the study psychologists who rated ACS performance on role-plays and training cases specialized in CBT treatment for anxiety disorders, an inter-rater reliability check was not conducted. Although the CALM study aimed to address issues of increasing dissemination and transportability of the intervention, there are no data yet to support whether it will accomplish that.

The ACS survey suggests areas for further refinement of the CALM training program. Based on ACS feedback, the training could have been streamlined in terms of duration and content. While the training model presented here was effective even with novice CBT clinicians, the hours of training provided in the study might be impractical in actual primary care settings. Future studies should empirically evaluate how much and what kind of training is essential to adequately train novice clinicians in CBT for use in real-world settings such as primary care. Another possible direction for dissemination of training in evidence-based interventions would be to create an autonomous version of self-guided CBT training available via the Internet.

In summary, clinicians without prior experience in CBT can be trained to deliver a computer-assisted, evidence-based treatment for anxiety disorders in primary care settings. A diverse training program blending didactic presentations, role-playing, video demonstrations of CBT skills, and closely monitored training cases was very acceptable to the clinicians-in-training and led to high levels of proficiency in their intervention delivery. An important future direction involves streamlining the CALM training program with the aim of decreasing overall training hours and thereby reducing the time and financial costs of training to trainees, supervisors, and health care settings.

Acknowledgments

This work was supported by the following NIMH grants: U01 MH070018 to RAND (PI: Cathy Sherbourne), U01 MH058915 to UCLA (PI: Michelle Craske), U01 MH057835 and K24 MH64122 to UCSD (PI: Murray Stein), UO1 MH057858 and K24 MH065324 to University of Washington (PI: Peter Roy-Byrne), U01-MH070022 to UAMS (PI: Greer Sullivan). The authors would like to thank Daniel E. Glenn and Tomislav D. Zbozinek for their contributions to this manuscript.

Footnotes

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

1. Kroenke K, Spitzer RL, Williams JB, Monahan PO, Lowe B. Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Annals of Internal Medicine. 2007;146:317–325. [PubMed]

2. Roy-Byrne PP, Craske MG, Stein MB, Greer S, Bystritsky A, Wayne K, et al. A randomized effectiveness trial of cognitive-behavioral therapy and medication for primary care panic disorder. Archives of General Psychiatry. 2005;62:290–298. [PMC free article] [PubMed]

3. Roy-Byrne P, Craske M, Sullivan G, Rose R, Edlund M, Lang A, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. Journal of the American Medical Association. 2010;303:1921–1928. [PMC free article] [PubMed]

4. Stein MB, Roy-Byrne PP, Craske MG, Campbell-Sills L, Lang AJ, Golinelli D, et al. Quality of and patient satisfaction with primary health care for anxiety disorders. Journal of Clinical Psychiatry. in press.

5. Pini S, Perkonnig A, Tansella M, Wittchen HU. Prevalence and 12-month outcome of threshold and subthreshold mental disorders in primary care. Journal of Affective Disorders. 1999;56:37–48. [PubMed]

6. Wells K, Sherbourne C, Duan N, Unützer J, Miranda J, Schoenbaum M, et al. Quality improvement for depression in primary care: do patients with subthreshold depression benefit in the long run? The American Journal of Psychiatry. 2005;16:1149–1157. [PubMed]

7. Wolf N, Hopko D. Psychosocial and pharmacological interventions for depressed adults in primary care: a critical review. Clinical Psychology Review. 2008;28:131–161. [PubMed]

8. Unützer J, Katon WJ, Callahan CM, Williams JW, Hunkeler E, Harpole L, et al. Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. Journal of the American Medical Association. 2002;288(22):2836–2845. [PubMed]

9. Katon W, Van Korff M, Lin E, Simon G, Walker E, Unützer J, et al. Stepped collaborative care for primary care patients with persistent symptoms of depression: a randomized trial. Archives of General Psychiatry. 1999;56:1109–1115. [PubMed]

10. Mynors-Wallis LM, Gath GH, Day A, Baker F. Randomized controlled trial of problem solving, antidepressant medication, and combined treatment for major depression in primary care. British Medical Journal. 2000;320:2630. [PMC free article] [PubMed]

11. Sullivan G, Craske MG, Sherbourne C, Edlund M, Rose R, Golinelli D, et al. Design of the Coordinated Anxiety Learning and Management study: innovations in collaborative care for anxiety disorders. General Hospital Psychiatry. 2007;29:379–387. [PMC free article] [PubMed]

12. Craske M, Rose R, Lang A, Welch S, Campbell-Sills L, Sullivan G, Sherbourne C, et al. Computer-assisted delivery of cognitive behavioral therapy for anxiety disorders in primary-care settings. Depression and Anxiety. 2009;26:235–242. [PMC free article] [PubMed]

13. Sholomskas D, Syracuse-Siewert G, Rounsaville B, Ball S, Nuro K, Carroll K. We don't train in vain: a dissemination trail of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology. 2005;73:106–115. [PMC free article] [PubMed]

14. Morgenstern J, Morgan T, McCrady B, Keller D, Carroll K. Manual-guided cognitive-behavioral therapy training: a promising method for disseminating empirically supported substance abuse treatments to the practice community. Psychology of Addictive Behaviors. 2001;15:83–88. [PubMed]

15. Hegel M, Barrett J, Oxman T. Training therapists in problem-solving treatment of depressive disorders in primary care: lessons learned from the “Treatment Effectiveness Project” Family, Systems, & Health. 2000;18:423–435.

16. Tiemens BG, Ormel J, Jenner JA, Van Der Meer K, Van Os TWDP, Van Den Brink RHS, et al. Training primary-care physicians to recognize, diagnose and manage depression: does it improve patient outcomes? Psychological Medicine. 1999;29:833–845. [PubMed]

17. Oishi S, Shoai R, Katon W, Callahan C, Unützer J. the IMPACT Investigators. Impacting late life depression: integrating a depression intervention into primary care. Psychiatric Quarterly. 2003;74:75–89. [PubMed]

18. King M, Davidson O, Taylor F, Haines A, Sharp D, Turner R. Effectiveness of teaching general practitioners skills in brief cognitive behavior therapy to treat patients with depression: randomised controlled trial. British Medical Journal. 2002;324:947. [PMC free article] [PubMed]

19. Weissman M, Verdeli H, Gameroff M, Bledsoe S, Betts K, Mufson L, et al. National survey of psychotherapy training in psychiatry, psychology, and social work. Archives of General Psychiatry. 2006;63:925–934. [PubMed]

20. Heatley C, Ricketts T, Forrest J. Training general practitioners in cognitive behavioural therapy for panic disorder: Randomized-controlled trial. Journal of Mental Health. 2005;14:73–82.

21. Sholomskas D, Carroll K. One small step for manuals: computer-assisted training in twelve-step facilitation. Journal of Studies on Alcohol. 2006;67:939–945. [PMC free article] [PubMed]

22. Montgomery E, Kunik M, Wilson N, Stanley M, Weiss B. Can paraprofessionals deliver cognitive-behavioral therapy to treat anxiety and depressive symptoms? Bulletin of the Menninger Clinic. 2010;74:45–62. [PubMed]

23. Sheehan DV, Lecrubier Y, Sheehan KH, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59(suppl 20):22–33. quiz 34-57. [PubMed]

24. Unützer J, Choi Y, Cook I, Oishi S. A web-based data management system to improve care for depression in a multicenter clinical trial. Psychiatric Services. 2002;53:671–673. 678. [PubMed]

25. Carroll K, Nich C, Sifry R, Nuro K, Frankforter T, Ball S, et al. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence. 2000;57:225–238. [PubMed]