Evaluation of the Commitment to Living (CTL) Curriculum: A 3-Hour Training for Mental Health Professionals to Address Suicide Risk

doi:10.1027/0227-5910/a000099

Crisis. Author manuscript; available in PMC 2012 December 22.

Published in final edited form as:

Crisis. 2012 January 1; 33(1): 30–38.

doi: 10.1027/0227-5910/a000099

PMCID: PMC3529124

NIHMSID: NIHMS426860

Evaluation of the Commitment to Living (CTL) Curriculum

A 3-Hour Training for Mental Health Professionals to Address Suicide Risk

Anthony R. Pisani, Wendi F. Cross, Arthur Watts, and Kenneth Conner

University of Rochester Medical Center, Rochester, NY, USA

Anthony R. Pisani University of Rochester Medical Center 301 Crittenden Blvd., Box PSYC Rochester, NY 14642 USA Tel. +1 585 275-3644 Fax +1 585 276-2065 ; Email: anthony_pisani/at/urmc.rochester.edu

The publisher's final edited version of this article is available at Crisis

Abstract

Background

Finding effective and efficient options for training mental health professionals to assess and manage suicide risk is a high priority.

Aims

To test whether an innovative, brief workshop can improve provider knowledge, confidence, and written risk assessment in a multidisciplinary sample of ambulatory and acute services professionals and trainees.

Methods

We conducted a pre/post evaluation of a 3 h workshop designed to improve clinical competence in suicide risk assessment by using visual concept mapping, medical records documentation, and site-specific crisis response options. Participants (N = 338 diverse mental health professionals) completed pre- and postworkshop questionnaires measuring their knowledge and confidence. Before and after the workshop, participants completed documentation for a clinical vignette. Trained coders rated the quality of risk assessment formulation before and after training.

Results

Participants' knowledge, confidence, and objectively-rated documentation skills improved significantly (p < .001), with large effect sizes. Participants' expectation of their ability to transfer workshop content to their clinical practice was high (mean = 4.10 on 1–5 scale).

Conclusions

Commitment to Living is a promising, innovative, and efficient curriculum for educating practicing clinicians to assess and respond to suicide risk. Well-designed, brief, suicide risk management programs can improve clinicians' knowledge, confidence, and skill.

Keywords: suicide, education, assessment, workshop, documentation

Introduction

Mental health professionals have a critical role to play in preventing suicide because they treat high risk patients. Suicidal symptoms and suicidal behavior are common among patients in mental health service settings (Brown, Beck, Steer, & Grisham, 2000; Harris & Barraclough, 1997), and many individuals who die by suicide have had recent contact with a mental health professional (Hunt et al., 2006; Luoma, Martin, & Pearson, 2002). To avoid preventable deaths, mental health professionals must competently identify, assess, and treat vulnerable patients. Mental health services rely heavily on continuing education workshops to equip staff with knowledge and skill in best practices. Integrating education into the demanding schedules of clinicians can be challenging because of the time commitment that is often required to achieve meaningful educational outcomes.

Education in the management of suicide risk can improve knowledge, skills, attitudes, and confidence among practicing clinicians, however the impact of suicide risk management workshops on mental health practice is largely unknown (Pisani, Cross, & Gould, 2011). Air Force clinicians' self-reported confidence, beliefs, and reported practice behaviors improved immediately and remained so six months after 12 h of education and exposure to a comprehensive practice guide on assessing and managing suicidal behavior (Oordt et al., 2009). Similarly, psychiatry residents who participated in 5 h of concentrated risk assessment training demonstrated improvements in the quality of their written assessments immediately following the training session, and these improvements were greater for a trained group than for a comparison group (McNiel et al., 2008). The attitudes and confidence of mental health staff who participated in an 8-h suicide management workshop improved over baseline immediately after training. This improvement was sustained among a subset of subjects who were surveyed four months afterward. However, improvement in interview skills measured via 15-minute role plays among 17 participants was not sustained after four months (Gask, Dixon, Morriss, Appleby, & Green, 2006). Fenwick et al. conducted a comparison between a 6-h suicide assessment workshop that had multiple active learning components and a 3-h program consisting mostly of lecture and paired participant discussion. Participants in both groups gained confidence from the trainings, and gains were sustained at 2-month follow-up (Fenwick, Vassilas, Carter, & Haque, 2004). Thus, a small body of research suggests that in-service education of 5 to 12 h duration may help clinicians better assess suicide risk among patients as measured by clinicians' self-reported attitudes and confidence.

To advance the limited literature on evidence-based education that addresses suicide risk, we conducted a pre-test/posttest evaluation of a workshop designed to build competence of a wide range of clinicians using a brief (3 h) training. This study tests whether the improvements in provider knowledge, confidence, and written risk assessment skill that have been observed in previous studies of longer training programs can be achieved through a brief workshop for a multidisciplinary sample of ambulatory and acute services professionals and trainees. This study is one of the first to objectively measure skill in written risk formulation and the first to examine potential differences in how ambulatory and acute services clinicians respond to suicide risk education.

For outcome measures, we examined clinicians' self-reported confidence and their knowledge as demonstrated on a multiple choice test. Clinician self-rating and knowledge scores are two important ways of measuring education effectiveness and allow comparisons with the few similar, existing training studies. As in the recent McNiel study (McNiel et al., 2008), we conducted objective ratings of clinician documentation to supplement and validate the self-reported outcome measures. Documentation is often the most tangible and legally relevant evidence of clinician behavior. Finally, we measured clinicians' assessment of how well the content they learned could be used in their practice settings. Thus, our specific study goals were to evaluate the following: (1) the program's effectiveness at improving trainee knowledge and confidence, (2) the program's effectiveness at building skill in risk assessment documentation, and (3) participants' perception of how the program's content can be used in their work settings.

Methods

Participants

Participants were 338 (266 women, 68 men, 4 unreported) staff members and trainees in the Department of Psychiatry at the University of Rochester. Participation in the workshop was mandatory for all clinical staff in the department; thus, the study encompassed the entire population of staff and trainee clinicians in this service system. Participant ages ranged from 21 to 75 years (mean = 42); 28% were trainees. Participants' primary clinical service settings were as follows: adult and child ambulatory services (outpatient and continuing day treatment, n = 192, 56.8%), adult and child acute services (inpatient, partial hospital, and psychiatric emergency departments, n = 113, 33.4%), and other services (consultation-liaison, psychological testing, mobile crisis and others, n = 33, 9.8%). Table 1 contains a summary of additional professional information about participants. This study received IRB approval under exempt status and did not require the use of informed consent.

Table 1

Summary of participant characteristics (N = 338)

Curriculum

Impetus for Development

The educational workshop was titled Commitment to Living: Understanding and Responding to Suicide Risk (CTL). One of the authors (ARP) developed CTL following the suicides of two patients in a clinic he directed. In the aftermath of these deaths, the demand for suicide assessment and management education across the system of care was high, with input from clinicians and administrators that a brief training was preferred in order to ensure that it could be rolled out quickly and with minimum disruption to clinical schedules.

Goals and Objectives

ARP developed CTL with the aim of creating a practical and authentic skill-building experience for clinicians with a wide range of experience and responsibilities. He drew on Bryan and Rudd's approach to assessment (Bryan & Rudd, 2006), his own training and experience as a clinician and educator, and the input of clinicians and suicide experts in the Department of Psychiatry. A key goal was to unite clinicians across service areas with a shared training experience and a common framework for assessment that could serve as a basis for ongoing educational activities. The practice recommendations that CTL offers are consistent with the American Psychiatric Association's clinical guidelines (American Psychiatric Association, 2003) and it targets 9 of the 24 core competencies for mental health professionals identified by the American Association of Suicidology and the Suicide Prevention Resource Center (Suicide Prevention Resource Center, 2006). See Table 2 for details.

Table 2

AAS/SPRC core competencies targeted by the CTL educational workshop

Learning Activities and Instructional Strategies

CTL is a 3-h workshop with two approximately 85-minute modules (Assessing Suicide Risk and Responding to Suicide Risk) and a 10-minute break in between. For each of the modules, participants experience lecture (20 min) interspersed with case-based discussions (45 min) and a documentation exercise (20 min).

CLT uses three innovative strategies for teaching suicide risk assessment and response. First, the content is delivered through visual concept mapping, an educational technique based on constructivist learning theory (Mintzes, Wandersee, & Novak, 1998; Novak, 1995) that has been tested elsewhere in health sciences education (Gonzalez, Palencia, Umana, Galindo, & Villafrade, 2008; Srinivasan et al., 2008; Vacek, 2009; West, Pomeroy, Park, Gerstenberger, & Sandoval, 2000; Wilgis & McConnell, 2008; Willemsen et al., 2008). An example of a concept map, with a few branches expanded for illustration purposes, is provided in Figure 1. Second, CTL teaches core competencies in assessment and safety planning almost entirely through medical record documentation examples and exercises. The decision to build the content around documentation was based on discussions with clinicians, administrators, and suicide experts in the department, who all agreed that documentation would provide a highly relevant “hook” for clinicians, and meet an important educational and regulatory need. Third, CTL provides context-specific options for responding to suicide risk. Prior to delivering CTL, ARP developed a 1-h structured interview and consultation to gather information from leaders of each clinical service about the specific options available to clinicians for consultation, referral, and safety planning. During this consultation, leaders are also prompted to develop plans to build from this training with ongoing clinical education and administrative support that will foster the transfer of learning to practice. He has since field-tested this 1-h interview format to customize the educational workshop for other service systems including a county mental health center and two state psychiatric facilities.

Figure 1

An example of a concept map.

Procedures

We delivered the CTL curriculum in 20 workshops between September 2007 and March 2009. Participation was mandatory for all clinical staff in the department concerned. Prior to registering, we sent participants a description of the educational workshop, the objectives of the workshop, and information that pre- and posttests were required as part of the department's evaluation of the program.

Participants registered for the workshop online. They completed a pretraining questionnaire that included: a self-evaluation of confidence in assessing and managing risk for suicide; a multiple-choice test of knowledge of suicide risk assessment and management; and a set of questions that gathered data about their professional discipline, gender, age, and extent of clinical experience with suicidal patients. At the start of the workshop, we presented all attendees with a clinical vignette about an at-risk patient. Attendees had 6 minutes to complete documentation of suicide risk. The CTL workshop included information on suicide risk and protective factors, examples of well-written suicide risk documentation, and hospital procedures for responding to elevated risk for suicide in different clinical settings. Immediately after the workshop, participants completed an evaluation that included a second documentation sample based on the same clinical vignette, re-administration of the confidence self-assessment and test of their knowledge of suicide risk assessment and management, and their perceptions of the likelihood that the knowledge gained in the workshop will transfer to their daily clinical practice.

Measures

Provider Confidence (Pre/post; Self-Report)

Participants completed a nine-item measure that assessed their confidence in suicide risk management on a Likert scale from 1 (strongly disagree) to 8 (strongly agree). The measure consisted of face valid items probing participants' confidence with key tasks in the care of suicidal patients. The scale showed high internal consistency in this study sample (α = .96). Example items from this scale are: “I feel well prepared to document a full assessment of risk for suicide” and “I feel well informed about how to identify and pursue the various options available in our hospital system to address elevated risk for suicide.”

Knowledge of Suicide Risk Assessment and Management (Pre/post; Multiple Choice Test)

Participants completed a 13-item, multiple-choice test assessing their knowledge of suicide risk assessment and management. To create the measure, we generated items based on the competencies prioritized in the workshop and through consultation with experts to identify items most relevant to clinical practice in a community mental health service. We also field-tested candidate items with first-year marriage and family therapy trainees and suicide research faculty. In the study sample, this scale had acceptable internal consistency (α = .79). Consistent with the structure of the training, seven questions tested knowledge about risk assessment and formulation of risk, and six items tested knowledge about clinical response and management. Example items from this scale are: “Once a patient voices suicidal ideation, the clinician should:” (correct answer: try to build on the therapeutic stance and alliance previously established.)

“Designation of `acute high risk' indicates that:” (correct answer: there is a high risk that the patient will take his/her own life soon.) “Which of the following is not recommended for managing a suicidal patient:” (correct answer: asking the patient to call to check in at the end of each day.)

Documentation of Risk Formulation (Pre/post; Objective Rating)

Trained coders scored the quality of participants' documentation of risk formulation with three separate items developed for the study. Two items were considered to be a proxy for the presence of a clinical judgment of risk: (1) whether the documentation included a summary of risk along a continuum (i.e., from “low” risk to “high” risk) and (2) whether the documentation included a time-based summary of risk (i.e., “acute,” “chronic”). The third item assessed the overall quality of documentation on a scale from 1 (very poor) to 8 (excellent).

We provided 25 h of training to two coders on how to score the documentation samples. They scored the documentation of participants for overall quality using exemplars that anchored ratings for “very poor” and “excellent” quality. Coders were masked to whether risk assessments were written before or after the workshop. To assess interrater reliability, coders double-coded a third of the documentation samples, remaining masked to the scores each other had assigned. This procedure allowed us to calculate intraclass correlation (ICC; two-way mixed model) values for absolute agreement. ICC values for each item were as follows: Item 1 (risk continuum, ICC = .79), Item 2 (acute/chronic distinction, ICC = .82), Item 3 (overall quality, ICC = .73). For those cases that were double-scored for the reliability check, we used averaged coder ratings in analyses.

Perception of Potential Transfer of Training (Post Only; Self-Report)

Participant perception of the potential for their learning in the workshop to transfer to their clinical work was assessed with an 18-item measure comprised of three subscales of The Learning Transfer System Inventory (Holton, Bates, Bookter, & Yamkovenko, 2007; Holton, Bates, & Ruona, 2000): (1) Motivation to Transfer, (2) Capacity to Transfer, and (3) Learning Readiness to Transfer. The total of all items was used as a composite perception of transfer of training (PTT) score. The PTT showed good internal consistency (α = .89), similar to that in our previous work using the scale (Cross, Cerulli, Richards, He, & Hermmann, 2010). Example items from this scale are: “I don't have time to implement this training,” “What is taught in training closely matches my job requirements,” and “I will get opportunities to use this training.”

Statistical Analyses

We analyzed the effect of the educational intervention on knowledge, confidence, and documentation skill using a generalized estimating equations (GEE) procedure. GEE permits comparison of means in a repeated measures design when data is missing completely at random (MCAR). We tested and confirmed the MCAR assumption by modeling missing responses as a function of observed responses using logistic regression. The analysis was conducted using the Genmod procedure in SAS. In order to provide a standardized metric of the observed effects and allow for a comparison of the present study outcomes to those of previous studies, we calculated within-subject effect sizes (pre/post change in pooled standard deviation units, or d for each of the pre/post outcomes of interest. Effect sizes were interpreted according to Cohen's guidelines for small (d = .20), medium (d = .50) and large (d = .80 or greater) effects (Cohen, 1988). We determined that we had sufficient power to detect small between-group differences in response to education among service areas and trainee groups using the G*Power3 program. We tested for group differences by comparing mean change scores using a general linear model (GLM) procedure. A p-value of .05 was used to define statistical significance throughout our analyses.

Results

Table 3 shows changes in scores on knowledge, confidence, and documentation skill from pre- to posttraining.

Table 3

Changes in pre- and postscores and effect sizes for main outcome measures

Knowledge and Confidence

Participants' knowledge and confidence scores improved significantly (p < .001) from pre- to posttraining. The mean knowledge score increased from 47% to 70% correct (effect size or d = 1.37). The mean confidence score increased from 4.9 to 6.3 on a scale of 1 to 7 (d = .95).

Documentation Skill

The number of participants who demonstrated core risk formulation skills increased significantly (p < .001) from preto posttraining, and the increase did not differ by service area or by trainee status. Before the workshop, 8% of participants made a distinction between acute and long-term risk in their documentation samples, compared with 49% who made this distinction after the workshop (d = 1.57). Before the workshop, 36% of participants made a summary judgment of risk on a low to high continuum, compared with 64% who made this distinction after the workshop (d = .61). The mean rating of overall quality of participants' documentation sample increased from 3.1 to 4.2 (d = .94).

Perception of Transfer of Training (PTT)

Participants' scores on the PTT were high (mean = 4.10 on 1 to 5 scale), indicating that participants perceived that they had sufficient knowledge, motivation, and resources to implement the educational content in their workplace.

Differences by Service Area and Trainee Status

As shown in Table 4, scores of participants who worked in ambulatory services, acute services, and other services all improved on measures of knowledge, confidence, and documentation of risk formulation. There was no significant difference in the mean change each of these groups experienced on any of the outcome variables, suggesting that scores for participants from all three service areas improved to approximately the same degree. Similarly, both trainees and nontrainees improved significantly on measures of knowledge, confidence, and documentation of risk formulation, with no significant difference in the mean change score between staff and trainees. Postworkshop PTT ratings also did not differ by service area or by trainee status.

Table 4

Changes in pre/post scores related to participants' service settings and professional status

Discussion

Participants in a novel 3-h workshop on the assessment and management of suicide risk gained knowledge, confidence, and skill in risk formulation based on assessments obtained immediately after the workshop. Participants reported strong confidence that they could implement the skills taught in the workshop in their usual practice setting. While mastery of any clinical skill requires practice and feedback over time, our findings are consistent with previous studies, which indicated that focused continuing education workshops can increase knowledge, confidence, and documentation skills (Gask et al., 2006; McNiel et al., 2008; Oordt et al., 2009). The results of our study extend previous work in clinician education in several important ways.

This study was conducted with a relatively large multidisciplinary sample of clinicians with clinical responsibilities across the spectrum of acute and outpatient care. This diverse sample showed large improvements on all self-report and objective measures, with no significant differences in change scores among participants from acute, ambulatory, and other services. Trainees and staff/faculty clinicians likewise benefited from the program to a similar degree. Participants saw the workshop as highly transferable to the workplace, regardless of their service setting. Together, these findings suggest that education in the core competencies of assessing and managing suicide risk can be accomplished through cross-services education, despite the diversity of clinical demands experienced in ambulatory, acute, and other services. These findings also suggest that a carefully designed curriculum can accommodate the needs of clinicians at various stages of professional development.

The design of the CTL curriculum presents opportunities to advance the state-of-the-art in clinician education. First, the curriculum demonstrated promising methods for teaching about risk assessment and response. The curriculum used visual concept mapping, documentation-driven delivery, and a structured procedure for customizing education to fit the response options available to clinicians in a given setting. Second, the efficiency with which this curriculum achieved fairly dramatic results has obvious practical advantages for clinicians and administrators, and also offers hope to those interested in improving the competence of the mental health workforce. Third, the preworkshop interview and consultation with clinical leaders results in a workshop that is tailored to the local clinical context. The consultation explicitly prompts leaders to use the brief educational experience as a starting point for ongoing education and administrative support for risk assessment. This direct engagement with stakeholders about how to enhance the transfer of learning into practice is novel, and serves as an example of a feasible step that developers can take to link education with implementation (Pisani et al., 2011). Future research is needed to determine what types of ongoing support lead to skill improvements that persist over time and extend to other aspects of patient care.

Constructs measured in this study contribute to the development of an evaluation science in the field of clinical education in managing suicide risk. The objective measurement of documentation skill in this study substantially extends findings of one previous study that measured documentation improvements in a small homogeneous sample (McNiel et al., 2008). This study is the first to use a validated instrument to assess how participants viewed the match between the education they received about suicide risk and the demands and resources present in their usual practice setting. Participants' assessment of this match has been shown in previous studies to be a critical factor in predicting transfer of training (Cross et al., 2010).

Limitations of the study are important to note. It is limited by the lack of a control group of clinicians who did not receive the educational intervention. Normed “gold standard” measures of suicide-specific knowledge, confidence, and skill are not available, but the measures used in this study were developed with expert input and had adequate psychometric properties. The brief measures of knowledge (13 items) and confidence (9 items) sampled key domains of competence and minimized participant burden, but could have missed important domains because of the limited number of items that it was feasible to present. The repeated measures design entails a risk that a portion of the large effects changes seen on posttest reflect a “practice effect.” Participants in this study were mandated by their employer to attend the educational workshop, which could have influenced self-report ratings and motivation to perform well on posttest measures. We attempted to minimize response bias by directly assuring and transparently demonstrating to participants that their identifying information would be kept separate from responses at every stage of data collection and analysis; however, the risk of bias could still exist. Finally, despite marked improvement on all domains measured, posttraining scores in knowledge, risk formulation, and overall quality of documentation leave additional room for improvement. It may be that a longer experiential training would have yielded even more dramatic improvements, though this is an empirical question. The knowledge test and documentation coding standards also may have required a higher degree of knowledge and skill than is necessary for competent practice.

Future research can build upon this study in a number of directions. First, observational studies are essential for the next generation of education research. Direct observation of patient care and/or standardized patient simulations would allow us to measure the effect of education on interviewing, crisis management, and safety planning. Second, future studies of CTL and other clinician educational workshops should include long-term follow-up to determine whether improvements achieved through education persist over time. Third, related to long-term skill development, we need implementation research to test strategies such as booster educational sessions, performance feedback, and clinician reminder systems to promote the transfer and continued use of skills into clinicians' daily practice. Fourth, models for teaching risk assessment need to be compared in terms of their effect on clinician decisions and interactions with patients. For example, do clinicians educated in different models of risk assessment make different judgments about patient lethality or the need for hospitalization? Likewise, future research should investigate which clinician characteristics and attitudes influence the uptake of new skills. Finally, comparative cost-effectiveness data are needed in order to evaluate the potential economic benefit of CTL and other types of suicide-risk intervention education.

In conclusion, CTL is a promising, innovative, and efficient curriculum for educating practicing clinicians to assess and respond to suicide risk. This evaluation demonstrated that a wide range of acute and ambulatory services staff and trainees can benefit from a singe well-designed curriculum. The study suggests the need for controlled observational research to test, compare, and enhance the long-term impact of education on clinical practice.

Acknowledgments

The authors would like to thank the many faculty, staff, and trainees in the Department of Psychiatry at the University of Rochester who contributed to the development of the Commitment to Living curriculum. The authors gratefully acknowledge Aaron Horowitz, Scott Michaels, Erin Rogers, and Dr. Constance D. Baldwin's scientific writing group for their assistance with this article.

Dr. Pisani received funding from the National Institute of Mental Health (NIMH T32 MH20061, Conwell, PI). Dr. Cross also received funding from NIMH (K23MH73615; K23MH073615-03S1).

Biographies

•	Anthony R. Pisani, PhD, is an Assistant Professor in Psychiatry (Psychology) and Pediatrics at the University of Rochester Center for the Study and Prevention of Suicide and Institute for the Family, Rochester, NY, USA. A psychologist and family therapist, he works with clinicians, service agencies, and schools to implement practical and effective approaches to preventing suicide.
•	Wendi F. Cross, PhD, is an Associate Professor of Psychiatry (Psychology) at the University of Rochester Medical Center, Rochester, NY, USA. She is a clinical psychologist specializing in implementation research. Her work focuses on testing models, methods and measures for training individuals charged with carrying forth a variety of evidence-based interventions into community settings.
•	Arthur Watts, BS, is a programmer in the Department of Biostatistics at the University of Rochester Medical Center, Rochester, NY, USA.
•	Kenneth Conner, PsyD, MPH, is Associate Professor of Psychiatry at the University of Rochester Medical Center and Clinical Research Core Chief at the VA VISN 2 Center of Excellence for Suicide Prevention, Rochester, NY, USA. He studies the role of substance use disorders in suicidal behavior and brief, practical interventions to reduce suicidal behavior.

References

American Psychiatric Association Assessment and treatment of patients with suicidal behaviors. 2003 Retrieved from http://www.psychiatryonline.com/pracGuide/pracGuideTopic_14.aspx.
Brown GK, Beck AT, Steer RA, Grisham JR. Risk factors for suicide in psychiatric outpatients: A 20-year prospective study. Journal of Consulting and Clinical Psychology. 2000;68:371–377. doi: http://dx.doi.org/10.1037/0022-006X.68.3.371. [PubMed]
Bryan CJ, Rudd M. Advances in the assessment of suicide risk. Journal of Clinical Psychology. 2006;62:185–200. [PubMed]
Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Erlbaum; Hillside, NJ: 1988.
Cross W, Cerulli C, Richards H, He H, Hermmann J. Predicting dissemination of a disaster mental health “train-the-trainer” program. Disaster Medicine and Public Health Preparedness. 2010;4:339–343. [PubMed]
Fenwick CD, Vassilas CA, Carter H, Haque M. Training health professionals in the recognition, assessment and management of suicide risk. International Journal of Psychiatry in Clinical Practice. 2004;8:117–121. doi: http://dx.doi.org/10.1080/13651500410005658.
Gask L, Dixon C, Morriss R, Appleby L, Green G. Evaluating STORM skills training for managing people at risk of suicide. Journal of Advanced Nursing. 2006;54:739–750. [PubMed]
Gonzalez HL, Palencia AP, Umana LA, Galindo L, Villafrade MLA. Mediated learning experience and concept maps: a pedagogical tool for achieving meaningful learning in medical physiology students. Advances in Physiology Education. 2008;32:312–316. [PubMed]
Harris E, Barraclough B. Suicide as an outcome for mental disorders: A meta-analysis. British Journal of Psychiatry. 1997;170:205–228. [PubMed]
Holton EF, III, Bates RA, Bookter AI, Yamkovenko V. Convergent and divergent validity of the Learning Transfer System Inventory. Human Resource Development Quarterly. 2007;18:385–419. doi: http://dx.doi.org/10.1002/hrdq.1210.
Holton EF, III, Bates RA, Ruona WE. Development of a generalized learning transfer system inventory. Human Resource Development Quarterly. 2000;11:333–360.
Hunt IM, Kapur N, Robinson J, Shaw J, Flynn S, Bailey H, Parsons R. Suicide within 12 months of mental health service contact in different age and diagnostic groups: National clinical survey. British Journal of Psychiatry. 2006;188:135–142. [PubMed]
Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: A review of the evidence. The American Journal of Psychiatry. 2002;159:909–916. [PubMed]
McNiel DE, Fordwood SR, Weaver CM, Chamberlain JR, Hall SE, Binder RL. Effects of training on suicide risk assessment. Psychiatric Services. 2008;59:1462–1465. doi: 10.1176/appi.ps.59.12.1462. [PubMed]
Mintzes J, Wandersee J, Novak JD. Teaching science for understanding: A constructivist view. Academic Press; San Diego, CA: 1998.
Novak JD. Concept mapping: A strategy for organizing knowledge. In: Glynn SM, Duit R, editors. Learning science in the schools: Research reforming practice. Erlbaum; Hillsdale, NJ: 1995. pp. 229–245.
Oordt MS, Jobes DA, Fonseca VP, Schmidt SM. Training mental health professionals to assess and manage suicidal behavior: Can provider confidence and practice behaviors be altered? Suicide and Life-Threatening Behavior. 2009;39:21–32. [PubMed]
Pisani AR, Cross WC, Gould MS. The assessment and management of suicide risk: State of workshop education. Suicide and Life-Threatening Behavior. 2011;41:255–276. doi: 10.1111/j.1943-278x.2011.00026.x. [PMC free article] [PubMed]
Srinivasan M, McElvany M, Shay JM, Shavelson RJ, West DC. Measuring knowledge structure: Reliability of concept mapping assessment in medical education. Academic Medicine. 2008;83:1196–1203. [PubMed]
Suicide Prevention Resource Center Core competencies in the assessment and management of suicide risk. Author; Newton, MA: 2006.
Vacek JE. Using a conceptual approach with a concept map of psychosis as an exemplar to promote critical thinking. The Journal of Nursing Education. 2009;48:49–53. [PubMed]
West DC, Pomeroy JR, Park JK, Gerstenberger EA, Sandoval J. Critical thinking in graduate medical education: A role for concept mapping assessment? Journal of the American Medical Association. 2000;284:1105–1110. [PubMed]
Wilgis M, McConnell J. Concept mapping: An educational strategy to improve graduate nurses' critical thinking skills during a hospital orientation program. Journal of Continuing Education in Nursing. 2008;39:119–126. [PubMed]
Willemsen AM, Jansen GA, Komen JC, van Hooff S, Waterham HR, Brites PM, van Kampen AH. Organization and integration of biomedical knowledge with concept maps for key peroxisomal pathways. Bioinformatics. 2008;24(16):i21–27. [PubMed]