|Home | About | Journals | Submit | Contact Us | Français|
Although residents commonly manage substance abuse disorders, optimal approaches to teaching these specialized interviewing and intervention skills are unknown.
We developed a Substance Abuse Objective Structured Clinical Exam (OSCE) to teach addiction medicine competencies using immediate feedback. In this study we evaluated OSCE performance, examined associations between performance and self-assessed interest and competence in substance abuse, and assessed learning during the OSCE.
Five-station OSCE, including different substance abuse disorders and readiness to change stages, administered during postgraduate year-3 ambulatory rotations for 2 years.
One hundred and thirty-one internal and family medicine residents.
Faculty and standardized patients (SPs) assessed residents' general communication, assessment, management, and global skills using 4-point scales. Residents completed a pre-OSCE survey of experience, interest and competence in substance abuse, and a post-OSCE survey evaluating its educational value. Learning during the OSCE was also assessed by measuring performance improvement from the first to the final OSCE station.
Residents performed better (P<.001) in general communication (mean ± SD across stations = 3.12 ± 0.35) than assessment (2.65 ± 0.32) or management (2.58 ± 0.44), and overall ratings were lowest in the contemplative alcohol abuse station (2.50 ± 0.83). Performance was not associated with residents' self-assessed interest or competence. Perceived educational value of the OSCE was high, and feedback improved subsequent performance.
Although internal and family medicine residents require additional training in specialized substance abuse skills, immediate feedback provided during an OSCE helped teach needed skills for assessing and managing substance abuse disorders.
Substance abuse is a common and challenging problem in medical settings. National estimates suggest that 25% to 40% of hospital admissions and 10% to 20% of general outpatient visits are related to addiction.1–3 The prevalence of unhealthy alcohol use is at least 7% to 20% among medical outpatients.4 Physician trainees in internal and family medicine often provide medical care to drug and alcohol users, but many studies have documented that physicians are reluctant to address substance abuse disorders and may have negative attitudes toward substance abusing patients.5 Ineffective screening and history taking may lead physicians to miss such disorders in up to two-thirds of patients.6 Even when substance abuse disorders are detected, physicians may not provide adequate interventions.3,6
Substance abuse training for resident physicians is limited and often offered only electively.7 Although the U.S. Preventive Services Task Force has established the efficacy of screening (1996)8 and counseling (2004)9 for alcohol use disorders, consistent training is lacking in primary care residencies.10 Training has been recommended for generalist physicians in: knowledge about substance abuse symptoms, signs, sequelae, and treatment; and specific interviewing skills, including screening and brief interventions.11 However, optimal methods to teach substance abuse competencies to resident physicians are currently unknown.
Residency programs are increasingly required to provide performance-based assessments to document clinical skills, and objective structured clinical exams (OSCEs) are the preferred or next best method for assessing many of these skills.12 An OSCE is a timed, multistation examination in which learners perform tasks such as interviews, physical exams, and counseling with standardized patients (SPs) in realistic settings.13,14 At each station learner performance is evaluated with specific checklists or global rating scales, completed by faculty proctors and/or SPs. Objective structured clinical exams enable the same clinical scenarios to be presented to many trainees15 and have become the gold standard for performance-based assessment.16 When immediate feedback on performance is provided, OSCEs can be formative in developing skills and enhancing knowledge.17,18 Objective structured clinical exams are widely used in residency training disciplines, including internal medicine,19,20 surgery,12 pediatrics,21 rehabilitation medicine,22 psychiatry,23,24 and family medicine.25 They have also been used to teach and assess specialized skills such as delivering bad news,26 addressing advanced care directives,27 demonstrating cultural competence,28 and ethical decision making.29
Previous studies have examined the use of SPs within and outside of formal OSCEs to assess addiction medicine skills following teaching interventions. Objective structured clinical exams administered before and after educational programs have demonstrated improvements in medical students' substance abuse assessment and management skills.30–32 Among residents, 2 studies used SPs to demonstrate the effect of intensive substance abuse training,33,34 and 1 used SPs to describe the need for improved skills.5 While many OSCEs use a single substance abuse, smoking, or behavior change station, none have focused completely on substance abuse. In addition, though physicians' perceptions of their own interest and competence in managing substance abuse may influence their ability to conduct screenings, brief interventions, and referrals,35,36 the association between these perceptions and performance measured by direct observation has not previously been studied.
Our main objective was to evaluate performance on a 5-station substance abuse OSCE designed to assess and teach competencies in addiction medicine to postgraduate year-3 (PGY-3) internal and family medicine residents. We also sought to compare residents' skills in managing alcohol and other substance abuse problems and to determine whether self-assessed interest and competence in substance abuse were associated with OSCE performance. Finally, we sought to evaluate the OSCE's educational value and to assess learning during the OSCE. This study was approved by Montefiore Medical Center's Institutional Review Board.
After reviewing substance abuse competencies recommended for primary care physicians,11 experts in primary care and addiction medicine constructed a blueprint covering different substances of abuse (heroin, cocaine, alcohol), readiness to change stages (precontemplation to maintenance),37,38 and representative ages, genders, and ethnicities (Table 1). Faculty then wrote case scenarios based on clinical experience, existing literature, and samples from a case resource book39; and cases were piloted and refined using role play.
Twelve faculty members in internal medicine, family medicine, and psychiatry served as evaluators. During 6 hours of faculty development, they role-played each case, underwent training to standardize ratings,40 agreed on teaching points, and practiced giving case-specific feedback. To ensure interrater reliability, faculty compared ratings and agreed upon expected performance for specific items on each rating form. Professional actors served as SPs and were trained using detailed scripts and practice with faculty members to standardize performance and feedback.
The substance abuse OSCE was administered during ambulatory blocks in outpatient settings similar or identical to those in which the residents practice. Residents rotated through all 5 stations and had 2 minutes before each encounter to read background information and a list of specific tasks. During each encounter residents had 10 minutes to build rapport and assess and manage the case, while a faculty member observed and completed a rating form. At the end of each encounter, faculty provided 5 minutes of direct feedback and delivered teaching points. The OSCE concluded with a faculty-led group review of each case. For each resident the OSCE required 2.5 hours, but simultaneous introductory and review discussions allowed 10 residents to complete the OSCE in 5 hours using 2 rotations.
Faculty completed a 17-item form including 15 specific items in 3 domains (6 general communication, 6 assessment, and 3 management items), and 2 global items (general organization and overall performance) (Table 2). Items evaluating general communication and the 2 global skills were uniform across stations, while items evaluating assessment and management skills were unique to each station. Each item was rated on a 4-point scale, ranging from 1 = needs much improvement to 4 = done excellently. Residents assessed their own overall station performance using the same 4-point scale. Standardized patients provided a global satisfaction rating (1 = would not come back, 2 = not sure if I would continue with doctor, 3 = would continue, 4 = one of the best doctors, would refer friends).
Faculty ratings resulted in both subscores for each of the 3 domains and station total scores. We used data from the specific items to calculate mean subscores for each domain in each station, and also to calculate an aggregate mean subscore for each domain across stations. In addition, we calculated mean station total scores using data from the 15 specific items for each station, and also mean station total scores for each station using 1 faculty global rating (overall performance). Finally, we calculated mean SP ratings and mean resident self-assessment ratings for each station and across all stations.
A summary OSCE score was generated for each resident by calculating a mean of the scores on the 15 specific items in all 5 stations (total of 75 items). The interstation reliability of the entire OSCE was evaluated using Cronbach's α, calculated from the 5 mean station total scores.
Residents' prior experience in substance abuse was assessed with 2 yes/no questions: previous substance abuse training, and attendance at 12-step meetings (e.g., Alcoholics Anonymous). Each question was analyzed separately to determine its association with the summary OSCE score. Because we were concerned that personal questions would inhibit survey responses, we did not collect information about residents' personal experience with substance abuse or its treatment.
Resident's interest and competence in substance abuse management were assessed using a 4-point scale, ranging from 1 = less skilled or interested to 4 = most skilled or interested. Summary scores for each scale were calculated, and reliability was determined using Cronbach's α. The association of each scale's summary score with the summary OSCE score was evaluated. Because we were interested in whether competence in alcohol abuse was different than in drug abuse, we asked 2 questions that were unique to alcohol abuse and then asked the same 2 questions about drug abuse. Scores from the 2 alcohol abuse competence questions were then compared with scores from the 2 drug abuse competence questions.
We evaluated learning by analyzing the effect of feedback on performance in subsequent stations. We determined whether mean station total scores and mean subscores in the 3 domains improved from each resident's first station to their last station. In addition, we asked residents to evaluate the OSCE's overall educational value with 5 questions, each of which had 3 responses (definitely, maybe, and no).
Differences between continuous variables were assessed using analysis of variance for 3 group comparisons and independent t tests for 2-group comparisons. Associations between continuous variables were assessed using Pearson's correlation coefficients. In analyses of the same residents' scores on 2 different questions or scales (e.g., alcohol and drug competence), paired t tests were used to compare differences. To evaluate whether station performance changed from the first to the last station, we calculated change scores and assessed their significance using 1 sample t tests.
Between August 2003 and November 2005, the substance abuse OSCE was administered to 131 PGY-3 residents, including 107 internal medicine and 24 family medicine residents. There were no differences between internal and family medicine residents in domain subscores, 15-item or global station total scores, or summary OSCE scores.
In each of the 5 stations residents performed better (P<.001) in general communication than either assessment or management. Aggregate subscores were highest in general communication (mean ± SD = 3.12 ± 0.35), next highest in assessment (2.65 ± 0.32), and lowest in management (2.58 ± 0.44).
Residents performed least well in the station portraying a contemplative alcohol abuser (mean global rating ± SD = 2.50 ± 0.83), and best in the station portraying a precontemplative cocaine user (mean global rating ± SD = 2.93 ± 0.74) (P<.001 for the difference between these 2 stations). The 2 types of faculty station total scores (15-item and global) were highly correlated (r = .87, P<.01), and both were correlated with SP scores (r = .70, P<.01; r = .68, P<.01, respectively). In 3 stations, faculty ratings were lower than SP ratings. Residents rated themselves lower than faculty in all stations except the contemplative alcohol abuser and lower than SPs in all stations.
The mean OSCE summary score was 2.83 (range 1.78 to 3.44, SD = 0.32), and the interstation reliability was moderate (Cronbach's α = 0.64).
One hundred and thirty residents (99%) completed the pre-OSCE survey. Sixty-four (50%) had completed some training in substance abuse; and 83 (64%) had attended 12-step meetings. Residents who had prior substance abuse training performed better than those without training (mean summary score ± SD = 2.88 ± 0.27 vs 2.75 ± 0.36, P = .03), but there was no difference between those attending or not attending 12-step meetings (2.83 ± 0.31 vs 2.78 ± 0.34, P = .4).
Reliability of the interest and competence scales was high (Cronbach's α = 0.95 and 0.85, respectively). There were no correlations between summary OSCE scores and either interest or competence. Residents reported greater competence in both alcohol-specific skills (screening for and diagnosing alcohol abuse) than in the analogous drug-specific skills (mean score for 2 alcohol questions: 2.58 ± 0.68 and 2.59 ± 0.70 vs mean score for 2 drug questions: 2.30 ± 0.62 and 2.38 ± 0.73, P<.001 for the comparisons of both questions).
Among 128 residents (98%), we were able to assess changes in performance from the first to the final OSCE station (Table 5). Because some OSCE sessions had fewer than 5 residents per rotation, the number of residents beginning with each station was unequal. Including all starting points, residents' mean station total scores improved from their first station to their last station (change = 0.14 ± 0.64, P<.01), as did their scores in assessment (change = 0.15 ± 0.78, P = .03) and in management (change = 0.20 ± 0.97, P = .02), but not in general communication. When each starting point was examined separately, improvements were not observed in all rotation orders because of small numbers of residents beginning with each station.
Resident impressions of the OSCE are summarized in Table 6. Half thought the stations resembled real-life clinical encounters, and the majority (73%) thought they provided a good cross-section of substance abuse issues. Over three-quarters of the residents felt the OSCE helped identify strengths and weaknesses, taught them something new, and provided valuable feedback.
Our results demonstrate that a substance abuse OSCE can teach addiction medicine competencies by providing performance-based assessments and immediate feedback. This study offers unique insights into residents' ability to interact with patients in different stages of change who are using different types of substances. It also reveals new information about trainees' interest and perceived competence in substance abuse assessment and management. As OSCEs can thoroughly test specific skill sets, our results show that assessment and management of substance abuse are equally difficult but significantly more challenging than general communication skills. We conclude that future educational interventions should target specific assessment and management skills.6,7,11
Because of the higher prevalence of alcohol abuse41 than other substance abuse disorders in medical settings, we hypothesized that residents would perform better on the alcohol stations. However, performance was weakest on the station portraying a contemplative alcohol-abusing patient, suggesting that this station presents uniquely difficult challenges. While contemplative patients are aware of their need to change, the ambivalence that characterizes this stage can produce resistance in the doctor patient encounter.42 Contemplation is the stage that most calls upon the use of motivational interviewing strategies, which emphasize empathic listening and address resistance by helping patients identify their own reasons for change.42 These strategies may employ communication techniques such as “ask, tell, ask,” which involve asking permission before delivering therapeutic suggestions and asking for the patient's reaction afterwards, or may include coaching the contemplative patient to weigh the pros and cons of change. Motivational interviewing techniques require specialized training and practice and may not be in the general skill set of resident trainees. To our knowledge, previous studies have not examined how different stages of readiness to change affect physician performance during clinical encounters. Because motivational techniques for contemplative patients could be applied to other behaviors such as smoking cessation and weight loss, further research is needed to determine how best to train residents to work with patients in different stages of change.
The high correlation of faculty global ratings and standardized patient ratings demonstrates interrater reliability and supports the convergent validity of the rating system. In contrast, residents' self-assessments were lower than faculty or SP ratings. Previous studies have shown that self-ratings do not correlate well with objective measures of performance, and that high performers in particular tend to underestimate themselves when compared with expert raters.43 Despite these limitations, self-assessment may encourage self-reflection and stimulate future learning.
We were surprised to find that residents' self-assessed interest and competence were not associated with OSCE performance. While residents' perceived themselves as more competent in alcohol use-specific than drug use-specific skills, these perceptions did not match their performance. This suggests that residents may not have accurate perceptions of their abilities in substance abuse management. A substance abuse OSCE may therefore help identify strengthens and weaknesses, as most of the residents confirmed.
Instant feedback is highly useful in forming trainee skills and has been shown to influence performance on subsequent stations with similar content.21,44,45 Objective structured clinical exams with immediate feedback delivered at each station have been very well received by medical trainees.44 Immediate feedback has the additional advantages of diminishing assessor fatigue and enhancing faculty teaching skills, while preserving reliability.18 In this OSCE feedback resulted in improved resident performance in subsequent stations, an important measure of skills acquisition and learning. As a formative assessment, our OSCE achieved its goal of teaching competencies in addiction medicine. Of note, residents who recounted prior substance abuse training performed better than those who had not. This finding further supports the influence of specific training on skill development.
A limitation of our OSCE is that its interstation reliability was moderate (Cronbach's α = 0.64), which may reflect the relatively small number of stations (n = 5) and the differences in raters' styles. It may also reflect the wide range of skills represented by items in the rating forms. For example, the domains of assessment and management included divergent tasks in different stations, while general communication skills were the same across stations. The limited number of stations evaluating each stage of change and substance restricts our ability to draw conclusions from performance differences. Although half of the residents had prior substance abuse training, an additional limitation is that the quantity and quality of these experiences is unknown and likely varied among respondents. We also did not control for residents' past personal experiences, which may influence both motivation and integration of structured training. Finally, in our assessment of the impact of feedback on performance improvement during the OSCE, we did not control for experience by including a control group not receiving feedback.
Conducting the substance abuse OSCE enabled us to tailor our curriculum to better meet the learning needs of our generalist trainees. Before implementing the OSCE, our curriculum included lectures during ambulatory blocks, skills training on alcohol diagnosis and management using simulated patients, and visits to 12-step meetings. New curricular innovations include inpatient lectures on addiction, cocaine use, methadone treatment, and medical detoxification; enhanced use of SPs to teach motivational interviewing skills; and seminars on prescription opiate use, buprenorphine, and physician impairment.
In conclusion, our 5-station OSCE provides a moderately reliable summative measure of our residents' skills in substance abuse and a great deal of information about resident performance in distinct skill areas. While residents perform well in general communication skills, they generally lack adequate skills for assessing and managing substance abuse disorders. In particular, residents need further training in motivational techniques for dealing with resistance in contemplative substance abusing patients. Our results also demonstrate the positive effect of feedback on learning addiction medicine competencies. Implementing the substance abuse OSCE has allowed us to employ new strategies for faculty development, to develop a novel educational program for trainees from different departments, and to develop curricular innovations in substance abuse.
We would like to thank Laxmi Modali for assistance with data analysis, Denise Hernandez for assistance with administering the OSCE, and all the residents and faculty who participated.
Portions of this research were presented at the Society of General Internal Medicine Mid-Atlantic Regional Meeting in March 2004, the Society of General Internal Medicine 27th Annual Meeting in May 2004, and the meeting of the Association of Medical Education and Research in Substance Abuse in November 2004. The project was supported by NIH R25 DA 14551 to Montefiore Medical Center (P.I. J. Arnsten).