Our results demonstrate that a substance abuse OSCE can teach addiction medicine competencies by providing performance-based assessments and immediate feedback. This study offers unique insights into residents' ability to interact with patients in different stages of change who are using different types of substances. It also reveals new information about trainees' interest and perceived competence in substance abuse assessment and management. As OSCEs can thoroughly test specific skill sets, our results show that assessment and management of substance abuse are equally difficult but significantly more challenging than general communication skills. We conclude that future educational interventions should target specific assessment and management skills.6,7,11
Because of the higher prevalence of alcohol abuse41
than other substance abuse disorders in medical settings, we hypothesized that residents would perform better on the alcohol stations. However, performance was weakest on the station portraying a contemplative alcohol-abusing patient, suggesting that this station presents uniquely difficult challenges. While contemplative patients are aware of their need to change, the ambivalence that characterizes this stage can produce resistance in the doctor patient encounter.42
Contemplation is the stage that most calls upon the use of motivational interviewing strategies, which emphasize empathic listening and address resistance by helping patients identify their own reasons for change.42
These strategies may employ communication techniques such as “ask, tell, ask,” which involve asking permission before delivering therapeutic suggestions and asking for the patient's reaction afterwards, or may include coaching the contemplative patient to weigh the pros and cons of change. Motivational interviewing techniques require specialized training and practice and may not be in the general skill set of resident trainees. To our knowledge, previous studies have not examined how different stages of readiness to change affect physician performance during clinical encounters. Because motivational techniques for contemplative patients could be applied to other behaviors such as smoking cessation and weight loss, further research is needed to determine how best to train residents to work with patients in different stages of change.
The high correlation of faculty global ratings and standardized patient ratings demonstrates interrater reliability and supports the convergent validity of the rating system. In contrast, residents' self-assessments were lower than faculty or SP ratings. Previous studies have shown that self-ratings do not correlate well with objective measures of performance, and that high performers in particular tend to underestimate themselves when compared with expert raters.43
Despite these limitations, self-assessment may encourage self-reflection and stimulate future learning.
We were surprised to find that residents' self-assessed interest and competence were not associated with OSCE performance. While residents' perceived themselves as more competent in alcohol use-specific than drug use-specific skills, these perceptions did not match their performance. This suggests that residents may not have accurate perceptions of their abilities in substance abuse management. A substance abuse OSCE may therefore help identify strengthens and weaknesses, as most of the residents confirmed.
Instant feedback is highly useful in forming trainee skills and has been shown to influence performance on subsequent stations with similar content.21,44,45
Objective structured clinical exams with immediate feedback delivered at each station have been very well received by medical trainees.44
Immediate feedback has the additional advantages of diminishing assessor fatigue and enhancing faculty teaching skills, while preserving reliability.18
In this OSCE feedback resulted in improved resident performance in subsequent stations, an important measure of skills acquisition and learning. As a formative assessment, our OSCE achieved its goal of teaching competencies in addiction medicine. Of note, residents who recounted prior substance abuse training performed better than those who had not. This finding further supports the influence of specific training on skill development.
A limitation of our OSCE is that its interstation reliability was moderate (Cronbach's α = 0.64), which may reflect the relatively small number of stations (n = 5) and the differences in raters' styles. It may also reflect the wide range of skills represented by items in the rating forms. For example, the domains of assessment and management included divergent tasks in different stations, while general communication skills were the same across stations. The limited number of stations evaluating each stage of change and substance restricts our ability to draw conclusions from performance differences. Although half of the residents had prior substance abuse training, an additional limitation is that the quantity and quality of these experiences is unknown and likely varied among respondents. We also did not control for residents' past personal experiences, which may influence both motivation and integration of structured training. Finally, in our assessment of the impact of feedback on performance improvement during the OSCE, we did not control for experience by including a control group not receiving feedback.
Conducting the substance abuse OSCE enabled us to tailor our curriculum to better meet the learning needs of our generalist trainees. Before implementing the OSCE, our curriculum included lectures during ambulatory blocks, skills training on alcohol diagnosis and management using simulated patients, and visits to 12-step meetings. New curricular innovations include inpatient lectures on addiction, cocaine use, methadone treatment, and medical detoxification; enhanced use of SPs to teach motivational interviewing skills; and seminars on prescription opiate use, buprenorphine, and physician impairment.
In conclusion, our 5-station OSCE provides a moderately reliable summative measure of our residents' skills in substance abuse and a great deal of information about resident performance in distinct skill areas. While residents perform well in general communication skills, they generally lack adequate skills for assessing and managing substance abuse disorders. In particular, residents need further training in motivational techniques for dealing with resistance in contemplative substance abusing patients. Our results also demonstrate the positive effect of feedback on learning addiction medicine competencies. Implementing the substance abuse OSCE has allowed us to employ new strategies for faculty development, to develop a novel educational program for trainees from different departments, and to develop curricular innovations in substance abuse.