Standardized Patient-Narrated Web-Based Learning Modules Improve Students’ Communication Skills on a High-stakes Clinical Skills Examination

doi:10.1007/s11606-011-1809-3

J Gen Intern Med. 2011 November; 26(11): 1374–1377.

Published online 2011 July 16. doi: 10.1007/s11606-011-1809-3

PMCID: PMC3208474

Standardized Patient-Narrated Web-Based Learning Modules Improve Students’ Communication Skills on a High-stakes Clinical Skills Examination

Christina A. Lee, MD,¹ Anna Chang, MD,^2,³ Calvin L. Chou, MD, PhD,^2,³ Christy Boscardin, PhD,² and Karen E. Hauer, MD²

¹Palo Alto Medical Foundation, 795 El Camino Real, Palo Alto, CA 94301 USA

²University of California, San Francisco, CA USA

³San Francisco Veterans Affairs Medical Center, San Francisco, CA USA

Christina A. Lee, Phone: +1-650-3214121, Email: christinaaudreylee/at/gmail.com.

Corresponding author.

Received November 8, 2010; Revised March 4, 2011; Accepted July 6, 2011.

Abstract

BACKGROUND

Use of web-based standardized patient (SP) modules is associated with improved medical student history-taking and physical examination skills on clinical performance examinations (CPX), but a benefit for communication skills has not been shown.

AIM

We describe an innovative web-based SP module using detailed SP and faculty commentary to teach communication skills.

SETTING

A public medical school in 2008–2009.

PARTICIPANTS

Fourth-year medical students.

PROGRAM DESCRIPTION

A 90-minute web-based module with three simulated clinical encounters was narrated by an expert clinician and SP to explain expected history-taking, physical examination, and communication skills behaviors. All 147 students were encouraged to review the module one month before the CPX.

PROGRAM EVALUATION

One hundred and six students (72%) viewed the web-based module. Students who watched the module performed significantly higher on the CPX communication score (+2.67%, p

0.01) and overall score (+2.12%, p

0.03), even after controlling for USMLE Step 1 and clerkship summary ratings. Use of the module did not significantly affect history/physical examination scores (+1.89%, p

0.12).

DISCUSSION

Students who watched an optional web-based SP module prior to the CPX performed higher than those who did not on communication skills. The web-based module appears to be an effective CPX preparatory activity to enhance communication performance.

KEY WORDS: medical education, clinical skills, standardized patient, communication

INTRODUCTION

Effective physician–patient communication correlates with improved health outcomes, patient adherence, and patient satisfaction.¹^–³ As key components of professional competence, interpersonal and communication skills learning begins in medical school, although communication skills teaching methods vary across schools.⁴ Most include experiential exercises such as interactive workshops, role-play with standardized patients (SPs), videotape review, and practice with peers.⁵^,⁶ The participation of SPs in teaching and evaluating communication skills is effective.⁷ Medical students who receive instruction from SPs reflecting the patient perspective demonstrate improved communication skills and higher satisfaction, even when compared to receipt of feedback from faculty or real patients.⁸^–¹¹ However, SP training is costly and time consuming.

Most United States medical schools conduct a comprehensive clinical skills examination to assess professional competence, including communication skills.¹² Preparatory educational activities reinforce key concepts and improve performance on high-stakes examinations.¹³ Because comprehensive clinical skills examinations generally involve SPs, practice with SP encounters is helpful for clinical skills test preparation.¹⁴^,¹⁵ However, there is limited data on optimal timing and format of test preparation strategies. Existing studies differ on whether preparation strategies employed just prior to testing (i.e. “last-minute” preparation) are effective.¹⁶^–¹⁹

Web-based learning in medical education is associated with positive learning outcomes and appears to be as effective as traditional teaching methods.²⁰^,²¹ Web-based SP teaching modules typically consist of videotaped clinical encounters between medical students and SPs with expert clinician commentary; they are more cost-effective, convenient for students, and easier to administer although less preferred by students than in-person SP modules.¹³ In our prior study, students who used a web-based SP module or in-person teaching exam with SPs to prepare months in advance for a high-stakes Clinical Performance Examination (CPX) scored higher than historical controls on history-taking and physical examination skills but showed no difference in communication skills.¹³ Given the importance of communication skills in clinical practice, we sought to develop a new method to enhance students’ communication skills performance.

We describe an innovative web-based SP module to teach communication skills effectively. We hypothesized that students who accessed optional web-based SP modules with detailed SP and faculty commentary proximal to a high-stakes clinical skills examination would demonstrate improved communication skills performance.

Setting and Participants

Participants were 147 early fourth-year medical students after their core clerkships at a public medical school in 2008–2009. All students participate in an eight-station, high-stakes CPX at the beginning of their fourth year; a passing score is required for graduation. The school is one of eight schools of the California Consortium for the Assessment of Clinical Competence (CCACC), which collaborates to determine examination and checklist content. Cases represent common ambulatory medicine problems and are incorporated into the CPX after being piloted by CCACC schools. In each clinical encounter, students spend 15 minutes with a SP performing a history and physical examination and communicating an initial assessment and plan. SPs evaluate students immediately after each encounter using faculty-developed checklists. CPX score reliability and correlation between performance in SP examinations and clinical practice have been reported previously.²²^–²⁴

Program Description

We created a web-based SP module with three videotaped clinical encounters, each between a different medical student and SP, based on common ambulatory chief complaints similar in content and difficulty level to CPX cases. Each case included 20 minutes of interaction between student and SP with five additional minutes of interspersed commentary from a clinician and three from a SP. Expert clinician voiceover intermittently explained clinical reasoning and expected clinical skills for the encounter, and SP voiceover critiqued students’ communication skills from the patient’s perspective. Clinician narratives were scripted and narrated by study authors (C.A.L., A.C., and C.L.C.) based on history/physical examination scoring checklists aligned with diagnostic evaluation of the patient's chief complaint. Patient narratives were spoken by SPs who had undergone 20+ hours of training for these cases and based their comments on a standardized, validated checklist for communication skills.²⁵

One month before the CPX, CPX directors emailed all 147 students recommending that they view at least two of the three module cases prior to the CPX. Students received two subsequent weekly reminder emails. Because students were already required to participate in a formative in-person SP examination as a preparatory activity six months prior, we timed the module to occur closer to the CPX date and made the module voluntary. Each student received a unique login and password for module access. The time of login and logout was recorded to determine total time students accessed the module. The medical school institutional review board approved this study.

Program Evaluation

The primary outcome measure was CPX communication scores; history/physical examination and overall performance scores were also examined.

We compared scores between students who used the web-based module and those who did not. A series of multiple regression analyses was conducted to examine the association between module use and CPX scores. To control for baseline differences between these two student groups, we used USMLE Step 1 scores and core clerkship ratings as covariates. Clerkship ratings were derived from the medical school’s 10-item summary evaluation form. Students’ average ratings across required core clerkships (Anesthesiology, Family Medicine, Internal Medicine, Neurology, Obstetrics/Gynecology, Pediatrics, Psychiatry, Surgery, and Surgical sub-specialties) were used to create the following covariates: 1) fund of knowledge; 2) history-taking; 3) physical exam; 4) oral presentation; 5) record keeping; 6) problem solving; 7) attributes and responsibility; 8) self-improvement; 9) relationships with patients and families; and 10) relationship with the health care team. We then examined use of the web-based module as the only predictor of CPX scores. Subsequently, covariates were added separately to this baseline model to examine the effect of each covariate on the relationship between module use and CPX scores. P values < 0.05 were considered statistically significant. SPSS Version 18.0 was used for all analyses.

Of the 106 students (72%) who viewed the web-based module before the CPX, 51 (48%) were male, consistent with the class composition. Students logged into the module from 1 to 1346 minutes (mean 86.4, median 29). Eighty-four (79%) students viewed the module within 48 hours of the CPX, and most (75, 71%) viewed it within 24 hours. There were no statistically significant differences in USMLE Step 1 scores or clerkship summary scores between students who viewed the module and those who did not.

Students who did not watch the module had mean CPX communication scores of 67.79% (SD

5.51) and overall scores of 68.80% (SD

6.64). Students who watched the module scored significantly higher on CPX communication (70.46%, p

0.004) and overall (70.92%, p=

0.03) (Table 1) compared to their classmates, even after controlling for USMLE scores and clerkship ratings (Table 2). Use of the module did not significantly affect history/physical examination scores (71.11%, p

0.12) in the baseline model or in the models including covariates. Three of 106 students (3%) who watched the module failed the CPX, compared to 7 of 41 students (17%) who did not watch the module (p

0.002).

Table 1

Correlation Between a Web-Based Module and Clinical Performance Examination (CPX) Scores

Table 2

Association of Web-Based Module Use with Clinical Performance Examination Score in Multivariate Models

DISCUSSION

Students who used an optional web-based SP module, predominantly as a last-minute test preparation tool, performed better on a high-stakes CPX by improving communication scores. Students who used the module scored 2.67 points higher on communication skills and 2.12 points higher overall compared to their peers, without a statistically significant difference in history/physical examination scores. This suggests that the improved overall performance is primarily due to better communication scores. We found no differences in baseline characteristics between students who used the module and those who did not; even after controlling for these variables, the effect persisted.

We previously reported that students who used web-based SP modules to prepare for a high-stakes CPX performed better than historical controls on history-taking and physical examination skills but showed no difference in communication skills.¹³ The current results complement our prior study by demonstrating that web-based SP learning modules can improve CPX communication scores if communication skills are emphasized as part of module content. Both studies used similar preparation formats in terms of difficulty level, demonstration of communication skills, and total interaction time; the main difference was that modules in this study were designed intentionally to address communication scores. Specifically, they contained more SP and faculty commentary than the previous module with more prior in-depth SP training. We infer that our new module improved communication scores because SP commentators highlighted expected communication behaviors. These findings are consistent with studies showing that students’ communication skills improve with feedback from SPs.¹⁰^,¹¹ Although our web-based SP module did not provide students with feedback about their individual performance, the SP commentary may have served as performance benchmark information by elucidating expected communication behaviors.²⁶^,²⁷

In contrast to our prior study, these results demonstrate no difference in history/physical examination performance for students who used the module as a last-minute study tool. Communication performance was the primary focus of the current intervention; we measured history/physical examination scores anticipating that students may have reached a ceiling in improvement in that domain from the in-person preparatory activity six months prior. Another possible explanation for the lack of impact on history/physical examination scores is that students were not required to complete history/physical examination checklists while they watched module cases as in our previous study,¹³ because they had already been exposed to similar checklists during other formative in-person SP exercises. A customized module tailored to meet students’ individual learning goals could further improve performance.

Most students (79%) used the web-based module within 48 hours of the CPX. Some studies have demonstrated that last-minute preparation correlates with higher test performance,¹⁹ but others have not.¹⁷^,¹⁸^,²⁸ It is unclear whether the module actually leads to clinical skills acquisition versus merely prompting students to review and use skills they have already learned (i.e. a test refresher). Successful “cramming” of communication skills prior to the CPX seems unlikely without effective baseline skills. The web-based module may convey information about CPX testing format and expectations so that students understand the CPX better and apply existing skills more effectively. Finally, the module may teach to the test instead of leading to sustained improvement in communication skills. Future studies should examine whether module use or timing correlates with long-term communication skills improvement.

On average, students who used the module had higher overall CPX scores by at least one-third of a standard deviation. The improvement is modest but significant given the high-stakes implications of CPX and USMLE Step 2 Clinical Skills examination performance. This module was a simple, inexpensive intervention that students can access year after year. The degree of improvement in CPX scores attributable to the module is similar to that obtained from a year-long comprehensive communication curricular intervention that required significantly more resources and time.²⁹

Our study has limitations. The intervention occurred at a single institution and was not randomized to eliminate self-selection. Although we included several baseline characteristics of the two student groups including multiple clerkship ratings in our regression analyses, some variables may not correlate with clinical skills performance.³⁰^,³¹ Additionally, we may not have accounted for all differences between groups. It is possible that students who participated in the intervention were more motivated or possessed better baseline communication skills. Motivation is an important component of learning and performance that may not be captured fully by USMLE scores or clerkship ratings, although our clerkship ratings include assessment of professional attributes and self-improvement.³²^–³⁴ Use of the module may also be a surrogate for other test preparation and learning strategies employed by intervention group students. Additionally, it was not possible to correlate time logged into the module with time viewing cases, nor could we determine how many cases students viewed.

In conclusion, our findings suggest that use of a web-based module with narrative commentary from SPs can lead to improved communication skills performance on clinical skills examinations when used as a last-minute test preparatory activity. These results indicate that web-based modules may be a resource-effective way to enhance existing communication skills curricula in medical schools, even when introduced shortly before clinical skills assessments. Future directions include determining whether students who watch the module demonstrate long-term retention of communication skills in clinical practice and optimizing intervention timing to maximize clinical performance and skill retention.

Acknowledgements

We thank Bernie Miller and Jewel Smally for assistance with module creation and examination administration and Christian Burke for technical assistance. This study was supported by a University of California, San Francisco Academy of Medical Educators Innovations Grant.

Conflict of Interest None disclosed.

REFERENCES

1. Stewart MA. Effective physician–patient communication and health outcomes: a review. CMAJ. 1995;152(9):1423–1433. [PMC free article] [PubMed]

2. Zolnierek KB, Dimatteo MR. Physician communication and patient adherence to treatment: a meta-analysis. Med Care. 2009;47(8):826–834. doi: 10.1097/MLR.0b013e31819a5acc. [PMC free article] [PubMed] [Cross Ref]

3. Bredart A, Bouleuc C, Dolbeault S. Doctor-patient communication and satisfaction with care in oncology. Curr Opin Oncol. 2005;17(4):351–354. doi: 10.1097/01.cco.0000167734.26454.30. [PubMed] [Cross Ref]

4. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226–235. doi: 10.1001/jama.287.2.226. [PubMed] [Cross Ref]

5. Kalet A, Pugnaire MP, Cole-Kelly K, Janicik R, Ferrara E, Schwartz MD, et al. Teaching communication in clinical clerkships: models from the Macy initiative in health communications. Acad Med. 2004;79(6):511–520. doi: 10.1097/00001888-200406000-00005. [PubMed] [Cross Ref]

6. Makoul G. Communication skills education in medical school and beyond. JAMA. 2003;289(1):93. doi: 10.1001/jama.289.1.93. [PubMed] [Cross Ref]

7. May W, Park JH, Lee JP. A ten-year review of the literature on the use of standardized patients in teaching and learning: 1996–2005. Med Teach. 2009;31(6):487–492. doi: 10.1080/01421590802530898. [PubMed] [Cross Ref]

8. Lorin S, Rho L, Wisnivesky JP, Nierman DM. Improving medical student intensive care unit communication skills: a novel educational initiative using standardized family members. Crit Care Med. 2006;34(9):2386–2391. doi: 10.1097/01.CCM.0000230239.04781.BD. [PubMed] [Cross Ref]

9. Levenkron JC, Greenland P, Bowley N. Using patient instructors to teach behavioral counseling skills. J Med Educ. 1987;62(8):665–672. [PubMed]

10. Rosen J, Spatz ES, Gaaserud AM, Abramovitch H, Weinreb B, Wenger NS, et al. A new approach to developing cross-cultural communication skills. Med Teach. 2004;26(2):126–132. doi: 10.1080/01421590310001653946. [PubMed] [Cross Ref]

11. Vannatta JB, Smith KR, Crandall S, Fischer PC, Williams K. Comparison of standardized patients and faculty in teaching medical interviewing. Acad Med. 1996;71(12):1360–1362. doi: 10.1097/00001888-199612000-00022. [PubMed] [Cross Ref]

12. Hauer KE, Hodgson CS, Kerr KM, Teherani A, Irby DM. A national study of medical student clinical skills assessment. Acad Med. 2005;80(10 Suppl):S25–S29. doi: 10.1097/00001888-200510001-00010. [PubMed] [Cross Ref]

13. Hauer KE, Chou CL, Souza KH, Henry D, Loeser H, Burke C, et al. Impact of an in-person versus web-based practice standardized patient examination on student performance on a subsequent high-stakes standardized patient examination. Teach Learn Med. 2009;21:284–290. doi: 10.1080/10401330903228307. [PubMed] [Cross Ref]

14. Broekkamp H, Van Hout-Wolters, Bernadette HAM. Students’ adaptation of study strategies when preparing for classroom tests. Educ Psychol Rev. 2007;19(4):401–428. doi: 10.1007/s10648-006-9025-0. [Cross Ref]

15. Yudkowsky R, Downing SM, Ommert D. Prior experiences associated with residents' scores on a communication and interpersonal skill OSCE. Patient Educ Couns. 2006;62(3):368–373. doi: 10.1016/j.pec.2006.03.004. [PubMed] [Cross Ref]

16. Shirar LE, Vu NV, Colliver JA, Barrows HS. A survey of study methods, preparation time, test-taking strategies, and perceptions of test validity on a clinical performance-based examination. Acad Med. 1992;67(10 Suppl):S10–S12. doi: 10.1097/00001888-199210000-00023. [PubMed] [Cross Ref]

17. McGaghie WC, Downing SM, Kubilius R. What is the impact of commercial test preparation courses on medical examination performance? Teach Learn Med. 2004;16(2):202–211. doi: 10.1207/s15328015tlm1602_14. [PubMed] [Cross Ref]

18. Zhang C, Rauchwarger A, Toth C, O'Connell M. Student USMLE step 1 preparation and performance. Adv Health Sci Educ Theory Pract. 2004;9(4):291–297. doi: 10.1007/s10459-004-3925-x. [PubMed] [Cross Ref]

19. Pilotti M, Chodorow M, Petrov R. The usefulness of retrieval practice and review-only practice for answering conceptually related test questions. J Gen Psychol. 2009;136(2):179–203. doi: 10.3200/GENP.136.2.179-204. [PubMed] [Cross Ref]

20. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300(10):1181–1196. doi: 10.1001/jama.300.10.1181. [PubMed] [Cross Ref]

21. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909–922. doi: 10.1097/ACM.0b013e3181d6c319. [PubMed] [Cross Ref]

22. Heine N, Garman K, Wallace P, Bartos R, Richards A. An analysis of standardised patient checklist errors and their effect on student scores. Med Educ. 2003;37(2):99–104. doi: 10.1046/j.1365-2923.2003.01416.x. [PubMed] [Cross Ref]

23. Guiton G, Hodgson CS, Delandshere G, Wilkerson L. Communication skills in standardized-patient assessment of final-year medical students: a psychometric study. Adv Health Sci Educ Theory Pract. 2004;9(3):179–187. doi: 10.1023/B:AHSE.0000038174.87790.7b. [PubMed] [Cross Ref]

24. Tamblyn R, Abrahamowicz M, Dauphinee WD, Hanley JA, Norcini J, Girard N, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288(23):3019–3026. doi: 10.1001/jama.288.23.3019. [PubMed] [Cross Ref]

25. Lang F, McCord R, Harvill L, Anderson DS. Communication assessment using the common ground instrument: psychometric properties. Fam Med. 2004;36(3):189–198. [PubMed]

26. Srinivasan M, Hauer KE, Der-Martirosian C, Wilkes M, Gesundheit N. Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination. Med Educ. 2007;41(9):857–865. doi: 10.1111/j.1365-2923.2007.02818.x. [PubMed] [Cross Ref]

27. Martin D, Regehr G, Hodges B, McNaughton N. Using videotaped benchmarks to improve the self-assessment ability of family practice residents. Acad Med. 1998;73(11):1201–1206. doi: 10.1097/00001888-199811000-00020. [PubMed] [Cross Ref]

28. Werner LS, Bull BS. The effect of three commercial coaching courses on step one USMLE performance. Med Educ. 2003;37(6):527–531. doi: 10.1046/j.1365-2923.2003.01534.x. [PubMed] [Cross Ref]

29. Yedidia MJ, Gillespie CC, Kachur E, Schwartz MD, Ockene J, Chepaitis AE, et al. Effect of communications training on medical student performance. JAMA. 2003;290(9):1157–1165. doi: 10.1001/jama.290.9.1157. [PubMed] [Cross Ref]

30. Rifkin WD, Rifkin A. Correlation between housestaff performance on the United States Medical Licensing Examination and standardized patient encounters. Mt Sinai J Med. 2005;72(1):47–49. [PubMed]

31. Simon SR, Volkan K, Hamann C, Duffey C, Fletcher SW. The relationship between second-year medical students' OSCE scores and USMLE step 1 scores. Med Teach. 2002;24(5):535–539. doi: 10.1080/0142159021000012586. [PubMed] [Cross Ref]

32. Mann KV. Motivation in medical education: how theory can inform our practice. Acad Med. 1999;74(3):237–239. doi: 10.1097/00001888-199903000-00011. [PubMed] [Cross Ref]

33. Sobral DT. What kind of motivation drives medical students' learning quests? Med Educ. 2004;38(9):950–957. doi: 10.1111/j.1365-2929.2004.01913.x. [PubMed] [Cross Ref]

34. Wilkinson TJ, Wells JE, Bushnell JA. Medical student characteristics associated with time in study: is spending more time always a good thing? Med Teach. 2007;29(2–3):106–110. doi: 10.1080/01421590601175317. [PubMed] [Cross Ref]

Articles from Journal of General Internal Medicine are provided here courtesy of
Society of General Internal Medicine