A total of 347 program directors completed the survey, representing a response rate of 10.1%. Program directors from a wide range of disciplines responded to the survey, and specialty distributions were reflective of national data () (36
). For instance, 45 surgery and surgical subspecialty program directors were included in the sample (13% of the sample), compared to 946 in the national sample (19.9% of all residencies). Primary care residencies were slightly overrepresented in our sample. For instance, 100 family medicine residency program directors were included in the sample (28.8% of the sample), compared to 636 in the national sample (13.4% of all residencies). Among the 347 respondents, 44 were identified as ‘surrogates,’ program director-selected surrogates (e.g., assistant/associate residency program directors) who completed the survey on behalf of the residency program director. Among those who completed the survey, 293 respondents completed the section on procedural skills and 284 completed the section on advanced communication skills. The attrition in survey completion was attributed to the length of the survey.
Residency program director respondents in comparison with national sample data, by specialty (n=347)
presents program directors’ opinions about the importance of assessing 28 procedural skills. Program directors considered a number of procedures to be important to assess (sum of ‘important’ and ‘extremely important’ responses): sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of EKG (89.4%), and interpretation of blood gas (88.7%). Skills such as osteopathic manipulative treatment (35.6%), obtaining a blood culture (37.0%), and PPD placement (38.4%) were considered less important.
Fig. 1 Importance of procedural skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ Values reflect (more ...)
With regard to the importance of assessing procedures, agreement varied among program directors of different specialties. ICC is an index representing proportion of the total variance explained by group effects with higher ICC values indicating larger group variation (lower agreement between specialty groups). ICC values ranged from 0.04 to 0.51. Signifying disagreement between program directors from different specialties, high levels of disparity between specialty groups (ICC values >0.30) were found for central line access, lumbar puncture, incision and drainage, splinting/casting, child birth (vaginal) and pelvic exam. Signifying agreement between program directors from different specialties, low levels of disparity (ICC values <0.10) were found for nasogastric tube placement, obtaining blood culture, cardiac resuscitation (BLS), phlebotomy, sterile technique, and injection (IM/SC). Some procedural skills, such as sterile technique and cardiac resuscitation (BLS), displayed both low group disparity and high importance ratings.
Presented in are program directors’ opinions regarding how the 28 procedural skills should be assessed. Program directors overwhelmingly reported that each of the procedures should be assessed in a formative fashion, followed by a combination of both formative and summative assessment. Compared to other procedures, ACLS (24.2%), BLS (23.6%), Neonatal Advanced Life Support (NALS) (21.3%), Pediatric Advanced Life Support (PALS) (20.7%), phlebotomy (15.2%), sterile technique (14.9%), injection (14.7%) and intravenous placement (14.7%) were considered to be procedures amenable to summative assessment.
Fig. 2 Type of assessment for procedural skills by program directors responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement purposes), formative (more ...)
The majority of program directors reported residency program faculty to be the most appropriate for assessing procedural skills (). Only for the phlebotomy skill, medical school faculty were regarded more appropriate than residency program faculty. A small number of program directors reported that resuscitation (ACLS, BLS, PALS and NALS) could be evaluated in a high-stakes testing environment (28.7, 27.6, 27.6 and 27.0%, respectively).
Program directors’ perception of who should be evaluating procedural skillsa
As for the most appropriate time to assess the procedural skills (), program directors reported that assessment of most procedures should be completed at the end of the first year of residency or later. Of the responses for ‘end of the first year of residency,’ the largest rates were reported for suturing (62.8%), lumbar puncture (61.1%), and incision and drainage (60.8%). A small number of skills were considered important to assess prior to the start of residency: BLS (68.9%), sterile technique (67.2%), ACLS (65.9%), and phlebotomy (63.5%).
Program directors’ perception of when procedural skills should be assesseda
Advanced communication and interpersonal skills
displays program directors’ opinions about the importance of assessing 18 advanced communication skills. Program directors considered most communication skills important to assess (sum of ‘important’ and ‘extremely important’ responses). Responses were the highest for demonstrating professionalism (99.6%), respectfulness (98.9%), good listening skills (98.6%), communication with nursing/ancillary staff (98.6%), and empathy (97.9%). The remaining skills each received ratings of importance higher than 78%. The ICC coefficients examining group agreement between program directors of different specialties ranged from near 0 to 0.13. No significant between-group variation was found.
Fig. 3 Importance of advanced communication skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ (more ...)
Regarding the 18 communication skills, program directors overwhelmingly reported that integrative evaluations using both summative and formative assessment should be utilized (). Exclusive summative assessment was not considered a suitable format by most program directors. The majority of the program directors reported residency program faculty to be the most appropriate to assess advanced communication skills ().
Fig. 4 Type of assessment for advanced communication skills by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement (more ...)
Program directors’ perception of who should be evaluating advanced communication skillsa
For all communication skills, except ‘demonstrating respectfulness’, program directors reported the end of first year of residency to be the most appropriate time for evaluation (). Of the responses for ‘end of the first year of residency,’ the largest rates were for handoffs (83.1%), referral to consultants-oral (76.4%) and dictation of medical record (77.5%).
Program directors’ perception of when advanced communication skills should be assesseda