This study aimed to verify the accuracy of self-assessment for surgical involvement and competency in a group of orthopedic surgery residents performing primary TKA and THA. Participants were accurate judges of their surgical involvement with substantial agreement with staff surgeon assessment and a structured surgical skills assessment form. They were also able to accurately self-assess their level of competency in comparison to staff surgeon opinion (κ = 0.67, 95% CI 0.50–0.84). However, residents on average tended to underestimate the level of complexity of cases in comparison to staff surgeon opinion.
The finding that residents underestimate the complexity of primary arthroplasty cases in comparison to staff surgeons is of utmost importance. This may represent a gap in the understanding of principles of arthroplasty or simply a lack of experience. This finding illustrates the importance of acquisition of knowledge, surgical experience and technical skills as a true reflection of competency.
Obtaining an accurate appraisal of resident involvement requires using a measurement tool that represents a true picture of the operative experience. In surgical education studies, staff surgeons have often been used as the expert opinion and gold standard when assessing residents. In our study, staff surgeons were able to assess resident involvement with a high degree of reliability in comparison to an objective surgical assessment form. Moreover, assessment forms, such as the one used in the present study, have previously been shown to have a high interrater reliability.7
These findings indicate that staff surgeons are able to provide a “true” account of the operative experience.
There are several possible explanations that may account for the better-than-expected self-assessment scores. These findings may reflect self-assessment abilities for the performance of a compartmentalized technical task, which may allow for a more objective self-evaluation. Additionally, continuous observation and informal feedback may occur more often for surgical skills than for other clinical skills. Furthermore, exposure to a large volume of operative arthroplasty cases may enable surgical residents to readily identify their level of involvement and competency. These results are not likely applicable to all orthopedic surgeries owing to varying degrees of complexity and residents’ limited exposure to uncommon procedures.
An accurate assessment of involvement in operative cases is valuable information not only to the trainee, but also to the training program. It serves as an important mode of feedback for residents and allows them to chart their involvement in cases over time. The information could also allow educators to identify residents who are not obtaining adequate operative experience and may require further training or remediation before the end of the surgical rotation. Furthermore, it may help to identify hospitals where residents are actively involved in operative cases, which, in turn, may be more ideal settings for training purposes.
There are limitations to this study. Only a subset of residents were eligible for participation in the study, as only 5 hospital sites were used as data collection sites. Furthermore, some residents were not performing arthroplasty during the data collection period. The accuracy of resident self-assessment may have been biased by the knowledge that staff surgeons were rating the residents. Residents may also have altered their responses because they were being evaluated by an objective observer. However, we do not feel these biases undermine the importance of our results. Being evaluated likely stimulated focused self-reflection throughout the cases, which may have improved residents’ ability to self-assess. Our results attest to the ability of residents to perform accurate self-assessments immediately after a focused technical task while being evaluated by a staff surgeon.
In the current study, a single categorical global scale was used to assess competency. This by no means is a comprehensive assessment tool. A single question is not enough to assess overall competency to perform primary TKA and THA, as competency encompasses multiple facets including knowledge acquisition, surgical acumen and technical skills. However, we feel it provides a quick, feasible and valuable assessment of residents’ performance on the cases performed. Simply because a resident feels competent to perform 1 case does not mean they have reached a level of competency at which they can perform all cases of primary arthroplasty, as no 2 cases are alike. If data on a multitude of cases were collected as part of the surgical case log, one could better monitor a global measure of competency over time.