Whereas technical examination measures have been validated,^1–6 procedural knowledge assessments in surgery have not been well documented. With advances in technology, changes in surgical equipment, the development of new surgical techniques and the rotation of new trainees in surgical departments, assessing a surgeon’s procedural competence becomes a valuable tool to ensure adequate performance in the OR. Furthermore, in the current climate of diminishing working hours and shorter training periods, the surgical profession must address the complex issue of how to teach surgery and how to assess an individual’s competence.⁷

An ideal, single objective measurement tool for the assessment of technical and procedural skills continues to elude us.⁸ Indeed, while trying to identify the nontechnical skills required by surgeons in the OR and assessing behavioural marker systems that have been developed for rating surgeons’ nontechnical skills, Yule and colleagues⁹ concluded that further work is required to develop a valid taxonomy for the appraisal of an individual surgeon’s skills.

Adequate acquisition of surgical competence throughout training is fundamental for successful patient care. If technical skills are acquired at a rate proportional to surgical exposure and education, then the knowledge of the steps of the surgical procedure should theoretically increase as well, and we expect that this ability should be similar among surgical trainees of the same rank. The purpose of the present study was to develop an operative knowledge assessment tool to quantify the trend in knowledge acquisition of the procedural steps and to establish the tool’s reliability to investigate the practical applications it may bring to future educational research.

Education behaviourist theory suggests that competence is best assessed by precise measures of performance generally documented by checklists.¹⁰ Therefore, our SPK tools were developed as step-wise assessments of each operation based on Fisher’s Mastery of Surgery, 5th edition.¹¹ The SPK tool lists all of the steps of each procedure, from opening to closure in chronological order and in checklist form. These checklists are then used as a marking grid by each investigator to grade participants.

We interviewed medical students, residents and staff in McGill University’s general surgery program about the SPK tools, asking them to enumerate in chronological order the steps that they were familiar with for each operation. A score was subsequently assigned per procedure, and errors of both commission (unnecessarily added or incorrectly ordered steps) and omission (missed steps) were discounted from the final score. To evaluate inter-rater reliability, interviews were recorded and scored by 2 observers separately. One observer scored the participant over the course of a face-to-face interview. To reduce performance anxiety, time constraints, distractions, overall stress and other potential confounding biases, this observer was chosen from outside of the surgical program and was previously unfamiliar to the study participants. The second observer scored the interview while listening to its audio recording at a later time. A total SPK score was calculated by adding the total score of each procedure out of 100 and dividing by 3.

Our results confirm our expectation that SPK should increase proportionally with level of training and surgical experience. These particular procedures were chosen because of their inherent frequency in the OR, as even entry-level residents are exposed to them on a regular basis. After all, barring breast biopsies and appendectomies, these are the most common operations performed at our centre. The steps of an open appendectomy were chosen as examples for the required task at hand for the residents before engaging in this exercise and, despite being more common than an open hemicolectomy, could thus not be used as part of the assessment tool.

Scores for specific procedures and overall SPK scores increased with level of training, supporting the construct validity of our assessment tool. In addition, with advancement in surgical training, within-group variability in overall SPK scores decreased, indicating a levelling of cognitive ability across members of the same academic rank with increased surgical education and exposure. Medical students displayed similar standard deviation estimates as residents in PGY-2 simply owing to poor performance as a whole cohort, thus displaying low variability among individual results. Finally, by PGY-5, surgeons displayed perfect SPK scores. The procedural knowledge for these basic surgical procedures plateaus, as one would hope, early in surgical training. On our assessment tool, residents in PGY-3 displayed similar scores to those of the staff surgeons. Whether this association holds true for more advanced procedures and whether this knowledge plateau will be maintained at the PGY-3 level as work-hour restrictions impact surgical training remains to be investigated.

There are potentially 2 practical applications of the SPK tool. The first is for curricula planning. The SPK tool can identify and address issues with particular procedures that are not being well taught in a program (e.g., inguinal hernia procedures performed by junior residents). As such, should surgical education curricula need to be reassessed, our tool may offer insight with respect to the current status of procedural knowledge of trainees of a given surgical centre and provide areas of focus on which efforts can be applied. Second, normative SPK scores could be established with additional data, and then the tool could be applied at an individual level, allowing identification of trainees who may need remedial attention to develop this aspect of surgical knowledge.

We believe that ours is the first study to attempt to assess the procedural component of surgical knowledge separately from basic clinical knowledge and surgical judgment or technical knowledge among surgical trainees.^1–6 We hypothesize that procedural knowledge should be similar among surgical trainees of the same rank but acknowledge that substantial variance may exist among surgical residents in how they acquire procedural skills. The purpose of our SPK tool is to measure that procedural knowledge once acquired, regardless of how it was obtained. Inherent heterogeneity in operative exposure may also exist within a program with multiple sites and among teaching staff, which may affect the acquisition of SPK.

The true purpose of creating a tool that measures the procedural knowledge of surgical trainees is not to discriminate against and decrease OR time for trainees who score poorly, but rather to accurately identify whether SPK is an area where we need to focus our resources to improve surgical education. Given the results of our pilot study, we intend to pursue future projects to correlate these results with technical in-training examination scores and validated intraoperative, procedure-specific performance checklist scores. In addition, validation of the SPK tool across multiple institutions and for more complex procedures is required.