|Home | About | Journals | Submit | Contact Us | Français|
Randomized, prospective study within an orthopedic surgery resident program at a large urban academic medical center.
To develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs to augment the clinical education of orthopedic and neurosurgical physicians-in-training.
Twenty participants comprising senior medical students and orthopedic surgical residents. Participants were randomized to control (n = 9) or intervention (n = 11) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 40-minute bioskills training module, while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported by each participant (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Decompression Score metrics.
When compared with the control group, the intervention group yielded a significant mean improvement in OSATS (P = .022) and PPDIS (P = .0001) scores. The Objective Decompression Scores improved in the intervention group with a trend toward significance (P = .058).
We conclude that a concise lumbar laminectomy bioskills training session can be a useful educational tool for to augment clinical education. Although no direct clinical correlation can be concluded from this study, the improvement in trainee’s technical and procedural skills suggests that Sawbones training modules can be an efficient and effective tool for teaching fundamental spine surgical skills outside of the operating room.
The challenge tasked to orthopedic and neurosurgical program directors centers on providing a comprehensive curriculum to physicians-in-training to ensure independence for core surgical competencies at completion of training. Furthermore, outside pressures adversely affecting trainees’ surgical education include work-hour restrictions, declining budgets, and medicolegal ramifications of surgical complications. Similar to many other disciplines involving the performance of complex skills and tasks, surgical skills demonstrate a “learning curve,” requiring frequent repetition of the learned skill to obtain competence. To this end, Gonzalvo et al concluded that approximately 40 to 80 pedicle screws were needed before a spine fellow was able to achieve reproducible accuracy consistent with attending staff skill.1
Simulated training sessions are commonly utilized for skill acquisition and early development in high-risk professions. However, critical analysis of simulated training suggests that it may not purely be the quantity of practice, but instead the quality that is essential to improving performance of complex skills. Studying elite concert musicians, Ericsson and colleagues reported that subjects touted “deliberate practice” to be essential to improved outcomes, requiring well-defined tasks, detailed and immediate feedback, and the opportunity for solitary practice to develop skills.2,3 Similarly, pilots using these principles with simulated crisis scenarios performed significantly better with assessment and decision-making for once-in-a-career crisis scenarios when compared with untrained controls.4
Historically, surgical simulation education (aka “bioskills”) programs have been poorly studied because of small sample sizes and inadequate outcome instruments to evaluate efficacy. Using a Sawbones model for a posterior laminectomy with 8 subjects, Harrop et al noted no statistically significant differences in pre- and postdidactic scores and in the Objective Structured Assessment of Technical Skills (OSATS).5 Conversely, Gottschalk and colleagues reported that a 3D simulation-training group exhibited significantly improved trajectory of cervical lateral mass screw placement.6
Bioskills training modules can be practical and effective forms of education within the time and budgetary constraints of many orthopedic and neurosurgical training programs. With budgetary and time constraints in mind, the purpose of our study was to develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs in order to augment the clinical education of orthopedic and neurosurgical physicians-in-training. We hypothesize that the institution of our concise training session would lead to improvement in simulated surgical skills performance compared to self-directed study.
A total of 20 participants were enrolled from an orthopedic residency training program, with levels of training represented ranging from fourth-year medical students to post graduate year (PGY) 5 orthopedic residents. The study design met the criteria for an institutional review board exemption at our institution, participation was optional, and participants gave informed consent prior to participation in the study. Sample size was limited to 20 participants due to funding constraints. Participants were stratified into 3 groups based on level of experience (fourth-year medical students, PGY 1&2, and PGY 3-5) to facilitate block randomization controlling for level of experience. Control (n = 9) and intervention (n = 11) groups were instructed not to disclose their assignments. All participants were instructed to perform a pretest 1-level laminectomy evaluation at the interspace of the third and fourth lumbar vertebra limited by a 10-minute time period and scored by an evaluator blinded to group assignments (a fellowship trained orthopedic spine attending). The 10-minute time period was instituted due to laboratory time constraints and felt to be adequate as prototype testing of the procedure required approximately 5 minutes. Surgical instruments required for the bony decompression component of a laminectomy procedure were made available for the participants, including kerrisons, curettes, rongeurs, pituitaries, high-speed surgical drills with burr tips, Woodson elevators, nerve root retractors, and nerve hooks. A prefabricated Sawbones model with lumbar spondylosis displaying bony overgrowth narrowing the central canal, lateral recesses, and foramina simulating degenerative spinal stenosis was used for testing in order to maintain a standardized testing experiences among participants. The Sawbone was affixed to a plywood base with 3″ drywall screws to stabilize the model. A miniature balloon was inserted into the central canal at the tested levels and filled with water at light tension to simulate the dural sac. The dorsal exposure was limited by a cardboard box and packing peanuts to simulate the view afforded by an open posterior lumbar approach.
Following the pretest, the blinded evaluator left the testing area. The control group participants then were instructed to leave the room for 40 minutes and allowed to read standard texts regarding performance of lumbar decompressions. The intervention group subsequently underwent a 40-minute training module conducted by a senior resident consisting of a 10-minute multimedia presentation highlighting the steps of the procedure, proper use of surgical instruments, and review of the 3-dimensional degenerative and normal anatomy using Sawbones models. Following the presentation, intervention group participants practiced lumbar decompressions on Sawbones models for 30 minutes with active feedback on surgical technique and adequacy of decompression from the presenter.
The posttest laminectomy evaluation for both groups was performed at the adjacent caudal level (L 4/5) and conducted immediately after intervention in a similar manner as the pretest (Figure 1). The participants and evaluator were blinded to pre- and posttest performance evaluations and no feedback was given to participants from the blinded evaluator either during or after testing sessions. Total completion time for the study was 75 minutes. Following the completion of the study, control group participants were offered an opportunity to complete the intervention training session for their educational benefit. An online survey was also conducted following the laboratory to inquire as to the participant’s satisfaction with the laboratory and identify what aspects were the most useful to their education. Included questions focused on participants’ preference for similar bioskills training modules be integrated into their education (yes/no response) as well as determining which components of the intervention were found to be valuable to their education (graded 1 to 10 using a global rating scale with scores of “1” and “10” reflecting “no utility” and “significant utility,” respectively).
The cost of the testing apparatus for each participant was approximately US$240. The Sawbones degenerative lumbar stenosis model was the most expensive component at US$231 per model. Plywood, screws, cardboard, and packing peanuts averaged less than US$10 per participant. Assembly time for each model was approximately 10 minutes.
Participants’ performance were evaluated using a modified OSATS scale, which has been previously utilized to assess proficiency with surgical tools, technical skills, and performance of the procedure.7–16 The participants completed a Physician Performance Diagnostic Inventory Scale (PPDIS) following the pre- and posttests, a survey given to assess the participants’ self-reported knowledge and skill. Last, following the pre- and posttesting, the blinded evaluator used a novel outcome metric to objectively grade the completeness of the decompression (Objective Decompression Score [ODS]), dividing the Sawbone model into discrete sections (central canal, lateral recesses, and foramina) and assigning scores (ranging 1-4) for completeness of decompression at each section. The ODS was added to provide an objective evaluation of the performed bony decompression in order to comprehensively and accurately evaluate participants. We believed quantitatively assessing the adequate performance of lumbar decompression is an important adjunct to the OSATS or PPDIS metrics and no comparable objective technical skill outcome metric was available in the reported literature.
Following data collection, the subsection scores of each outcome metric (OSATS, ODS, and PPDIS) for each participant’s pre- and posttest were recorded as composite scores (eg, a score of “2” for each of the 5 OSATS subsections would be reported as a score of “10”). The composite scores describe an overall assessment of participants’ performance and allow analysis of the generalized treatment effect from pre- to posttesting for each outcome metric. The global rating scale-type format within our outcome metrics was amenable to treatment as continuous variables and suitable for composite analysis. (See Supplemental Material for OSATS, PPDIS, and ODS grading forms, available in the online version of the article.)
The overall control and intervention groups were compared using the recorded outcome metrics (OSATS, PPDIS, and ODS). In order to determine the size of effect resulting from the intervention or control group, the mean change in participants’ composite scores from pre- to posttesting was analyzed. The control group’s mean change from pre- to posttesting accounted for the expected improvement seen with repeat testing without interval simulation training. Furthermore, using the mean change from pre- to posttesting rather than reporting absolute pre- or posttest scores controlled for participants’ baseline technical skills and proficiency with open lumbar laminectomy. A 2-sample t test assuming nonparametric distributions was used to compare the mean change from pre- to posttest scores for each outcome measure within the control and intervention groups. We avoided attributing significance to subgroup analyses due to insufficient subgroup sample sizes (n = 3 or 4).
The intervention group yielded a significant improvement in OSATS scores compared with the control group (P = .022). The intervention group demonstrated a 31.4% improvement of OSATS scores (mean +6.273, SD 4.474, potential score range 5-25) compared to the control group’s improvement of 4.4% (mean +0.889, SD 3.855). Similarly, the intervention group reported a significant improvement in PPDIS scores of 27.7% (mean +5.545, SD 2.66, potential score range 5-25), compared to 0.6% for the control group (mean +0.112, SD 0.354; P = .0001). ODS also improved in the intervention group, with an improvement of 35.4% (mean +6.364, SD 4.945, potential score range 6-24) that trended toward significance compared to the control group’s improvement of 12.3% (mean 2.222, SD 2.333; P = .058). See Tables 1 and and22.
Participants reported high levels of satisfaction with the bioskills training module on the post-laboratory participant questionnaire. All participants (n = 15) who underwent the intervention (11 randomized in protocol, 4 voluntary following completion of lab) found it valuable to their education and recommended integration of similar bioskills modules into surgical education. Participants identified hands-on training with the Sawbones and surgical instruments (average rating 9.4), receiving a focused presentation on how to perform the procedure (average rating 9.27), and obtaining focused feedback during the intervention (average rating 8.93) as the most useful aspects of the intervention. No participants noted issues due to time constraints imposed on pre- and posttesting.
The education of spine surgeons-in-training is a challenging task. Trainees require autonomy to develop and refine complex technical skills. However, this must be balanced with providing excellent patient care and minimizing technical errors resulting in complications. Spine surgery requires a highly technical set of surgical skills with a known increased propensity for technical error and complications while acquiring these skills.1,17,18 Similar to other high-risk professions requiring performance of complex skills, simulation training demonstrates value for trainees through safe development of fundamental skills, with the goal of improving familiarity and performance with subsequent real-world experiences.19,20 While discussing the rationale of their institution’s implementation of surgical simulation training, Gasco et al stated,
If a single patient can benefit from this adjuvant form of training, then it becomes unarguably worthwhile. Every patient is important, and high-quality, standardized surgery is expected of us and those we have the obligation to educate, with results that are carried across to all surgeons.21(S44-S45)
We believe the utility of our Sawbones laminectomy training sessions is to augment the clinical experience. At our institution, Sawbones simulation training is used to teach fundamental surgical skills to improve familiarity with instrument handling to allow an emphasis on teaching higher level surgical skills and concepts in the operating room. In a similar manner, our reported improvement in OSATS and objective decompression scores with the Sawbones training module suggests improved familiarity and confidence with surgical instruments, steps of the procedure, and 3-dimensional patho-anatomy, potentially allowing operating room education to focus on higher-level clinical decision making.
Few studies have attempted to quantitatively analyze the effect of skills training on the performance of simulated spine procedures. Historically speaking, spine bioskills modules have had difficulty demonstrating significant improvement for its participants.22–24 Some important limitations from these studies include small sample size, standardization of outcome measures, and assessment tools. However, several recent studies of orthopedic simulation programs have been able to demonstrate significantly improved cervical lateral mass and thoracic screw placement.6,8 It appears that those studies that utilize a standardized method of assessment and intervention during a protected time period (approximately 1 hour) have the highest success rate of demonstrating significant improvement in performance.
This study functions to introduce our novel approach to evaluate the efficacy and utility of bioskills training for simulated lumbar laminectomy. Our 40-minute intervention performed by a senior resident using a multimedia presentation and hands-on Sawbones skills practice was derived from commonly observed practices in bioskills training and can be readily reproducible by residency training programs. The intervention group’s significant improvements in PPDIS and OSATS scores and trends toward improvement with ODS suggest the relevance of bioskills training modules for surgical education. The results of our study propose that a concise, inexpensive, well-designed training module can improve participants’ technical skills and proficiency with simulated performance of lumbar laminectomy. However, the unvalidated testing and evaluation techniques used in our study limits definitively ascribing significance to the observed improvements within the intervention group.
Several limitations were encountered in the development and implementation of our model. While focusing enrollment on a specific level of training (ie, junior residents) would improve homogeneity of groups and decrease confounding variables, the inclusion of various levels of training was necessary to enroll 20 participants since the study was limited to a single orthopedic residency training program. Although the block randomization allowed for groups balanced for levels of experience, the breadth of experience represented within small sample sizes creates heterogeneity within the groups. The use of Sawbones models for skills assessment for pre- and posttesting provided a relatively affordable, reproducible, and controlled environment for data collection. However, the fidelity of the training model can be improved in subsequent studies through addition of surrounding anatomic structures and improving the consistency of Sawbones bone material to more closely replicate bone. Additionally, the transferability of subjective and performance skills obtained during a Sawbones training lab to clinical performance has yet to be proven and was not specifically addressed in our study. Furthermore, we were not able to assess the durability of the intervention effect as posttesting occurred immediately after the intervention. We are currently in the process of assessing the durability of the learned skills and knowledge retention over time compared to traditional programs alone.
While the OSATS metric has been extensively utilized and validated for the evaluation of resident’s performance of technical skills,7,9,10,25 its use as a tool to evaluate the efficacy of simulation laboratories has only been recently described.11–16,26 Anderson and colleagues suggested that any study that utilized the OSATS as a stand-alone assessment measure ignored critical objective assessments regarding the end product of the tested technical procedure.27 For this reason, we supplemented the OSATS evaluation with a blinded, objective assessment of the decompression procedure. While the intervention group’s ODS failed to reach statistical significance, we believe the strong trend toward improvement supports the validity of the significant improvement in OSATS scores seen with the intervention group. Additionally, although the PPDIS can experience response bias and the Hawthorne effect, the outcome metric has been used in similar medical education studies and provides insight on participants’ perceived efficacy and value of the bioskills simulation training.21,28 Furthermore, these assessment tools seem to be best suited for parametric distributions of technical skills and may exhibit floor/ceiling effects with assessing participants at the extreme ends of the spectrum. Last, although our sample size (n = 20) represented a majority of our training program, this limited subgroup statistical analyses. The use of novel evaluation metrics precluded our ability to perform power analysis and sample size determination. However, the effect sizes and variance found with our study will facilitate future power analyses conducted for future bioskills training studies.
Despite the aforementioned limitations, we believe Sawbones simulation training can contribute positively to surgical education. Due to the favorable data obtained from this lumbar laminectomy training module study, we are currently developing similar spine bioskills research studies. The improvement in trainees’ subjective and objective skills assessments with a brief approximately 40-minute Sawbones training session suggests acquisition of essential fundamental surgical skills can be efficiently and effectively conducted outside of the operating room. Furthermore, trainees reported both a high level of satisfaction with simulation training and improved self-rated instrument handling and procedural skills following simulation training. Although no clinical correlation of learned surgical skills was conducted, we believe simulation-training sessions can be relevant and effective for teaching surgical skills and techniques that can subsequently be refined in the operating room.
A brief lumbar laminectomy bioskills training module can be a useful resident and student educational tool, leading to significantly improved scores in residents’ subjective performance and trending toward improvement in objective grading of completeness of decompression. We believe that the observed improvement in participant skills has educational relevance and that similar simulation training modules can be powerful tools to augment trainee’s surgical education.
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: An internal institutional grant was secured for acquisition of simulation models.
Supplemental Material: Supplementary material for this article is available online.