|Home | About | Journals | Submit | Contact Us | Français|
To evaluate the effect of simulation-based mastery learning (SBML) on internal medicine residents' lumbar puncture (LP) skills, assess neurology residents' acquired LP skills from traditional clinical education, and compare the results of SBML to traditional clinical education.
This study was a pretest-posttest design with a comparison group. Fifty-eight postgraduate year (PGY) 1 internal medicine residents received an SBML intervention in LP. Residents completed a baseline skill assessment (pretest) using a 21-item LP checklist. After a 3-hour session featuring deliberate practice and feedback, residents completed a posttest and were expected to meet or exceed a minimum passing score (MPS) set by an expert panel. Simulator-trained residents' pretest and posttest scores were compared to assess the impact of the intervention. Thirty-six PGY2, 3, and 4 neurology residents from 3 medical centers completed the same simulated LP assessment without SBML. SBML posttest scores were compared to neurology residents' baseline scores.
PGY1 internal medicine residents improved from a mean of 46.3% to 95.7% after SBML (p < 0.001) and all met the MPS at final posttest. The performance of traditionally trained neurology residents was significantly lower than simulator-trained residents (mean 65.4%, p < 0.001) and only 6% met the MPS.
Residents who completed SBML showed significant improvement in LP procedural skills. Few neurology residents were competent to perform a simulated LP despite clinical experience with the procedure.
Lumbar puncture (LP) is commonly performed by physicians-in-training who often learn vicariously by observing procedures performed by peers. This method leads to uneven skill acquisition and trainee discomfort.1
The American Board of Psychiatry and Neurology,2 the American Association of Neurology,3 and the Accreditation Council for Graduate Medical Education4 have no formal policies to ensure the competence of neurology residents at performing LP. The American Board of Internal Medicine does not require procedural competence in LP, but advises use of simulation training before procedures are performed on patients.5 Despite this, many residents will ultimately practice in a setting where competence in LP is required.
Simulation technology increases procedural skill by providing the opportunity for deliberate practice in a safe environment.6 Researchers at Northwestern University use simulation-based education to train medical residents to mastery skill levels in procedures such as central venous catheter insertion,7–9 thoracentesis,10 and advanced cardiac life support.11 Mastery learning is a stringent form of competency-based education that requires trainees to acquire clinical skill measured against a fixed achievement standard.12 In mastery learning, educational practice time varies but results are uniform. This approach improves patient care outcomes13,14 and is more effective than clinical training alone.15
The current study had 3 aims: first, to evaluate the impact of a simulation-based mastery learning (SBML) intervention on internal medicine residents' LP skills; second, to assess neurology residents' acquired LP skills from traditional clinical education; third, to compare the results of SBML with traditional clinical education for LP skill acquisition.
This was a pretest-posttest design with a comparison group16 of an SBML intervention in LP performance. Postgraduate year (PGY) 1 internal medicine residents who received the intervention were compared to PGY2–4 neurology residents who received clinical training alone. Intervention group (simulator-trained) internal medicine residents underwent a pretest in LP clinical skills using a simulator, completed an SBML educational intervention, and a skills posttest. PGY2–4 neurology residents (traditionally trained) completed an LP clinical skills examination (baseline test) and served as a comparison group (figure 1). The definition of traditional training was conventional clinical neurology residency education at participating institutions.3,4
Intervention group participants were all PGY1 internal medicine residents (n = 58) at Northwestern University from July–September 2010. PGY2–4 neurology residents (n = 49) at 3 university-affiliated academic centers in Chicago, IL, were eligible to serve as the traditionally trained comparison group. Neurology residents from Northwestern University were not eligible to participate in the study because they previously received simulation-based education in LP. Participating neurology residents underwent baseline testing from December 2010 to June 2011.
The Northwestern University Institutional Review Board approved the study, and all participants provided informed consent before participating.
This study was conducted using an LP simulator from Kyoto Kagaku (Torrance, CA) which costs approximately $2,000.17 This LP simulator represents a patient's lower torso and resembles lumbar spinal anatomy closely. The skin and tissues provide life-like resistance to a spinal needle, and allow trainees to measure an opening pressure and collect CSF (figure 2).
Simulator-trained PGY1 internal medicine residents underwent a clinical skills examination using a 21-item LP performance checklist (pretest). Subsequently, they completed a 3-hour education session featuring the New England Journal of Medicine video on LP,18 an interactive LP demonstration, and deliberate practice with directed feedback.19 All participants were required to complete the educational program regardless of pretest performance. A senior faculty member (J.H.B.) with expertise in SBML and LP supervised all sessions. Immediately after the educational intervention, simulator-trained residents were required to meet or exceed a minimum passing score (MPS) on a clinical skills examination (posttest) using the checklist. Residents who did not achieve the MPS engaged in more deliberate practice and were retested until the MPS was reached—the key feature of mastery learning.12 Traditionally trained PGY2–4 neurology residents completed a baseline test in LP using the simulator and 21-item checklist. All skills examinations were video recorded. Residents were blind to individual items on the checklist during testing and training sessions.
We developed the 21-item LP checklist using relevant sources18,20 and rigorous step-by-step methods.7–11,21,22 Each skill or action was listed in order and given equal weight using a dichotomous scoring system (done correctly or incorrectly). The checklist was designed by 1 author with expertise in LP (J.H.B.) and reviewed for completeness and accuracy by 3 authors with expertise in LP, simulation-based education, and checklist design (T.S., E.R.C., D.B.W.). We pilot tested the checklist on a group of 5 medical educators who were not study subjects to estimate reliability and face validity. The final checklist was a consensus between authors.
A multidisciplinary panel of 10 clinical experts determined the MPS using the Angoff (item-based) and Hofstee (group-based) standard setting methods.23,24 The panel was composed of Northwestern University faculty members board certified in neurology (n = 2), anesthesiology (n = 2), emergency medicine (n = 3), and internal medicine (n = 3). Each panelist received instruction in standard setting and used the Angoff and Hofstee methods to assign pass/fail standards. The mean of the Angoff and Hofstee scores was used as the final MPS.24
A second rater rescored a 50% random sample of video recorded clinical skills examinations using the 21-item checklist to assess inter-rater reliability. This examiner was blind to examinee group assignment, pre/posttest status, and the aims of the study.
Participants provided demographic data including age, gender, training program affiliation, and year of training. Each resident assessed procedural confidence on a self-rating scale (0 = not confident to 100 = very confident) and provided information about experience performing LP (number of procedures performed in actual clinical care).
Primary outcome measures were performance of simulator-trained and traditionally trained residents on the clinical skills examinations. Secondary outcome measures were correlations between clinical skills examination performance and resident demographics including year of training, self-confidence, and prior LP experience.
Checklist score reliability was estimated by calculating interrater reliability using the mean kappa coefficient across all checklist items. Internal medicine residents' mean pre- and posttest checklist scores were compared using a paired t test. Mean internal medicine posttest scores were compared to mean neurology baseline scores using the independent t test. χ2 tests were used to compare individual checklist items between the 2 groups. Multiple linear regression was used to evaluate between group differences in posttest (internal medicine) and baseline test scores (neurology) while controlling for age, gender, year of training, self-confidence, and LP procedure experience. Multiple linear regression was also used to assess relationships within groups. In simulator-trained internal medicine residents, we measured associations between posttest clinical skills examination performance and age, gender, reported self-confidence, LP procedure experience, and pretest performance. In the traditionally trained neurology residents, we measured associations between baseline clinical skills examination performance and age, gender, year of training, institution affiliation, reported self-confidence, and LP procedure experience.
All 58 internal medicine residents participated in the study and completed the entire protocol. Thirty-six of 49 (74%) eligible neurology residents participated in the study and completed the entire protocol. The other 26% did not participate due to scheduling difficulties. Fourteen PGY2, 11 PGY3, and 11 PGY4 neurology residents enrolled in the study. Table 1 shows baseline demographics of all 94 residents.
Interrater reliability was high across the 21 checklist items (κn = 0.75). The expert panel set the MPS used in the clinical skills examination at 85%. One of 58 (2%) PGY1 internal medicine residents met the MPS at pretest. Fifty-five of 58 (95%) met the MPS at posttest after the 3-hour intervention. The 3 PGY1 residents who did not meet the MPS at immediate posttest subsequently reached the MPS within 1 hour of further training. In the traditionally trained group of PGY2–4 neurology residents, 2 of 36 (6%) met or exceeded the MPS at baseline testing. Figure 3 is a graphic portrayal of internal medicine and neurology residents' clinical examination scores.
Simulator-trained residents showed a highly significant 107% pretest-to-posttest gain as cohort checklist performance mean scores increased from 46.3% (SD = 17.4) to 95.7% (SD = 4.9; p < 0.001). Traditionally trained neurology residents scored a mean of 65.4% (SD = 14.2) on the checklist. Posttest scores of simulator-trained internal medicine residents were 46% higher than scores of traditionally trained neurology residents (p < 0.001). Results remained significant after controlling for age, gender, year of training, self-confidence, and LP procedural experience (p < 0.001). At the individual item level, simulator-trained residents displayed significantly higher skill on 18 of the 21 items on the LP checklist (table 2).
Regression analysis also showed no relationship within groups between clinical skills examination performance and age, gender, year of training, institution affiliation, reported self-confidence, and LP procedural experience.
Use of SBML featuring an opportunity for deliberate practice and feedback produced a large and consistent skill improvement and almost eliminated performance variability among residents. By contrast, neurology residents from 3 Chicago academic institutions demonstrated poor procedural skill despite standard residency training and substantial experience with actual LPs. Surprisingly, more than half of neurology residents did not correctly identify the anatomic location for an LP and were unable to list routine tests ordered on CSF. These results illustrate the limitations of traditional clinical training and suggest a role for simulation training during neurology residency.
This study contributes to knowledge about procedural skill acquisition among medical trainees. There was no association between resident demographic variables such as age, gender, institution, and year of training and LP procedural skill. Within-group self-confidence and number of LP procedures performed in actual clinical care also did not correlate with measured procedural skill. This confirms results of studies of other medical procedures where trainee clinical experience did not predict procedural skill.7–10,21 In an earlier study of 501 LPs, a surrogate for procedural skill (incidence of post LP headache) was not related to the number of years in practice or training level of the clinician who performed the procedure.25 This consistent finding is a reminder that clinical experience alone is not a proxy for skill26 and that procedural competence should only be assessed using reliable methods involving simulation or clinical observation.
A recently published article calls for additional educational research in neurology.27 Our study evaluated the effect of simulation-based education with mastery learning for resident education using rigorous assessment standards. Earlier studies used simulation to evaluate LP skills of internal medicine,28 emergency medicine,29 and pediatric residents,30 but mastery standards were not imposed. Use of the mastery model improves clinical care in advanced cardiac life support13 and central venous catheter insertion.14 Further study is required on the effect of SBML on neurology resident education.
This study has several limitations. First, the study enrolled a relatively small group of trainees although they represent multiple training programs. Second, we used the LP model for both education and testing, potentially confounding the improvement in posttest scores. However, this does not diminish the pronounced impact simulation-based training had on LP skills. Third, only 74% of neurology residents at the 3 institutions participated, and neurology residents did not complete simulation-based education. We have no reason to believe our results would be different if the entire cohort of neurology residents participated in the educational intervention. Fourth, posttesting occurred immediately after training potentially enhancing recall of the procedure. Further study is required to address long-term retention of LP skills. We previously showed that simulation-based education for central venous catheter insertion and advanced cardiac life support is associated with significant skills retention 1 year after training.31,32 Finally, in service of translational science goals33 we have not yet linked improved LP skills in the simulated environment to improved clinical health care delivery practices and better patient outcomes.
This study demonstrates that simulation-based education boosts LP skills beyond traditional training methods. Few neurology residents in our study were competent to perform a simulated LP despite substantial clinical experience with the procedure. Residents who completed simulation-based education displayed better procedural skills despite significantly less clinical experience with the procedure. In light of these results, we believe a procedural standard should be set and documented for all residents prior to performing an LP in actual clinical care.
The authors thank the Northwestern University internal medicine residents and participating Chicago neurology residency programs for their dedication to education and patient care and Drs. Douglas Vaughan and Mark Williams for their support and encouragement of this work.
Editorial, page 115
J. Barsuk was involved in study concept and design, collecting data, statistical analysis and interpretation of data, study supervision, and drafting and revising the manuscript. E. Cohen was involved in study concept and design, collecting data, statistical analysis and interpretation of data, study supervision, and drafting and revising the manuscript. T. Caprio was involved in collecting and interpreting study data, and revising the manuscript. W. McGaghie was involved in study concept and design, statistical analysis and interpretation of data, study supervision, and revising the manuscript. T. Simuni was involved in study concept and design, interpretation of data, and revising the manuscript. D. Wayne was involved in study concept and design, statistical analysis and interpretation of data, study supervision, obtaining funding, and revising the manuscript.
The authors report no disclosures relevant to the manuscript. Go to Neurology.org for full disclosures.