|Home | About | Journals | Submit | Contact Us | Français|
This article presents a comparison of the effectiveness of traditional clinical education toward skill acquisition goals versus simulation-based medical education (SBME) with deliberate practice (DP).
This is a quantitative meta-analysis that spans twenty years, 1990 to 2010. A search strategy involving three literature databases, 12 search terms, and four inclusion criteria was used. Four authors independently retrieved and reviewed articles. Main outcome measures were extracted to calculate effect sizes.
Of 3,742 articles identified, 14 met inclusion criteria. The overall effect size for the 14 studies evaluating the comparative effectiveness of SBME compared to traditional clinical medical education was 0.71 (95% confidence interval, 0.65–0.76; P < .001).
Although the number of reports analyzed in this meta analysis is small, these results show that SBME with DP is superior to traditional clinical medical education in achieving specific clinical skill acquisition goals. SBME is a complex educational intervention that should be introduced thoughtfully and evaluated rigorously at training sites. Further research on incorporating SBME with DP into medical education is needed to amplify its power, utility, and cost-effectiveness.
This article addresses the comparative effectiveness of traditional methods of clinical medical education, especially the Halstedian “see one, do one, teach one” approach,1 versus simulation-based medical education (SBME) with deliberate practice (DP). SBME2–4 engages learners in lifelike experiences with varying fidelity designed to mimic real clinical encounters. DP embodies strong and consistent educational interventions grounded in information processing and behavioral theories of skill acquisition and maintenance.5–8 DP has at least nine elements (List 1).
The goal of DP is constant skill improvement, not just skill maintenance. The power of DP has been demonstrated in many professional domains including sports, commerce, performing arts, science, and writing.9 Research shows that DP is a much more powerful predictor of professional accomplishment than experience or academic aptitude.6
Comparative effectiveness research (CER), also known as patient-centered outcomes research, refers to studies that compare the benefits and liabilities of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions.10–12 The aim is to “make head-to-head comparisons of different health care interventions [that] outline the effectiveness—or benefits and harms—of treatment options.”13 Conventional clinical treatment options include drugs, surgery, rehabilitation, and preventive interventions that (1) improve patient health, (2) contribute to quality of life, and (3) boost longevity. Treatment options grounded in comparative research have efficacy in controlled laboratory settings and are also effective in clinical patient care where health care delivery, its receipt, and patient adherence vary widely.14
U.S. CER policies have been published recently by the Institute of Medicine (IOM) under the title, Knowing What Works in Health Care: A Roadmap for the Nation.15 Complementary work by the U.S. Agency for Healthcare Research and Quality (AHRQ) outlines comparative health care research priorities.16 These expressions of CER policies and priorities focus on conventional treatment options. However, they do not address the value of a skillful medical and health professions workforce and the importance of its education for the delivery of effective health care. We assert that human capital, embodied in competent physicians and other health care professionals, is an essential feature of health care delivery even though IOM policies and AHRQ research priorities are silent about the contribution of health professions education to health care delivery.
The purpose of medical education at all levels is to prepare physicians with the knowledge, skills, and features of professionalism needed to deliver quality patient care. Medical education research seeks to make the enterprise more effective, efficient, and economical. Short and long-run goals of research in medical education are to show that educational programs contribute to physician competence measured in the classroom, simulation laboratory, and patient care settings. Improved patient outcomes linked directly to educational events are the ultimate goal of medical education research and qualify this scholarship as translational science.17
This article reviews and evaluates evidence about the comparative effectiveness of SBME with DP versus traditional clinical education. The goal of the study is to perform a “head-to-head” comparison of these two educational methods toward the goal of clinical skill acquisition. This is a quantitative meta-analysis of SBME research that spans twenty years, from 1990 to 2010. The comparative review is selective and critical. We also believe it is exhaustive because the number of existing comparative studies is small.
This article was prepared using most reporting conventions described in the MOOSE (Meta-analysis of Observational Studies in Epidemiology) Statement18 and the QUOROM Statement19 for reports of meta-analyses of randomized controlled trials.
Quantitative research synthesis begins with a systematic search of existing literature. Our search strategy covered three literature databases (MEDLINE, Web of Knowledge, PsychINFO) and employed 12 single search terms and concepts (clinical education, clinical outcomes, deliberate practice, fellows, mastery learning, medical education, medical simulation training, medical students, patient outcomes, quality of care, residents, simulator) and their Boolean combinations. We searched publications from 1990 to April, 2010. We also reviewed reference lists of all selected manuscripts to identify additional reports. The intent was to perform a detailed and thorough search of peer reviewed publications that have been judged for academic quality to evaluate the comparative effectiveness of SBME with DP versus traditional clinical education.
Four authors (WCM, SBI, ERC, and DBW) independently retrieved articles using the 12 search terms and reviewed titles and abstracts. The full text of each article thought to be eligible for the study was also reviewed by the four authors. Four inclusion criteria were used to select the pool of eligible studies for the final analysis: each study had to (1) feature SBME with DP as an educational intervention; (2) have an appropriate comparison group featuring traditional, clinical education or a pre-intervention baseline measurement for single group designs; (3) assess trainee skill acquisition rather than knowledge or attitudes; and (4) present sufficient data to enable effect size (ES) calculation. Conflicts were resolved by consensus.
The following data were extracted from selected studies: (1) study design (i.e., randomized trial, cohort study, case-control study, pre-post baseline study); (2) sample size; (3) outcome variables (i.e., what competency was assessed); and (4) reported skill outcome values (mean and standard deviation).
Two authors (WCM, ERC) abstracted information about the main outcome measure for each study and performed statistical analyses. For studies with a comparison group, effect sizes were calculated as the difference in means between the intervention and control groups, divided by the pooled standard deviation. For these studies the intervention group comprised all medical trainees receiving SBME with DP while the control group included all medical trainees receiving traditional clinical education. Effect size calculations for pre-post baseline studies (within-subjects designs) were performed by dividing the mean difference between post and pretest outcomes by the pretest standard deviation. When sufficient data were not available we used t-test values and degrees of freedom to calculate effect size correlation.20 Effect sizes were derived for individual studies and then combined across research reports. Effect size estimates were corrected for sample size. Correlation coefficients were calculated from effect size estimates. For each outcome of interest, pooled estimates and 95% confidence intervals (CIs) of effect size correlations were calculated using an inverse-variance weighted random effects meta-analysis.20 Statistical significance was defined as P < .05. Data analyses were done using Comprehensive Meta-Analysis, Version 2 (Biostat, Englewood, NJ).
Each reviewer screened the 3,742 citations identified using our search strategies. We excluded papers if they were not original research, did not involve medical learners, or were not published in English. This left us with a subset of 328 articles for further review. This group was evaluated in detail by each author to reach consensus on whether the articles met the inclusion criteria described above until full consensus was reached. Of this group, 314 were excluded from the final analysis. These 314 papers were excluded because they did not feature DP, did not have a comparison group, the intervention was not simulation-based, or because data were insufficient. Several studies were included even though the term “deliberate practice” was not used in text. In these cases, descriptions of the type, intensity, and quality of the educational interventions were synonymous with the DP model.
The search strategy and inclusion and exclusion criteria resulted in a final set of 14 research reports addressing medical clinical skill acquisition.21–34 The 14 journal articles are listed in Table 1 in four descending categories ordered by the rigor of their research design.14
A total of 633 learners participated, including 389 internal medicine, surgical, and emergency medicine residents, 226 medical students, and 18 internal medicine fellows. The SBME studies address a number of competencies and skills including advanced cardiac life support, laparoscopic surgical techniques, central venous catheter insertion, cardiac auscultation, and thoracentesis. Six of the studies demonstrated improvement in laparoscopic surgical skills including cholecystectomy, instrument and camera navigation and handling, and suturing live tissues.22–26,34 Two of the studies showed improved performance and adherence to American Heart Association advanced cardiac life support guidelines including responses during actual patient codes.21,30 Cardiac auscultation skills including identification and interpretation of heart sounds and murmurs were improved among medical students and residents in two studies.27,29 Four of the studies demonstrated improved ability among residents and fellows to perform three invasive procedures (hemodialysis catheter insertion, thoracentesis, central venous catheter insertion).28,31–33
Results from the meta-analysis of the 14 studies comparing SBME with DP versus traditional clinical education are displayed quantitatively and as a forest plot in Figure 1.35 The figure shows CER results with 95% CIs for each individual study and overall. The magnitude of boxes shown is a relative sample size indicator. Without exception and with very high confidence the CER data favor SBME with DP in comparison to traditional clinical education or a pre-intervention baseline measure. Every study exceeds the null value without statistical overlap. The overall effect size correlation (0.71) qualifies as a large effect size36 and summarizes the quantitative power of SBME with DP educational interventions for skill acquisition compared to traditional clinical education.
Only a small number of studies were identified that address “head to head” comparative effectiveness of SBME with DP and traditional clinical education or a pre-intervention baseline. However, the results of this meta-analysis are clear and unequivocal. The meta-analytic outcomes favoring SBME with DP are powerful, consistent, and without exception. There is no doubt that SBME is superior to traditional clinical education for acquisition of a wide range of medical skills represented in this study: advanced cardiac life support, laparoscopic surgery, cardiac auscultation, hemodialysis catheter insertion, thoracentesis, and central venous catheter insertion. We are confident that demonstrations of the utility and cost effectiveness37 of educational interventions featuring SBME with DP will increase as the technology is applied to other skill acquisition and maintenance opportunities in health care.
A growing body of evidence shows that clinical skills acquired in medical simulation laboratory settings transfer directly to improved patient care practices and better patient outcomes. Examples of improved patient care practices linked directly to SBME include studies of better management of difficult obstetrical deliveries (e.g., shoulder dystocia),38 laparoscopic surgery,39 and bronchoscopy.40 Better patient outcomes linked directly to SBME have been reported in several studies using historical control groups that address reductions in catheter-related bloodstream infections41 and postpartum outcomes (e.g., brachial palsy injury,38 neonatal hypoxic-ischemic encephalopathy42) among newborn infants. Such work suggests that traditional, clinical education is insufficient if the goal is skill acquisition and downstream patient safety.
The power and utility of SBME with DP toward the goal of skill acquisition is no longer in doubt, especially compared to traditional models of clinical education. However, we also acknowledge that SBME with DP is a complex intervention that has a variety of elements including a long implementation chain, features that mutate as a result of refinement and adaptation to local circumstances, and represent open systems that feed back on themselves. Pawson and colleagues assert, “As interventions are implemented, they change the conditions that made them work in the first place.”43 There is much to learn about the organizational effects of SBME with DP on the medical schools and postgraduate residency programs that adopt these new educational technologies. Best practices to develop faculty to teach using SBME with DP also warrant attention. Finally, we agree with Eva, who asserts that the medical education community needs to “move away from research that is intended to prove the effectiveness of our educational endeavours and towards research that aims to understand the complexity inherent in those activities.”44
The results of this CER study underscore the importance of using new ways to invest and grow human capital embodied in a highly skilled workforce to improve health care delivery and patient safety. CER policies and priorities should endorse the importance of medical and health professions education in addition to investments in basic science research, drug design, medical device fabrication, and other established mechanisms of medical translational science. A recent conference hosted by Harvard Medical School involving educational leaders from eight other U.S. medical schools concluded, “…investigation of the efficacy of simulation in enhancing the performance of medical school graduates received the highest [priority] score.”45 Enhancement of the traditional clinical educational model with evidence-based practices like SBME with DP should be a high priority for medical education policy and research.
This study has several limitations. First, the final number of research studies contained in the meta-analysis (14) is small even though the data involve 633 medical learners. Second, the meta-analysis primarily addresses acquisition of medical procedural skills. It does not cover acquisition of many other clinical skills, such as judgment under pressure, medical decision-making, situation awareness, teamwork, or professional behavior. It is not known if the DP model is suited to these skills and research is warranted. Third, we are aware of many potential sources of bias that may affect meta-analyses of quasi-experimental research including cohort, case-control, and pre-post studies.46 Despite these limitations, the direction, strength, and consistency of results from this study indicate the outcomes are robust. We conclude that DP is a key variable in rigorous SBME research and training. Further CER on SBME with DP versus traditional clinical education will refute or endorse this conclusion.
The authors acknowledge the graphical expertise of Sheila Macomber. They thank Douglas Vaughan, MD, at Northwestern University and Michael S. Gordon, MD, PhD, at the University of Miami for their support of this work.
Funding Support: This research was supported in part by the Jacob R. Suker, MD, professorship in medical education and by grant UL 1 RR025741 from the National Center for Research Resources, National Institutes of Health (NIH). (Dr. McGaghie) The NIH had no role in the preparation, review, or approval of the manuscript.
Other disclosure: None.
Ethical approval: Not applicable.
*Source: McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education. Effectiveness of continuing medical education: American College of Chest Physicians evidence-based educational guidelines. CHEST 2009; 135: 62S–68S.
†Source: Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006; 21: 251–256.
Dr. William C. McGaghie, Jacob R. Suker, MD, professor of medical education, professor of preventive Medicine, and director of evaluation, NUCATS Institute, Northwestern University Feinberg School of Medicine, Chicago, Illinois.
Dr. S. Barry Issenberg, Professor of medicine and assistant director, Gordon Center for Research in Medical Education, University of Miami Miller School of Medicine, Miami, Florida.
Ms. Elaine R. Cohen, Research assistant, department of medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois.
Dr. Jeffrey H. Barsuk, Assistant professor of medicine, division of hospital medicine Northwestern University Feinberg School of Medicine, Chicago, Illinois.
Dr. Diane B. Wayne, Associate professor of medicine and director, internal medicine residency training program, Northwestern University Feinberg School of Medicine, Chicago, Illinois.