Surgical excellence is traditionally defined in terms of technical performance, with little regard for the importance of interpersonal communication and leadership skills. Studies in the aviation industry have stressed the role of human factors in causing error and, in an attempt to reduce the occurrence of adverse events, led to the organisation of simulation based training scenarios. Similar strategies have recently been employed for the surgical team with the development of a simulated operating theatre project. This enables technical and non-technical performance of the surgeon and circulating staff to be assessed by experts situated in an adjacent control room, and provides an opportunity for constructive feedback. The scenarios have good face validity and junior surgeons can benefit from the process of learning new technical skills in a realistic environment. The effect of external influences such as distractions, new technology, or a crisis scenario can also be defined, with the ultimate aim of reducing the number of adverse events arising in the real operating room.
Non-technical skills are seen as an important contributor to reducing adverse events and improving medical management in healthcare teams. Previous research on the effectiveness of teams has suggested that shared mental models facilitate coordination and team performance. The purpose of the study was to investigate whether demonstrated teamwork skills and behaviour indicating shared mental models would be associated with observed improved medical management in trauma team simulations.
Revised versions of the 'Anesthetists' Non-Technical Skills Behavioural marker system' and 'Anti-Air Teamwork Observation Measure' were field tested in moment-to-moment observation of 27 trauma team simulations in Norwegian hospitals. Independent subject matter experts rated medical management in the teams. An independent group design was used to explore differences in teamwork skills between higher-performing and lower-performing teams.
Specific teamwork skills and behavioural markers were associated with indicators of good team performance. Higher and lower-performing teams differed in information exchange, supporting behaviour and communication, with higher performing teams showing more effective information exchange and communication, and less supporting behaviours. Behavioural markers of shared mental models predicted effective medical management better than teamwork skills.
The present study replicates and extends previous research by providing new empirical evidence of the significance of specific teamwork skills and a shared mental model for the effective medical management of trauma teams. In addition, the study underlines the generic nature of teamwork skills by demonstrating their transferability from different clinical simulations like the anaesthesia environment to trauma care, as well as the potential usefulness of behavioural frequency analysis in future research on non-technical skills.
The aviation domain provides a better analogy for the "temporary" teams that are found in acute medical specialities than industrial or military teamwork research based on established teams. Crew resource management (CRM) training, which emphasises portable skills (for whatever crew a pilot is rostered to on a given flight), has been recognised to have potential application in medicine, especially for teams in the operating theatre, intensive care unit, and emergency room. Drawing on research from aviation psychology that produced the behavioural marker system NOTECHS for rating European pilots' non-technical skills for teamwork on the flightdeck, this paper outlines the Anaesthetists Non-Technical Skills behavioural rating system for anaesthetists working in operating theatre teams. This taxonomy was used as the design basis for a training course, Crisis Avoidance Resource Management for Anaesthetists used to develop these skills, based in an operating theatre simulator. Further developments of this training programme for teams in emergency medicine are outlined.
To review the literature on the use of simulation in the development of non-technical skills in nursing
The potential risks to patients associated with learning 'at the bedside' are becoming increasingly unacceptable, and the search for innovative education and training methods that do not expose the patient to preventable errors continues. All the evidence shows that a significant proportion of adverse events in health care is caused by problems relating to the application of the 'non-technical' skills of communication, teamwork, leadership and decision-making.
Simulation is positively associated with significantly improved interpersonal communication skills at patient handover, and it has also been clearly shown to improve team behaviours in a wide variety of clinical contexts and clinical personnel, associated with improved team performance in the management of crisis situations. It also enables the effective development of transferable, transformational leadership skills, and has also been demonstrated to improve students' critical thinking and clinical reasoning in complex care situations, and to aid in the development of students' self-efficacy and confidence in their own clinical abilities.
High fidelity simulation is able to provide participants with a learning environment in which to develop non-technical skills, that is safe and controlled so that the participants are able to make mistakes, correct those mistakes in real time and learn from them, without fear of compromising patient safety. Participants in simulation are also able to rehearse the clinical management of rare, complex or crisis situations in a valid representation of clinical practice, before practising on patients.
Communication; decision-making; non-technical skills; nurse education; simulation; situation awareness; teamwork; team training; interprofessional.
The use of simulation in the undergraduate nursing curriculum is gaining popularity and is becoming a foundation of many nursing programs. The purpose of this paper is to highlight a new simulation teaching strategy, virtual reality (VR) simulation, which capitalizes on the technological skills of the new generation student. This small-scale pilot study focused on improving interpersonal skills in senior level nursing students using VR simulation. In this study, a repeated-measure design was used to evaluate the effectiveness of VR simulation on improving student's performance over a series of two VR scenarios. Using the Emergency Medicine Crisis Resource Management (EMCRM) tool, student performance was evaluated. Overall, the total EMCRM score improved but not significantly. The subscale areas of communication (P = .047, 95% CI: − 1.06, −.007) and professional behavior (P = .003, 95% CI: − 1.12, −.303) did show a significant improvement between the two scenario exposures. Findings from this study show the potential for virtual reality simulations to have an impact on nursing student performance.
Despite a tremendous growth research in surgical simulation remains uncoordinated and unfocused. The objective of this study was to develop research priorities for surgical simulation.
Using a systematic methodology (Delphi), members of the Association for Surgical Education submitted 5 research questions on surgical simulation. An expert review panel categorized and collapsed the submitted questions and redistributed them to the membership to be ranked using a priority scale from 1(lowest) to 5(highest). The results were analyzed and categorized by consensus in distinct topics.
Sixty members submitted 226 research questions that were reduced to 74. Ratings ranged from 2.19–4.78. Topics included simulation effectiveness and outcomes, performance assessment and credentialing, curriculum development, team training and non-technical skills, simulation center resources and personnel, simulator validation, and other. The highest ranked question was “Does simulation training lead to improved quality of patient care, patient outcomes and safety?”
Research priorities for surgical simulation were developed using a systematic methodology and can be used to focus surgical simulation research in areas most likely to advance the field.
surgical simulation; research priorities; research agenda; Delphi methodology
Medical algorithms, technical skills, and repeated training are the classical cornerstones for successful cardiopulmonary resuscitation (CPR). Increasing evidence suggests that human factors, including team interaction, communication, and leadership, also influence the performance of CPR. Guidelines, however, do not yet include these human factors, partly because of the difficulties of their measurement in real-life cardiac arrest. Recently, clinical studies of cardiac arrest scenarios with high-fidelity video-assisted simulations have provided opportunities to better delineate the influence of human factors on resuscitation team performance. This review focuses on evidence from simulator studies that focus on human factors and their influence on the performance of resuscitation teams. Similar to studies in real patients, simulated cardiac arrest scenarios revealed many unnecessary interruptions of CPR as well as significant delays in defibrillation. These studies also showed that human factors play a major role in these shortcomings and that the medical performance depends on the quality of leadership and team-structuring. Moreover, simulated video-taped medical emergencies revealed that a substantial part of information transfer during communication is erroneous. Understanding the impact of human factors on the performance of a complex medical intervention like resuscitation requires detailed, second-by-second, analysis of factors involving the patient, resuscitative equipment such as the defibrillator, and all team members. Thus, high-fidelity simulator studies provide an important research method in this challenging field.
Cardiopulmonary resuscitation; leadership; team behavior
Few validated instruments exist to measure pediatric code team skills. The goal of this study was to develop an instrument for the assessment of resuscitation competency and self-appraisal using multirater and gap analysis methodologies.
Multirater assessment with gap analysis is a robust methodology that enables the measurement of self-appraisal as well as competency, offering faculty the ability to provide enhanced feedback. The Team Performance during Simulated Crises Instrument (TPDSCI) was grounded in the Accreditation Council for Graduate Medical Education competencies. The instrument contains 5 competencies, each assessed by a series of descriptive rubrics. It was piloted during a series of simulation-based interdisciplinary pediatric crisis resource management education sessions. Course faculty assessed participants, who also did self-assessments. Internal consistency and interrater reliability were analyzed using Cronbach α and intraclass correlation (ICC) statistics. Gap analysis results were examined descriptively.
Cronbach α for the instrument was between 0.72 and 0.69. The overall ICC was 0.82. ICC values for the medical knowledge, clinical skills, communication skills, and systems-based practice were between 0.87 and 0.72. The ICC for the professionalism domain was 0.22. Further examination of the professionalism competency revealed a positive skew, 43 simulated sessions (98%) had significant gaps for at least one of the competencies, 38 sessions (86%) had gaps indicating self-overappraisal, and 15 sessions (34%) had gaps indicating self-underappraisal.
The TPDSCI possesses good measures of internal consistency and interrater reliability with respect to medical knowledge, clinical skills, communication skills, systems-based practice, and overall competence in the context of simulated interdisciplinary pediatric medical crises. Professionalism remains difficult to assess. These results provide an encouraging first step toward instrument validation. Gap analysis reveals disparities between faculty and self-assessments that indicate inadequate participant self-reflection. Identifying self-overappraisal can facilitate focused interventions.
Simulation is increasingly used for teaching medical procedures. The goal of this study was to assess learner preferences for how simulators should be used in a procedural curriculum.
A 26-item survey was constructed to assess the optimal use of simulators for the teaching of medical procedures in an internal medicine residency curriculum. Survey domains were generated independently by two investigators and validated by an expert panel (n = 7). Final survey items were revised based on pilot survey and distributed to 128 internal medicine residents.
Of the 128 residents surveyed, 106 (83%) responded. Most responders felt that simulators should be used to learn technical skills (94%), refine technical skills (84%), and acquire procedural teaching skills (87%).
Respondents felt that procedures most effectively taught by simulators include: central venous catheterization, thoracentesis, intubation, lumbar puncture, and paracentesis. The majority of learners felt that teaching should be done early in residency (97%).
With regards to course format, 62% of respondents felt that no more than 3-4 learners per simulator and an instructor to learner ratio of 1:3-4 would be acceptable.
The majority felt that the role of instructors should include demonstration of technique (92%), observe learner techniques (92%), teach evidence behind procedural steps (84%) and provide feedback (89%). Commonly cited barriers to procedural teaching were limitations in time, number of instructors and simulators, and lack of realism of some simulators.
Our results suggest that residents value simulator-based procedural teaching in the form of small-group sessions. Simulators should be an integral part of medical procedural education.
The traditional training of surgeons focused exclusively on developing knowledge, clinical expertise, and technical (surgical) skills. However, analyses of the reasons for adverse events in surgery have revealed that many underlying causes originate from behavioural or non-technical aspects of performance (eg, poor communication among members of the surgical team) rather than from a lack of surgical (ie, technical) skills. Therefore, technical skills appear to be necessary but not sufficient to ensure patient safety. Paying attention to non-technical skills, such as team working, leadership, situation awareness, decision making, and communication, will increase the likelihood of maintaining high levels of error-free performance. Identification and training of non-technical skills has been developed for high-risk careers, such as civil aviation and nuclear power. Only recently, training in non-technical skills has been adopted by the surgical world and anaesthetists. Non-technical skills need to be tailored to the environment where they are used, and eye surgery has some substantial differences compared with other surgical areas, for example, high volume of surgery, use of local anaesthetics, and very sophisticated equipment. This review highlights the need for identification of the non-technical skills relevant to eye surgeons and promotion of their use in the training of eye surgeons.
surgery; complications; education; training; safety
The major determinant of a patient's safety and outcome is the skill and judgment of the surgeon. While knowledge base and decision processing are evaluated during residency, technical skills—which are at the core of the profession—are not evaluated. Innovative state of the art simulation devices that train both surgical tasks and skills, without risk to patients, should allow for the detection and analysis of errors and "near misses". Studies have validated the use of a sophisticated endoscopic sinus surgery simulator (ES3) for training residents on a procedural basis. Assessments are proceeding as to whether the integration of a comprehensive ES3 training programme into the residency curriculum will have long term effects on surgical performance and patient outcomes. Using various otolaryngology residencies, subjects are exposed to mentored training on the ES3 as well as to minimally invasive trainers such as the MIST-VR. Technical errors are identified and quantified on the simulator and intraoperatively. Through a web based database, individual performance can be compared against a national standard. An upgraded version of the ES3 will be developed which will support patient specific anatomical models. This advance will allow study of the effects of simulated rehearsal of patient specific procedures (mission rehearsal) on patient outcomes and surgical errors during the actual procedure. The information gained from these studies will help usher in the next generation of surgical simulators that are anticipated to have significant impact on patient safety.
Purpose: This evaluation study sought to assess the impact of an evidence-based medicine (EBM) course on students' self-perception of EBM skills, determine their use of EBM skills, and measure their performance in applying EBM skills in a simulated case scenario.
Methods: Pre- and post-surveys and skills tests were developed to measure students' attitudes toward and proficiency in EBM skills. Third-year students completed the voluntary survey and skills test at the beginning and completion of a twelve-week clerkship in internal medicine (IM) co-taught by medical and library faculty. Data were analyzed using the Mann-Whitney U test for a two-tailed test.
Results: A statistically significant increase was found in the students' self-assessment of skills. Students reported using the journal literature significantly more frequently during the clerkship than before, although textbooks remained their number one resource. A majority of students reported frequent use of EBM skills during the clerkship. Statistically significant improvement in student performance was also found on the posttest, although the level of improvement was more modest than that found on the post-surveys.
Conclusion: The introduction of EBM skills to students during a clinical clerkship provides students an opportunity to practice EBM skills and reinforces the use of evidence in making patient-care decisions.
To examine, in a simulated environment, rural nurses’ ability to assess and manage patient deterioration using measures of knowledge, situation awareness and skill performance.
Nurses’ ability to manage deterioration and ‘failure to rescue’ are of significant concern with questions over knowledge and clinical skills. Simulated emergencies may help to identify and develop core skills.
An exploratory quantitative performance review. Thirty five nurses from a single ward completed a knowledge questionnaire and two video recorded simulated scenarios in a rural hospital setting. Patient actors simulated deteriorating patients with an Acute Myocardial Infarction (AMI) and Chronic Obstructive Pulmonary Disease (COPD) as the primary diagnosis. How aware individuals were of the situation (levels of situation awareness) were measured at the end of each scenario.
Knowledge of deterioration management varied considerably (range: 27%-91%) with a mean score of 67%. Average situation awareness scores and skill scores across the two scenarios (AMI and COPD) were low (50%) with many important observations and actions missed. Participants did identify that ‘patients’ were deteriorating but as each patient deteriorated staff performance declined with a reduction in all observational records and actions. In many cases, performance decrements appeared to be related to high anxiety levels. Participants tended to focus on single signs and symptoms and failed to use a systematic approach to patient assessment.
Knowledge and skills were generally low in this rural hospital sample with notable performance decrements as patients acutely declined. Educational models that incorporate high fidelity simulation and feedback techniques are likely to have a significant positive impact on performance.
Education; nursing; patient deterioration; simulation; situation awareness.
Problem: Advance cardiac life support (ACLS) training does not address coordination of team resources to improve the ability of teams to deliver needed treatments reliably and rapidly. Our objective was to use a human simulation training educational environment to develop multidisciplinary team skills and improve medical emergency team (MET) performance. We report findings of a crisis team training course that is focused on organization.
Setting: Large center for human simulation training at a university affiliated tertiary care hospital.
Participants: Ten courses were delivered and 138 clinically experienced individuals were trained (69 critical care nurses, 48 physicians, and 21 respiratory therapists). All participants were ACLS trained and experienced in responding to cardiac arrest situations.
Course design: Each course had four components: (1) a web based presentation and pretest before the course; (2) a brief reinforcing didactic session on the day of the course; (3) three of five different simulated scenarios; each followed by (4) debriefing and analysis with the team. Three of five simulator scenarios were used; scenario selection and order was random. Trainees did not repeat any scenario or role during the training. Participants were video recorded to assist debriefing. Debriefing focused on reinforcing organizational aspects of team performance: assuming designated roles independently, completing goals (tasks) assigned to each role, and directed communication.
Measures for improvement: Participants graded their performance of specific organizational and treatment tasks within specified time intervals by consensus. Simulator "survival" depended on supporting oxygenation, ventilation, circulation within 60 seconds, and delivering the definitive treatment within 3 minutes.
Effects of change: Simulated survival (following predetermined criteria for death) increased from 0% to 89%. The initial team task completion rate was 10–45% and rose to 80–95% during the third session.
Lessons learnt: Training multidisciplinary teams to organize using simulation technology is feasible. This preliminary report warrants more detailed inquiry.
Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.
The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.
Validity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.
We designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.
Transthoracic echocardiography; Echocardiography; Assessment; Ultrasound; Global rating; Checklist
Surgical appraisal and revalidation are key components of good surgical practice and training. Assessing technical skills in a structured manner is still not widely used. Laparoscopic surgery also requires the surgeon to be competent in technological aspects of the operation.
Checklists for generic, specific technical, and technological skills for laparoscopic cholecystectomies were constructed. Two surgeons with >12 years postgraduate surgical experience assessed each operation blindly and independently on DVD. The technological skills were assessed in the operating room.
One hundred operations were analyzed. Eight trainees and 10 consultant surgeons were recruited. No adverse events occurred due to technical or technological skills. Mean interrater reliability was kappa=0.88, P=<0.05. Construct validity for both technical and technological skills between trainee and consultant surgeons were significant, Mann-Whitney P=<0.05.
Our study demonstrates that technical and technological skills can be measured to assess performance of laparoscopic surgeons. This technical and technological assessment tool for laparoscopic surgery seems to have face, content, concurrent, and construct validities and could be modified and applied to any laparoscopic operation. The tool has the possibility of being used in surgical training and appraisal. We aim to modify and apply this tool to advanced laparoscopic operations.
Assessment; Laparoscopic; Technical; Technology
Many medical schools and hospitals throughout the world are equipped with a simulation center for the purpose of training anesthesiologists to perform both technical and non-technical skills. Because induction, maintenance, and emergence of general anesthesia are critical to patient welfare, various simulation mannequins and tools are utilized for the purpose of training anesthesiologists for safer patient care. Traditionally, anesthesia residency training mostly consisted of didactic lectures and observations. After completion of "traditional" training, anesthesia residents were allowed to perform procedures on patients under supervision. However, simulation would be a more effective training tool for which to teach anesthesiologists the skills necessary to perform invasive procedures, such as endotracheal intubation, central venous catheter insertion, and epidural catheter insertion. Recently, non-technical skills, such as the Anesthesia Non-Technical Skills developed by anesthesiologists from Aberdeen University, have been emphasized as an important training resource. Technical skills and non-technical skills can be learned by anesthesiology residents through a standardized and organized simulation program. Such programs would be beneficial in training anesthesia residents to work efficiently as a team in the operation room.
Anesthesia; Non-technical skill; Simulation; Simulator; Training
There is no consensus among emergency medical services (EMS) systems as to the optimal numbers and training of EMS providers who respond to the scene of prehospital cardiac arrests. Increased numbers of providers may improve the performance of cardiopulmonary resuscitation (CPR), but this has not been studied as part of a comprehensive resuscitation scenario.
To compare different all-paramedic crew size configurations on objective measures of patient resuscitation using a high-fidelity human simulator.
We compared two-, three-, and four- person all-paramedic crew configurations in the effectiveness and timeliness of performing basic life support (BLS) and advanced life support (ALS) skills during the first 8 minutes of a simulated cardiac arrest scenario. Crews were compared to determine differences in no-flow fraction (NFF) as a measure of effectiveness of CPR and time to defibrillation, endotracheal intubation, establishment of intravenous access, and medication administration.
There was no significant difference in mean NFF among the two-, three-, and four-provider crew configurations (0.32, 0.26, and 0.27, respectively; p = 0.105). More three- and four-person groups completed ALS procedures during the scenario, but there was no significant difference in time to performance of BLS or ALS procedures among the crew size configurations for completed procedures. There was a trend toward lower time to intubation with increasing group size, though this was not significant using a Bonferroni-corrected p-value of 0.01 (379, 316, and 263 seconds, respectively; p = 0.018).
This study found no significant difference in effectiveness of CPR or in time to performance of BLS or ALS procedures among crew size configurations, though there was a trend toward decreased time to intubation with increased crew size. Effectiveness of CPR may be hindered by distractions related to the performance of ALS procedures with increasing group size, particularly with an all-paramedic provider model. We suggest a renewed emphasis on the provision of effective CPR by designated providers independent of any ALS interventions being performed.
cardiac arrest; emergency medical services; crew configuration; cardiopulmonary resuscitation
Damages and loss of life sustained during an earthquake results from falling structures and flying glass and objects. To address these and other problems, new information technology and systems as a means can improve crisis management and crisis response. The most important factor for managing the crisis depends on our readiness before disasters by useful data.
This study aimed to determine the Earthquake Information Management System (EIMS) in India, Afghanistan and Iran, and describe how we can reduce destruction by EIMS in crisis management.
Materials and Methods:
This study was an analytical comparison in which data were collected by questionnaire, observation and checklist. The population was EIMS in selected countries. Sources of information were staff in related organizations, scientific documentations and Internet. For data analysis, Criteria Rating Technique, Delphi Technique and descriptive methods were used.
Findings showed that EIMS in India (Disaster Information Management System), Afghanistan (Management Information for Natural Disasters) and Iran are decentralized. The Indian state has organized an expert group to inspect issues about disaster decreasing strategy. In Iran, there was no useful and efficient EIMS to evaluate earthquake information.
According to outcomes, it is clear that an information system can only influence decisions if it is relevant, reliable and available for the decision-makers in a timely fashion. Therefore, it is necessary to reform and design a model. The model contains responsible organizations and their functions.
Crisis management; destruction; earthquake; information systems; natural disaster
The transition from basic skills training in a skills lab to procedure training in the operating theater using the traditional master-apprentice model (MAM) lacks uniformity and efficiency. When the supervising surgeon performs parts of a procedure, training opportunities are lost. To minimize this intervention by the supervisor and maximize the actual operating time for the trainee, we created a new training method called INtraoperative Video-Enhanced Surgical Training (INVEST).
Ten surgical residents were trained in laparoscopic cholecystectomy either by the MAM or with INVEST. Each trainee performed six cholecystectomies that were objectively evaluated on an Objective Structured Assessment of Technical Skills (OSATS) global rating scale. Absolute and relative improvements during the training curriculum were compared between the groups. A questionnaire evaluated the trainee’s opinion on this new training method.
Skill improvement on the OSATS global rating scale was significantly greater for the trainees in the INVEST curriculum compared to the MAM, with mean absolute improvement 32.6 versus 14.0 points and mean relative improvement 59.1 versus 34.6% (P = 0.02).
INVEST significantly enhances technical and procedural skill development during the early learning curve for laparoscopic cholecystectomy. Trainees were positive about the content and the idea of the curriculum.
Training; Minimally invasive surgery; Video; INVEST; Operating theater; Cholecystectomy
Proficient delivery of motivational interviewing (MI) is often determined by global rating of relational elements or cumulative tallies of technical elements. Yet limited empirical evidence exists to clarify how relational and technical elements are associated, or if rates of skill indices and their constituent technical elements vary within a clinical encounter.
This study sought to document temporal variance in rates of MI skill indices and their constituent technical elements during brief clinical encounters with a standardized patient wherein delivery was “MI-proficient”, and to distinguish those temporal patterns from those observed in encounters with “MI-inconsistent” delivery.
Data were accessed from a large MI training trial wherein relational and technical elements of MI delivery were scored for 503 recordings of a simulated 20-minute clinical encounter. Notably, independent raters tallied technical elements in 5-minute segments, allowing evaluation of potential variance among the encounter’s quartile intervals. Global ratings of MI spirit identified subsets of recordings with MI-proficient (n = 49) and MI-inconsistent (n = 43) delivery for stratified analyses.
Analyses contrast temporal trajectories of technical aspects of MI-proficient and MI-inconsistent delivery, with the former characterized by: 1) elicitation and reflective listening as primary opening strategies; 2) increased depth of reflective listening as a predominant strategy in subsequent, focused therapeutic discussion; and 3) increased use of elicitation and information provision in change planning as the encounter approached conclusion.
Findings are generally consistent with seminal descriptions of MI (Miller and Rollnick, 1991, 2002), and document temporal aspects of skilful MI delivery in brief encounters.
Motivational interviewing; psychotherapy process; addiction
Defining valid, reliable, defensible, and generalizable standards for the evaluation of learner performance is a key issue in assessing both baseline competence and mastery in medical education. However, prior to setting these standards of performance, the reliability of the scores yielding from a grading tool must be assessed. Accordingly, the purpose of this study was to assess the reliability of scores generated from a set of grading checklists used by non-expert raters during simulations of American Heart Association (AHA) MegaCodes.
The reliability of scores generated from a detailed set of checklists, when used by four non-expert raters, was tested by grading team leader performance in eight MegaCode scenarios. Videos of the scenarios were reviewed and rated by trained faculty facilitators and by a group of non-expert raters. The videos were reviewed “continuously” and “with pauses.” Two content experts served as the reference standard for grading, and four non-expert raters were used to test the reliability of the checklists.
Our results demonstrate that non-expert raters are able to produce reliable grades when using the checklists under consideration, demonstrating excellent intra-rater reliability and agreement with a reference standard. The results also demonstrate that non-expert raters can be trained in the proper use of the checklist in a short amount of time, with no discernible learning curve thereafter. Finally, our results show that a single trained rater can achieve reliable scores of team leader performance during AHA MegaCodes when using our checklist in continuous mode, as measures of agreement in total scoring were very strong (Lin’s Concordance Correlation Coefficient = 0.96; Intraclass Correlation Coefficient = 0.97).
We have shown that our checklists can yield reliable scores, are appropriate for use by non-expert raters, and are able to be employed during continuous assessment of team leader performance during the review of a simulated MegaCode. This checklist may be more appropriate for use by Advanced Cardiac Life Support (ACLS) instructors during MegaCode assessments than current tools provided by the AHA.
simulation; education; checklist; reliability; ACLS
The aims of this research are to begin to understand health care teams in their operational environment, establish metrics of performance for these teams, and validate a series of scenarios in simulation that elicit team and technical skills. The focus is on defining the team model that will function in the operational environment in which health care professionals work.
Simulations were performed across the United States in 70- to 1000-bed hospitals. Multidisciplinary health care teams analyzed more than 300 hours of videos of health care professionals performing simulations of team-based medical care in several different disciplines. Raters were trained to enhance inter-rater reliability.
The study validated event sets that trigger team dynamics and established metrics for team-based care. Team skills were identified and modified using simulation scenarios that employed the event-set-design process. Specific skills (technical and team) were identified by criticality measurement and task analysis methodology.
In situ simulation, which includes a purposeful and Socratic Method of debriefing, is a powerful intervention that can overcome inertia found in clinician behavior and latent environmental systems that present a challenge to quality and patient safety. In situ simulation can increase awareness of risks, personalize the risks, and encourage the reflection, effort, and attention needed to make changes to both behaviors and to systems.
For performance assessment during simulation, mostly observers rate the trainees' performance using checklists. Simulator outcome may provide immediate and objective feedback to the participants but requires additional work for the accurate scenario design. High-fidelity simulators are based on physiologic models and store all changes of the simulator conditions during the scenarios and may therefore be used for the assessment of performance. In the present work, the design of a simulator script for the assessment of resuscitation skills using an Emergency Care Simulator (ECS, METI, Sarasota, Florida) is described.
A standardized resuscitation simulator script and a visual basic-based macro were programmed for the immediate and automated extraction of performance-related variables from the log files. The following parameters were assessed: mean cardiac output, time until return of spontaneous circulation, no-flow-time, no-flow-time fraction, the time until the first defibrillation, the number and fraction of indicated and non-indicated defibrillations. Furthermore, mean deviation of defibrillation interval from the 2 minutes interval, the mean interval of defibrillations and the time until the first administration of epinephrine were calculated. As an example, the results of resuscitation efforts according to 2005 guidelines by five teams that consisted of one emergency physician and two paramedics are presented. No data are provided about its validity and reliability.
The tool can be used to assess adherence to European and American cardiopulmonary resuscitation guidelines (both 2005 and 2010) and to compare simulator outcome if different guidelines are trained and applied according to specific curricula. It represents an example of how simulator outcome can be used for performance assessment and may help to design more complex test-scenarios including the field of critical incidents in anesthesia.
Colonoscopy requires training and experience to ensure accuracy and safety. Currently, no objective, validated process exists to determine when an endoscopist has attained technical competence. Kinematics data describing movements of laparoscopic instruments have been used in surgical skill assessment to define expert surgical technique. We have developed a novel system to record kinematics data during colonoscopy and quantitatively assess colonoscopist performance.
To use kinematic analysis of colonoscopy to quantitatively assess endoscopic technical performance.
Prospective cohort study.
Tertiary-care academic medical center.
This study involved physicians who perform colonoscopy.
Application of a kinematics data collection system to colonoscopy evaluation.
Main Outcome Measurements
Kinematics data, validated task load assessment instrument, and technical difficulty visual analog scale.
All 13 participants completed the colonoscopy to the terminal ileum on the standard colon model. Attending physicians reached the terminal ileum quicker than fellows (median time, 150.19 seconds vs 299.86 seconds; p < .01) with reduced path lengths for all 4 sensors, decreased flex (1.75 m vs 3.14 m; P = .03), smaller tip angulation, reduced absolute roll, and lower curvature of the endoscope. With performance of attending physicians serving as the expert reference standard, the mean kinematic score increased by 19.89 for each decrease in postgraduate year (P < .01). Overall, fellows experienced greater mental, physical, and temporal demand than did attending physicians.
Small cohort size.
Kinematic data and score calculation appear useful in the evaluation of colonoscopy technical skill levels. The kinematic score appears to consistently vary by year of training. Because this assessment is nonsubjective, it may be an improvement over current methods for determination of competence. Ongoing studies are establishing benchmarks and characteristic profiles of skill groups based on kinematics data.