Growing concern about rising costs and potential harms of medical care has stimulated interest in assessing physicians’ ability to minimize the provision of unnecessary care.
To assess whether graduates of residency programs characterized by low-intensity practice patterns are more capable of managing patients’ care conservatively, when appropriate, and whether graduates of these programs are less capable of providing appropriately aggressive care.
DESIGN, SETTING, AND PARTICIPANTS
Cross-sectional comparison of 6639 first-time takers of the 2007 American Board of Internal Medicine certifying examination, aggregated by residency program (n = 357).
Intensity of practice, measured using the End-of-Life Visit Index, which is the mean number of physician visits within the last 6 months of life among Medicare beneficiaries 65 years and older in the residency program’s hospital referral region.
MAIN OUTCOMES AND MEASURES
The mean score by program on the Appropriately Conservative Management (ACM) (and Appropriately Aggressive Management [AAM]) subscales, comprising all American Board of Internal Medicine certifying examination questions for which the correct response represented the least (or most, respectively) aggressive management strategy. Mean scores on the remainder of the examination were used to stratify programs into 4 knowledge tiers. Data were analyzed by linear regression of ACM(or AAM) scores on the End-of-Life Visit Index, stratified by knowledge tier.
Within each knowledge tier, the lower the intensity of health care practice in the hospital referral region, the better residency program graduates scored on the ACM subscale (P < .001 for the linear trend in each tier). In knowledge tier 4 (poorest), for example, graduates of programs in the lowest-intensity regions had a mean ACM score in the 38th percentile compared with the 22nd percentile for programs in the highest-intensity regions; in tier 2, ACM scores ranged from the 75th to the 48th percentile in regions from lowest to highest intensity. Graduates of programs in low-intensity regions tended, more weakly, to score better on the AAM subscale (in 3 of 4 knowledge tiers).
CONCLUSIONS AND RELEVANCE
Regardless of overall medical knowledge, internists trained at programs in hospital referral regions with lower-intensity medical practice are more likely to recognize when conservative management is appropriate. These internists remain capable of choosing an aggressive approach when indicated.
Important changes are occurring in how the medical profession approaches assessing and maintaining competence. Physician support for such changes will be essential for their success.
To describe physician attitudes towards assessing and maintaining competence.
Cross-sectional internet survey.
Random sample of 1,000 American College of Physicians members who were eligible to participate in the American Board of Internal Medicine Maintenance of Certification program.
Questions assessed physicians’ attitudes and experiences regarding: 1) self-regulation, 2) feedback on knowledge and clinical care, 3) demonstrating knowledge and clinical competence, 4) frequency of use and effectiveness of methods to assess or improve clinical care, and 5) transparency.
Surveys were completed by 446 of 943 eligible respondents (47 %). Eighty percent reported it was important (somewhat/very) to receive feedback on their knowledge, and 94 % considered it important (somewhat/very) to get feedback on their quality of care. However, only 24 % reported that they receive useful feedback on their knowledge most/all of the time, and 27 % reported receiving useful feedback on their clinical care most/all of the time. Seventy-five percent agreed that participating in programs to assess their knowledge is important to staying up-to-date, yet only 52 % reported participating in such programs within the last 3 years. The majority (58 %) believed physicians should be required to demonstrate their knowledge via a secure examination every 9–10 years. Support was low for Specialty Certification Boards making information about physician competence publically available, with respondents expressing concern about patients misinterpreting information about their Board Certification activities.
A gap exists between physicians’ interest in feedback on their competence and existing programs’ ability to provide such feedback. Educating physicians about the importance of regularly assessing their knowledge and quality of care, coupled with enhanced systems to provide such feedback, is needed to close this gap.
board certification; physician attitudes; feedback; medical education; transparency
Although competency-based medical education has become the standard for physician training in the West, many developing countries have not yet adopted competency-based training. In 2009 in the United Arab Emirates, the government regulatory and operational authorities for healthcare in Abu Dhabi mandated a wide-scale reform of the emirate’s postgraduate residency programs to the competency-based framework of the newly formed Accreditation Council for Graduate Medical Education-International (ACGME-I). This article briefly describes the rationale for competency-based medical education and provides an overview of the transition from traditional, time-based residency training to competency-based postgraduate medical education for the Pediatrics residency programs in Abu Dhabi. We will provide data on the initial impact of this transition on resident performance and patient outcomes in a Pediatrics residency program in an academic medical center in the United Arab Emirates.
Pediatrics; Competency-based medical education; Outcomes-based training; International medical education; Residency
Understanding clinical reasoning is essential for patient care and medical education. Dual-processing theory suggests that nonanalytic reasoning is an essential aspect of expertise; however, assessing nonanalytic reasoning is challenging because it is believed to occur on the subconscious level. This assumption makes concurrent verbal protocols less reliable assessment tools.
Functional magnetic resonance imaging was used to explore the neural basis of nonanalytic reasoning in internal medicine interns (novices) and board-certified staff internists (experts) while completing United States Medical Licensing Examination and American Board of Internal Medicine multiple-choice questions.
The results demonstrated that novices and experts share a common neural network in addition to nonoverlapping neural resources. However, experts manifested greater neural processing efficiency in regions such as the prefrontal cortex during nonanalytical reasoning.
These findings reveal a multinetwork system that supports the dual-process mode of expert clinical reasoning during medical evaluation.
Dual-process theory; expertise; functional MRI; medical education; neural efficiency; nonanalytical reasoning
Understanding clinical reasoning is essential for patient care and medical education. Dual‐processing theory suggests that nonanalytic reasoning is an essential aspect of expertise; however, assessing nonanalytic reasoning is challenging because it is believed to occur on the subconscious level. This assumption makes concurrent verbal protocols less reliable assessment tools.
Functional magnetic resonance imaging was used to explore the neural basis of nonanalytic reasoning in internal medicine interns (novices) and board‐certified staff internists (experts) while completing United States Medical Licensing Examination and American Board of Internal Medicine multiple‐choice questions.
The results demonstrated that novices and experts share a common neural network in addition to nonoverlapping neural resources. However, experts manifested greater neural processing efficiency in regions such as the prefrontal cortex during nonanalytical reasoning.
These findings reveal a multinetwork system that supports the dual‐process mode of expert clinical reasoning during medical evaluation.
Dual‐process theory; expertise; functional MRI; medical education; neural efficiency; nonanalytical reasoning
The educational milestones were designed as a criterion-based framework for assessing resident progression on the 6 Accreditation Council for Graduate Medical Education competencies.
We obtained feedback on, and assessed the construct validity and perceived feasibility and utility of, draft Internal Medicine Milestones for Patient Care and Systems-Based Practice.
All participants in our mixed-methods study were members of competency committees in internal medicine residency programs. An initial survey assessed participant and program demographics; focus groups obtained feedback on the draft milestones and explored their perceived utility in resident assessment, and an exit survey elicited input on the value of the draft milestones in resident assessment. Surveys were tabulated using descriptive statistics. Conventional content analysis method was used to assess the focus group data.
Thirty-four participants from 17 programs completed surveys and participated in 1 of 6 focus groups. Overall, the milestones were perceived as useful in formative and summative assessment of residents. Participants raised concerns about the length and complexity of some draft milestones and suggested specific changes. The focus groups also identified a need for faculty development. In the exit survey, most participants agreed that the Patient Care and Systems-Based Practice Milestones would help competency committees assess trainee progress toward independent practice.
Draft reporting milestones for 2 competencies demonstrated significant construct validity in both the content and response process and the perceived utility for the assessment of resident performance. To ensure success, additional feedback from the internal medicine community and faculty development will be necessary.
To study relationships between clinical skill measures assessed at the beginning of general internists' careers and their career outcomes and practice characteristics.
General Internist Community Tracking Study Physician Survey respondents (2000–2001, 2004–2005) linked with residency program evaluations and American Board of Internal Medicine board certification examination score records; n = 2,331.
Cross-sectional regressions of career outcome and practice characteristic measures on board examination scores/success, residency evaluations interacted with residency type, and potential confounding variables.
Failure to achieve board certification was associated with $27,206 (18 percent, p < .05) less income and 14.9 percent more minority patients relative to physicians scoring in the bottom quartile on their initial examination who eventually became certified (p < .01). Other skill measures were not associated with income. Scoring in the top rather than bottom quartile on the board certification examination was associated with 9 percent increased likelihood of reporting high career satisfaction (p < .05). Among physicians trained in community hospital residency programs, lower evaluations were associated with 14.5 percent higher share of minority patients (p < .05). Both skill measures were associated with practice type.
There are associations between early career skill measures and career outcomes. In addition, minority patients are more likely to be treated by physicians with lower early career clinical skills measures than nonminority patients.
Clinical skill indicators; board certification; practice characteristics; career outcomes; general internists
The internal medicine milestones were developed to advance outcomes-based residency training and will play an important role in the next accreditation system.
As an element of our program's participation in the internal medicine educational innovations project, we implemented a milestones-based evaluation process in our general medicine and pulmonary-critical care rotations on July 1, 2010.
Outcomes assessed included survey-rated acceptability to participating faculty, residents, and clinical competency committee members.
Faculty and residents agreed that the milestones promoted a common understanding of what knowledge, skills, and attitudes should be displayed at particular points in residents' professional development and enhanced evaluators' ability to provide specific performance feedback. Most residents and faculty members agreed that the milestones promoted fairness and uniformity in the evaluation process. Clinical competency committee members agreed the milestones improved the quality of information available for deliberations and resulted in more uniform promotion standards. Faculty rated the use of too many milestones per form/tool at a mean of 7.3 (where 1 was minimally problematic, and 10 was maximally problematic) and the potential for evaluator fatigue (mean, 8.2) as the most significant challenges to the use of milestones. Eight of 12 faculty members would recommend milestones in other programs; 4 were uncertain.
Despite logistical challenges, educators and trainees found that milestones promoted a common understanding of what knowledge, skills and attitudes should be displayed at particular stages of training; permitted greater specificity in performance feedback; and enhanced uniformity and fairness in promotion decisions.
The American Board of Internal Medicine approved the use of Practice Improvement Modules (PIMs) to help training programs teach and assess practice-based learning and improvement (PBLI) and systems-based practice (SBP).
We surveyed individuals who ordered a PIM in a residency or fellowship training program between June 2006 and August 2009. The 43 programs that volunteered to participate completed a 30-minute anonymous online survey.
Program directors or associate program directors led the PIM process in 30 programs (70%). Trainees' degrees of involvement in PIMs were highly variable between programs, and several respondents felt that trainees were either not sufficiently engaged or not engaged with enough consistency. The most common activity for trainee involvement was data collection through patient surveys or chart review, although only 17 programs (40%) provided protected time for this activity. Few trainees participated in higher level activities such as data analysis or identification for areas of improvement or were given leadership roles; yet most respondents reported that completing the PIM helped trainees learn basic principles of QI and develop competence in PBLI and SBP and that PIM completion improved the program's ability to develop QI initiatives and resulted in program or institutional improvements, including sustainable improvement in patient care. Most respondents reported that the outcome warranted the effort to complete PIMs.
PIMs may be a valuable but underused educational experience for trainees as well as training programs. Focusing on particular themes and facets of PIMs may facilitate implementation.
To examine the importance of patient-based measures and practice infrastructure measures of the patient-centered medical home (PCMH).
A total of 3,671 patient surveys of 202 physicians completing the American Board of Internal Medicine (ABIM) 2006 Comprehensive Care Practice Improvement Module and 14,457 patient chart reviews from 592 physicians completing ABIM's 2007 Diabetes and Hypertension Practice Improvement Module.
We estimated the association of patient-centered care and practice infrastructure measures with patient rating of physician quality. We then estimated the association of practice infrastructure and patient rating of care quality with blood pressure (BP) control.
Patient-centered care measures dominated practice infrastructure as predictors of patient rating of physician quality. Having all patient-centered care measures in place versus none was associated with an absolute 75.2 percent increase in the likelihood of receiving a top rating. Both patient rating of care quality and practice infrastructure predicted BP control. Receiving a rating of excellent on care quality from all patients was associated with an absolute 4.2 percent improvement in BP control. For reaching the maximum practice-infrastructure score, this figure was 4.5 percent.
Assessment of physician practices for PCMH qualification should consider both patient based patient-centered care measures and practice infrastructure measures.
Patient-centered care; practice infrastructure; medical home; blood pressure control
Burnout is prevalent in residency training and practice and is linked to medical error and suboptimal patient care. However, little is known about how burnout affects clinical reasoning, which is essential to safe and effective care. The aim of this study was to examine how burnout modulates brain activity during clinical reasoning in physicians. Using functional Magnetic Resonance Imaging (fMRI), brain activity was assessed in internal medicine residents (n = 10) and board-certified internists (faculty, n = 17) from the Uniformed Services University (USUHS) while they answered and reflected upon United States Medical Licensing Examination and American Board of Internal Medicine multiple-choice questions. Participants also completed a validated two-item burnout scale, which includes an item assessing emotional exhaustion and an item assessing depersonalization. Whole brain covariate analysis was used to examine blood-oxygen-level-dependent (BOLD) signal during answering and reflecting upon clinical problems with respect to burnout scores. Higher depersonalization scores were associated with less BOLD signal in the right dorsolateral prefrontal cortex and middle frontal gyrus during reflecting on clinical problems and less BOLD signal in the bilateral precuneus while answering clinical problems in residents. Higher emotional exhaustion scores were associated with more right posterior cingulate cortex and middle frontal gyrus BOLD signal in residents. Examination of faculty revealed no significant influence of burnout on brain activity. Residents appear to be more susceptible to burnout effects on clinical reasoning, which may indicate that residents may need both cognitive and emotional support to improve quality of life and to optimize performance and learning. These results inform our understanding of mental stress, cognitive control as well as cognitive load theory.
expertise; burnout; clinical reasoning; cognitive load; fMRI
To describe parents’ and adolescents’ perceptions about vaccination.
Qualitative interviews of 22 mothers/grandmothers and 25 10- to 14-year-olds.
Themes emerged in 3 focus areas. (a) Understanding: Both adults and adolescents had difficulty understanding concepts of risks, benefits, prevention, and vaccination. (b) Decision making: Adults saw vaccination as an opportunity to help their adolescent develop skills for transition to adulthood. Adolescents worried about being lied to (reinforced by being told “it won’t hurt”), physical pain, and cleanliness. (c) Preventing sexually transmitted infections: Adults were divided between those who felt their child would not need such a vaccine and those who wanted to “be safe” to protect their child in the future.
At the same time that even basic concepts about vaccination should be explained to both adults and adolescents, adolescence represents a time for learning about responsible decision making. Discussion regarding the risks and benefits of vaccines can be part of transitioning to adult decision making.
Maintenance of certification examination performance is associated with quality of care. We aimed to examine relationships between electronic medical knowledge resource use, practice characteristics and examination scores among physicians recertifying in internal medicine.
We conducted a cross-sectional study of 3,958 United States physicians who took the Internal Medicine Maintenance of Certification Examination (IM–MOCE) between January 1, 2006 and December 31, 2008, and who held individual licenses to one or both of two large electronic knowledge resource programs. We examined associations between physicians’ IM–MOCE scores and their days of electronic resource use, practice type (private practice, residency teaching clinic, inpatient, nursing home), practice model (single or multi-specialty), sex, age, and medical school location.
In the 365 days prior to the IM–MOCE, physicians used electronic resources on a mean (SD, range) of 20.3 (36.5, 0–265) days. In multivariate analyses, the number of days of resource use was independently associated with increased IM–MOCE scores (0.07-point increase per day of use, p = 0.02). Increased age was associated with decreased IM–MOCE scores (1.8-point decrease per year of age, p < 0.001). Relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM–MOCE scores by 29.1 and 20.0 points, respectively (both p < 0.001).
Frequent use of electronic resources was associated with modestly enhanced IM–MOCE performance. Physicians involved in residency education clinics and hospital inpatient practices had higher IM–MOCE scores than physicians working in private practice settings.
Internal Medicine Maintenance of Certification Examination; IM–MOCE; scores; education
Quality improvement (QI) activities are an important part of residency training. National studies are needed to inform best practices in QI training and experience for residents. The impact of the Institutional Review Board (IRB) process on such studies is not well described.
This observational study looked at time, length, comfort level, and overall quality of experience for 42 residency training programs in obtaining approval or exemption for a nationally based educational QI study.
For the 42 programs in the study, the time period to IRB approval/exemption was highly variable, ranging from less than 1 week to 56.5 weeks; mean and median time was approximately 18 weeks (SD, 10.8). Greater reported comfort with the IRB process was associated with less time to obtain approval (r = −.50; P < .01; 95% CI, −0.70 to −0.23). A more positive overall quality of experience with the IRB process was also associated with less time to obtain IRB approval (r = −.60; P < .01; 95% CI, −0.74 to −0.36).
The IRB process for residency programs initiating QI studies shows considerable variance that is not explained by attributes of the projects. New strategies are needed to assist and expedite IRB processes for QI research in educational settings and reduce interinstitutional variability and increase comfort level among educators with the IRB process.
To investigate the feasibility, reliability, and validity of comprehensively assessing physician-level performance in ambulatory practice.
Data Sources/Study Setting
Ambulatory-based general internists in 13 states participated in the assessment.
We assessed physician-level performance, adjusted for patient factors, on 46 individual measures, an overall composite measure, and composite measures for chronic, acute, and preventive care. Between- versus within-physician variation was quantified by intraclass correlation coefficients (ICC). External validity was assessed by correlating performance on a certification exam.
Data Collection/Extraction Methods
Medical records for 236 physicians were audited for seven chronic and four acute care conditions, and six age- and gender-appropriate preventive services.
Performance on the individual and composite measures varied substantially within (range 5–86 percent compliance on 46 measures) and between physicians (ICC range 0.12–0.88). Reliabilities for the composite measures were robust: 0.88 for chronic care and 0.87 for preventive services. Higher certification exam scores were associated with better performance on the overall (r = 0.19; p <.01), chronic care (r = 0.14, p = .04), and preventive services composites (r = 0.17, p = .01).
Our results suggest that reliable and valid comprehensive assessment of the quality of chronic and preventive care can be achieved by creating composite measures and by sampling feasible numbers of patients for each condition.
Comprehensive assessment; quality of care; primary care; composite measures
The Internal Medicine In-Training Exam (IM-ITE) assesses the content knowledge of internal medicine trainees. Many programs use the IM-ITE to counsel residents, to create individual remediation plans, and to make fundamental programmatic and curricular modifications.
To assess the association between a multiple-choice testing program administered during 12 consecutive months of ambulatory and inpatient elective experience and IM-ITE percentile scores in third post-graduate year (PGY-3) categorical residents.
Retrospective cohort study.
One hundred and four categorical internal medicine residents. Forty-five residents in the 2008 and 2009 classes participated in the study group, and the 59 residents in the three classes that preceded the use of the testing program, 2005–2007, served as controls.
A comprehensive, elective rotation specific, multiple-choice testing program and a separate board review program, both administered during a continuous long-block elective experience during the twelve months between the second post-graduate year (PGY-2) and PGY-3 in-training examinations.
We analyzed the change in median individual percent correct and percentile scores between the PGY-1 and PGY-2 IM-ITE and between the PGY-2 and PGY-3 IM-ITE in both control and study cohorts. For our main outcome measure, we compared the change in median individual percentile rank between the control and study cohorts between the PGY-2 and the PGY-3 IM-ITE testing opportunities.
After experiencing the educational intervention, the study group demonstrated a significant increase in median individual IM-ITE percentile score between PGY-2 and PGY-3 examinations of 8.5 percentile points (p < 0.01). This is significantly better than the increase of 1.0 percentile point seen in the control group between its PGY-2 and PGY-3 examination (p < 0.01).
A comprehensive multiple-choice testing program aimed at PGY-2 residents during a 12-month continuous long-block elective experience is associated with improved PGY-3 IM-ITE performance.
Internal Medicine In-Training Exam; multiple-choice testing; medical knowledge
To examine attitudes and knowledge about vaccinations in postpartum mothers.
This cross-sectional study collected data via written survey to postpartum mothers in a large teaching hospital in Connecticut. We used multivariable analysis to identify mothers who were less trusting with regard to vaccines.
Of 228 mothers who participated in the study, 29% of mothers worried about vaccinating their infants: 23% were worried the vaccines would not work, 11% were worried the doctor would give the wrong vaccine, and 8% worried that “they” are experimenting when they give vaccines. Mothers reported that the most important reasons to vaccinate were to prevent disease in the baby (74%) and in society (11%). Knowledge about vaccination was poor; e.g., 33% correctly matched chicken pox with varicella vaccine. Mothers who were planning to breastfeed (P = .01), were primiparous (P = .01), or had an income <$40,000 but did not receive support from the women, infants, and children (WIC) program were less trusting with regard to vaccines (P = .03). Although 70% wanted information about vaccines during pregnancy, only 18% reported receiving such information during prenatal care.
Although the majority of infants receive vaccines, their mothers have concerns and would like to receive immunization information earlier. Mothers who are primiparous, have low family incomes but do not qualify for the WIC program, or are breastfeeding may need special attention to develop a trusting relationship around vaccination. Mothers would benefit from additional knowledge regarding the risks and benefits of vaccines particularly during prenatal care.
Vaccinations; Mothers; Attitudes; Trust; Postpartum period
Assessing physicians’ clinical performance using statistically sound, evidence-based measures is challenging. Little research has focused on methodological approaches to setting performance standards to which physicians are being held accountable.
Determine if a rigorous approach for setting an objective, credible standard of minimally-acceptable performance could be used for practicing physicians caring for diabetic patients.
Retrospective cohort study.
Nine hundred and fifty-seven physicians from the United States with time-limited certification in internal medicine or a subspecialty.
The ABIM Diabetes Practice Improvement Module was used to collect data on ten clinical and two patient experience measures. A panel of eight internists/subspecialists representing essential perspectives of clinical practice applied an adaptation of the Angoff method to judge how physicians who provide minimally-acceptable care would perform on individual measures to establish performance thresholds. Panelists then rated each measure’s relative importance and the Dunn–Rankin method was applied to establish scoring weights for the composite measure. Physician characteristics were used to support the standard-setting outcome.
Physicians abstracted 20,131 patient charts and 18,974 patient surveys were completed. The panel established reasonable performance thresholds and importance weights, yielding a standard of 48.51 (out of 100 possible points) on the composite measure with high classification accuracy (0.98). The 38 (4%) outlier physicians who did not meet the standard had lower ratings of overall clinical competence and professional behavior/attitude from former residency program directors (p = 0.01 and p = 0.006, respectively), lower Internal Medicine certification and maintenance of certification examination scores (p = 0.005 and p < 0.001, respectively), and primarily worked as solo practitioners (p = 0.02).
The standard-setting method yielded a credible, defensible performance standard for diabetes care based on informed judgment that resulted in a reasonable, reproducible outcome. Our method represents one approach to identifying outlier physicians for intervention to protect patients.
Electronic supplementary material
The online version of this article (doi:10.1007/s11606-010-1572-x) contains supplementary material, which is available to authorized users.
clinical performance assessment; standard setting; composite measures; diabetes care
Many have called for ambulatory training redesign in internal medicine (IM) residencies to increase primary care career outcomes. Many believe dysfunctional, clinic environments are a key barrier to meaningful ambulatory education, but little is actually known about the educational milieu of continuity clinics nationwide.
We wished to describe the infrastructure and educational milieu at resident continuity clinics and assess clinic readiness to meet new IM-RRC requirements.
National survey of ACGME accredited IM training programs.
Directors of academic and community-based continuity clinics.
Two hundred and twenty-one out of 365 (62%) of clinic directors representing 49% of training programs responded. Wide variation amongst continuity clinics in size, structure and educational organization exist. Clinics below the 25th percentile of total clinic sessions would not meet RRC-IM requirements for total number of clinic sessions. Only two thirds of clinics provided a longitudinal mentor. Forty-three percent of directors reported their trainees felt stressed in the clinic environment and 25% of clinic directors felt overwhelmed.
The survey used self reported data and was not anonymous. A slight predominance of larger clinics and university based clinics responded. Data may not reflect changes to programs made since 2008.
This national survey demonstrates that the continuity clinic experience varies widely across IM programs, with many sites not yet meeting new ACGME requirements. The combination of disadvantaged and ill patients with inadequately resourced clinics, stressed residents, and clinic directors suggests that many sites need substantial reorganization and institutional commitment.New paradigms, encouraged by ACGME requirement changes such as increased separation of inpatient and outpatient duties are needed to improve the continuity clinic experience.
clinic; resident education; ACGME; primary care
Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one’s self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment.
Self-appraisal; Feedback; Confidence; Self-assessment; Performance improvement
The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project requires that residency program directors objectively document that their residents achieve competence in 6 general dimensions of practice.
In November 2007, the American Board of Internal Medicine (ABIM) and the ACGME initiated the development of milestones for internal medicine residency training. ABIM and ACGME convened a 33-member milestones task force made up of program directors, experts in evaluation and quality, and representatives of internal medicine stakeholder organizations. This article reports on the development process and the resulting list of proposed milestones for each ACGME competency.
The task force adopted the Dreyfus model of skill acquisition as a framework the internal medicine milestones, and calibrated the milestones with the expectation that residents achieve, at a minimum, the “competency” level in the 5-step progression by the completion of residency. The task force also developed general recommendations for strategies to evaluate the milestones.
The milestones resulting from this effort will promote competency-based resident education in internal medicine, and will allow program directors to track the progress of residents and inform decisions regarding promotion and readiness for independent practice. In addition, the milestones may guide curriculum development, suggest specific assessment strategies, provide benchmarks for resident self-directed assessment-seeking, and assist remediation by facilitating identification of specific deficits. Finally, by making explicit the profession's expectations for graduates and providing a degree of national standardization in evaluation, the milestones may improve public accountability for residency training.
The Outcome Project requires high-quality assessment approaches to provide reliable and valid judgments of the attainment of competencies deemed important for physician practice.
The Accreditation Council for Graduate Medical Education (ACGME) convened the Advisory Committee on Educational Outcome Assessment in 2007–2008 to identify high-quality assessment methods. The assessments selected by this body would form a core set that could be used by all programs in a specialty to assess resident performance and enable initial steps toward establishing national specialty databases of program performance. The committee identified a small set of methods for provisional use and further evaluation. It also developed frameworks and processes to support the ongoing evaluation of methods and the longer-term enhancement of assessment in graduate medical education.
The committee constructed a set of standards, a methodology for applying the standards, and grading rules for their review of assessment method quality. It developed a simple report card for displaying grades on each standard and an overall grade for each method reviewed. It also described an assessment system of factors that influence assessment quality. The committee proposed a coordinated, national-level infrastructure to support enhancements to assessment, including method development and assessor training. It recommended the establishment of a new assessment review group to continue its work of evaluating assessment methods. The committee delivered a report summarizing its activities and 5 related recommendations for implementation to the ACGME Board in September 2008.