Over the past decade, EBM has become a major driving force world wide, impacting medical education, policymaking, and research. The teaching of evidence-based medicine has been increasingly integrated into curricula at all levels of medical education as advocated by the Medical School Objectives Program developed by the Association of American Medical Colleges (AAMC) [8
]. Like most medical schools, the Carver College of Medicine at the University of Iowa has integrated EBM into its curriculum and our data indicates that students who are soon to graduate can demonstrate proficiency at critically appraising an article about diagnostic testing. But in a simulated clinical encounter, few students are able take the last step of using EBM in diagnosing medical illness. We found almost uniform failure of our students to correctly revise a pretest probability of disease given a test result despite their earlier demonstration of competency with Bayes' Theorem on their Laboratory Medicine clerkship. This finding raises questions about whether our students can fully utilize their EBM training in the clinical setting.
Our finding that incoming residents demonstrate similar levels of skill at critically appraisal and also are unable to revise a pretest probability implies that our medical students' skill deficit is not solely due to a local curricular problem. Because the residents had only just graduated from 12 different US medical schools, this finding suggests that many medical school graduates are able to critically appraise articles on diagnostic testing but few are able to take the next step- that of revising the probability of disease given a test result.
There are few other assessments of the EBM skills of graduating medical students using simulated clinical encounters. In an earlier study, 3rd
year medical students demonstrated good performance at critically appraisal [9
], a finding which is similar to ours. These results generally replicated classroom-based studies on the success of critical appraisal instruction [10
]. To our knowledge, only one other study has investigated whether students are able to integrate critically appraised information about a diagnostic test into clinical decision making. The results of this earlier study also raised concerns of students' abilities to transfer their EBM skills to simulated clinical encounters [11
Our study has a number of limitations. The first is the very small sample size. However, it is unlikely that a larger sample size would change our conclusion that by the end of medical school students have largely mastered critical appraisal of an article on diagnostic testing but are unable to use this information to revise a patient's probability of disease. A second limitation is that these data were collected at only one medical school. However, as we find the same pattern of competencies in the recently graduated students who are entering our Internal Medicine program it is likely that our findings apply to many other medical schools. A third limitation is that we had our two groups of subjects critically appraise two different articles although they were very similar from this perspective. A final limitation is that it is possible that our subjects would have demonstrated competency in probability revision if we had provided them with a Bayes nomogram or computer spreadsheet. But we wanted to assess whether our students could apply EBM skills to a clinical encounter without any other external supports except for a simple calculator. In the same way, we do not allow our students to take handbooks or other work aids into their 15 minute OSCE encounters with standardize patients.
The poor performance of our students and residents at probability revision is worrisome although previous studies have shown that many clinicians do not master Bayesian inference. Nearly 25 years ago, Casscells documented that few students or faculty at Harvard Medical School were able to correctly complete a probability revision problem [12
]. Eddy duplicated this finding in a second group of physicians [13
]. However, some cognitive psychologists suggest that humans have most likely always used Bayesian inference in order to survive in our uncertain world. They argue that it is the probability format of the numbers and not the inference task that makes most people fail at the task [14
]. A promising line of research suggests that learners show sustained mastery of Bayesian inference using probabilities if they are taught how to first translate probabilities into natural frequencies [15
]. Whether this will prove to be the solution deserves study.