Within the framework of a needs assessment, residents were surveyed to identify issues directly related to the utility of the Journal Club. This opportunity to reorganize also allowed reexamination of the goals and objectives for the Journal Club in line with those suggested by others [1]. A formal curriculum review with new goals and objectives was put in place, acknowledged by renaming the sessions as "Evidence Based Medicine /Journal Club". Paramount was a focus on critical appraisal and clinical epidemiology from the Evidence-based perspective.

A syllabus of material on the approach to Evidence-based Medicine was chosen to provide the skills necessary to allow the residents to become comfortable with identifying valid articles, potentially impacting their practice of medicine [2]. For this component of the objectives we combined the search for "poem's" (Patient Oriented Evidence that Matters) from Slawson and Shaunessey[3,4] with Sackett's emphasis on clinical epidemiology and the rigorous assessment of validity [5]. The importance of this knowledge base was reinforced by using the first didactic session of the academic year to provide a formal review of the materials. At this session, incoming residents began with a formal test of their skills. They were then presented with the syllabus of selected articles from the JAMA series on the Users Guides to the Literature [6], Slawson and Shaunsessey's work on becoming information masters [3,4], web site references and access to relevant texts.

In order to measure the progress in meeting the objectives of the revised Journal Club, a written pre-test and post-test was chosen as a primary method of evaluating each Journal Club session. This consisted of 10 to 12 questions focused on the principles of Evidence Based Medicine and clinical epidemiology, as well as the key content of the articles. The items were prepared by the Director for Journal Club (JSC) and appeared on both the Pre and Post-tests. Keeping in mind the subjective nature of creating appropriate questions it was felt that well constructed questions could still demonstrate both reliability and validity. Reliability has been equated to the repeatability or stability of responses on repeated administration. Face validity was demonstrated via clear unambiguous questions. Content validity was achieved through inclusion of questions incorporating the key concepts of Evidence Based Medicine and clinical epidemiology. This can be seen in a representative quiz presented initially to faculty and annually to incoming residents to assess their EBM skills (appendix 1).

As analysis of the results indicated a non-normal distribution of scores and therefore results are presented using the median rather than the mean for each module. Median group performance on the Pre-test improved from 54.5 % to 78.9 % over the course of the year (figure ​(figure1).1). A significant linear trend for Pre-test scores was shown between modules (F 89.17, p < .001). Median Post-test results moved from 63.6 % to 81.6 % with a significant linear trend (F 85.84, p < .001) (figure ​(figure2).2). The overall Pre to Post-test difference reached significance (F 2.04, p =.046).

Identifying residents with problems on the Pre-test or a lack of response on the Post-test was obvious and allowed timely information to be forwarded to their academic advisors for remediation. As a bonus, tracking attendance via a turned in quiz was facilitated and this information also was forwarded to the residents academic advisor.

When critical appraisal skills among resident physicians have been formally assessed, correct responses have varied between 33 and 42 % suggesting limited integration of these concepts [8]. This was confirmed at our institution with an overall average of 32% for incoming PGY1 residents over the preceding three years. Even after specific targeted interventions, correct responses have increased only to 67 % [9]. The attainment of a Pre-test median score of 79.0 % at the end of the first year of our intervention was certainly comparable and significantly exceeded those results [9].

Studies have also indicated that there is no correlation between self-assessed competence and actual ability in these skills [10]. With the Pre-test there is direct feedback regarding comprehension and competence. This therefore gives us a quantitative assessment of the individual residents facility with these critical concepts, as they evolve over time.