|Home | About | Journals | Submit | Contact Us | Français|
Evidence-based medicine (EBM) is the conscientious, explicit, and judicious use of the current best evidence in decision-making for the care of patients. Teaching best evidence practice in residency should include both formal or freestanding content, as well as integration into clinical scenarios and patient care. We sought to assess the attitudes, experience and knowledge of EBM in urology residency training across Canada.
An anonymous, cross-sectional, self-report questionnaire was completed by a convenience sample of 29 residents, including all chief urology residents in English-speaking programs across Canada. The survey included both open-ended and closed-ended questions designed to assess familiarity and attitudes towards EBM and potential barriers to developing EBM skills in a surgical training program. Questions were formatted to determine the understanding of statistical and analytical concepts, as well as familiarity of available EBM resources. Descriptive and correlative statistics were used to analyze the responses.
The response rate was 100%. An overwhelming majority of residents felt that EBM is an important component of the urology residency and journal club was the most common vehicle for discussing best evidence concepts. However, there was significant variation in the presence of freestanding, formal curricula across programs, with only 28% of residents signifying that they received any formal training in their program. The apparent level of understanding of important EBM terminology and resources appears to be limited. The most frequently stated barriers to incorporating EBM curricula into urology training were time constraints and a perceived lack of expert educators.
This self-report survey of urology chief residents identified the overwhelming acceptance of the importance of EBM in their training. Although best evidence practices appears to be addressed in journal clubs and in real-life clinical experiences, the obvious lack of familiarity and understanding of EBM content and resources would suggest a need for redoubling efforts to ensure appropriate exposure and instruction in our training programs.
La médecine factuelle vise l’utilisation consciencieuse, explicite et judicieuse des meilleures données actuelles dans le processus décisionnel concernant les soins aux patients. L’enseignement de cette approche factuelle aux étudiants en résidence doit inclure du contenu officiel et indépendant, ainsi que des exercices d’intégration dans des scénarios cliniques et des pratiques de soins. Nous avons tenté d’évaluer les attitudes, l’expérience et les connaissances vis-à-vis la médecine factuelle dans les programmes de résidence en urologie au Canada.
Un questionnaire anonyme d’auto-évaluation globale a été rempli par un échantillon de commodité comprenant 29 résidents, y compris tous les chefs-résidents des programmes d’urologie des universités anglophones du Canada. Le sondage comprenait des questions ouvertes et fermées visant à évaluer le niveau de familiarité et les attitudes vis-à-vis la médecine factuelle et les obstacles pouvant nuire au développement de compétences en médecine factuelle dans un programme de formation en chirurgie. Les questions étaient formulées de manière à permettre de déterminer le niveau de compréhension des concepts statistiques et analytiques, ainsi que le niveau de familiarité avec les ressources existantes de la médecine factuelle. Les réponses ont été analysées à l’aide de méthodes statistiques descriptives et corrélatives.
Le taux de réponse obtenu était de 100 %. Une majorité écrasante de résidents croyait que la médecine factuelle était une composante cruciale de leur formation en urologie et que l’examen en groupe d’articles publiés (Journal Club) représentait le moyen le plus fréquent de discuter des concepts de la médecine factuelle. Néanmoins, on a noté une variation significative dans la présence de contenu indépendant et officiel dans les différents programmes; en effet, seulement 28 % des résidents ont indiqué recevoir une formation officielle dans le cadre de leurs études. Le niveau apparent de compréhension des principaux termes et ressources liés à la médecine factuelle semble limité. Les obstacles les plus souvent mentionnés à l’intégration d’un contenu sur la médecine factuelle dans la formation en urologie étaient les contraintes de temps et un manque perçu d’éducateurs bien versés sur le sujet.
Ce sondage mené auprès de chefs-résidents en urologie a permis de montrer que ces derniers valorisent grandement le rôle de la médecine factuelle dans leur formation. Même si les pratiques factuelles semblent être abordées dans les groupes d’examen d’articles et les expériences cliniques réelles, le manque évident de familiarité et de compréhension des concepts et des ressources liés à la médecine factuelle porte à croire qu’il faut redoubler les efforts afin de s’assurer que les résidents soient suffisamment exposés à cette approche et reçoivent la formation requise pendant leur résidence.
Over the past decade there has been growing interest in the role and structure of evidence-based medicine (EBM) curricula in residency programs. Evidence-based medicine refers to the conscientious, explicit and judicious use of the current best evidence in making medical decisions1 and most appropriately involves the integration of published evidence with clinical acumen to help make decisions in the unique context of the needs of the individual patient.2 A growing body of evidence has underscored the need to advocate for EBM training in residencies.3–8 Teaching with an EBM approach facilitates development of the skills needed to bring the most current information to the real-time practice of medicine and has led for calls to ensure principles of EBM, including biostatistics, critical appraisal and medical informatics, are included into graduate medical education programs.9
In most residency programs, this training has traditionally been relegated to journal clubs, in which small group discussions focus on articles chosen for their relevance to a particular specialty and several survey studies have shown that these learning venues are most commonly employed.10–13 Although journal clubs may facilitate teaching of epidemiologic and critical appraisal principles, their efficacy in developing EBM skills to enhance individual patient decision-making is limited and has generally not been well-examined.14 Instead, there has been a changing focus towards a “free-standing” EBM curricula within residency programs with the recognition that such content needs to be further integrated into clinical practice on the wards and in clinics to ensure that learners use EBM in their own practices.
There have been several resident survey and focused interview studies demonstrating the barriers to teaching EBM in residency, including lack of time, resources and qualified teachers, and these barriers are likely amplified in surgical residency programs.15 To our knowledge, these issues of incorporation of best evidence theory have never been explored in a urology residency. Our objective was to determine the contemporary attitudes and knowledge towards EBM, as well as the prevalence and structure of EBM curricula in urology residency programs in Canada.
This prospective study surveyed, as a convenience sample, senior urology residents (n = 29) participating in a national review course, and included all PGY-5 chief residents in the English-speaking Canadian urology training programs. Participation was completely voluntary and confidentiality was maintained at all times as no identifying information was recorded in the survey results. Ethics approval was attained from the Queen’s University Institutional Review Board and explanations for the objectives of the study and assurance of confidentiality were distributed to the residents responding to the survey.
The cross-sectional, self-report questionnaire focused on the attitudes, experience and working knowledge of EBM, including questions exploring the residents’ ability to understand analytical concepts, their awareness of EBM resources, and how well EBM was incorporated into their respective residency training curricula. The survey included 16 closed-ended questions using a 5-point Likert scale to determine the state of EBM education in their programs. The residents were also asked to rate their own knowledge and understanding of statistical and analytic EBM concepts on a 4-point scale with 1) indicating no desire to understand, 2) indicating no understanding but desire to learn, 3) indicating some understanding and 4) indicating understanding and the ability to explain the concept to others. As well, the respondents were asked to rate their familiarity with different EBM resources on a similar 4-point scale. A pick-list and an open-ended question were used to determine existing barriers to incorporating EBM into their training programs. Finally, to determine the personal experience of incorporating best evidence into clinical practice, the respondents were given 3 clinical scenarios that were felt to be amenable to the incorporation of high-level of evidence in decision-making. For each of these scenarios, respondents were asked to rate, with a 5-point Likert scale, 1) whether there was consensus amongst staff regarding the most appropriate management, 2) what was the most common management they experienced in residency and 3) what they would likely do in their own practice.
Questionnaire development resulted from an initial experience with a previous survey construction for similar attitudes for specialty residents. Residents and educators involved in both undergraduate and postgraduate programs were asked to assess and modify the survey for clarity. Descriptive statistics (mean +/− standard deviation) is presented for the questionnaire responses; however, for ease of reporting purposes for the questions using the 5-point Likert scale, agreement responses of 4 and 5 to were grouped together, as were the disagreement responses of 1 and 2. A Mann-Whitney test was used to determine significance between responses of the 5-point Likert scale. Spearman or Pearson tests, depending on normality of distribution, were used to demonstrate correlations of respondents to questions using the Likert scale. The GraphPad Prism 4 statistical software package (GraphPad Software, Inc., San Diego, CA) was used for analysis.
The responses from this self-report survey suggest considerable variation of the existence or stature of EBM in Canadian undergraduate programs. Only 55% of chief residents agreed or strongly agreed (mean response 3.48 ± SD 1.38) that they received formal exposure to EBM in medical school. However, most respondents agreed (4.14 ± 0.79) that formal training in EBM within urology residency would be valuable and 62% agreed (3.89 ± 0.82) that formal EBM content should be a mandatory component of training. Interestingly, there were no respondents that disagreed or strongly disagreed on the value or need for a mandatory, “freestanding” EBM curricula within their residency. As well, there was no statistically significant correlation between a previous formal exposure to EBM and the attitudes towards EBM during residency.
In contrast to the apparent positive attitude towards EBM in residency training, there was a concerning disparity across respondents (28% agreement) regarding the presence of a formal, “freestanding” EBM curriculum in urology programs (2.86 ± 1.87), although there was more agreement (3.05 ± 1.14) that EBM concepts were used in an “integrated” approach in real-time clinical scenarios. Furthermore, there was a good correlation of responses between these 2 questions (p < 0.001, r = 0.7203) suggesting that those programs incorporating formal training in EBM were those most likely to incorporate these best-evidence concepts in the clinic or on the ward. Only 41% of residents positively responded that they were aware of other EBM curriculum offered at their centres or universities outside of their residency programs (3.15 ± 1.36). Consistent with previous studies on residents’ EBM experiences, 97% of respondents acknowledged participation in regular journal clubs in their programs (4.75 ± 0.51) and many (66% agreement, 3.87 ± 1.02) felt that there was appropriate emphasis on critical appraisal skills at these meetings.
We asked the residents how effectively they felt their programs evaluated the acquisition of EBM skills and these responses mirrored those regarding their exposure of EBM in their programs. Only 45% of respondents agreed that they were effectively evaluated in the more formal aspects of medical information acquisition (3.03 ± 1.15) or critical appraisal (3.24 ± 1.24). However, 59% agreed (3.38 ± 1.18) that they were consistently evaluated in their ability to apply best-evidence information to a particular clinical question, as was their ability to use this information to help in patient decision-making (3.79 ± 0.90). The residents that felt that their programs emphasized critical appraisal skills in journal clubs also reported that they were more likely to have their EBM skills routinely evaluated in all four of the these aspects outlined above (p value for all Spearman correlations < 0.01).
We next questioned the residents on what they felt were the most appropriate mechanisms to help integrate EBM concepts into urological training. Most respondents agreed (76%, 4.12 ± 0.97) that a formal, freestanding curriculum delivered by trained educators would be beneficial. Only 58% felt (3.70 ± 0.98) that facilitating the use of best evidence summaries, such as the Cochrane Collaboration, would be useful, whereas 79% agreed (4.16 ± 0.75) that specialty specific guidelines (such as those of the American Urological Association and the Canadian Urological Association) are appropriate means of applying best evidence information in their practices.
The residents were asked to rate their familiarity of 10 commonly used EBM-related statistical or analytical terms. The results are consistent with previous reports; the level of understanding of such terms is very much dependent on the questions that are asked (Table 1). As an example, despite only 3 residents (10%) responding that they did not understand the term meta-analysis, only 17 (59%) felt they understood the term well enough to be able to describe it to another person.
Residents were asked to record their own awareness and familiarity with 8 web-based EBM resources including Bandolier, ACP Journal Club, Clinical Evidence, Centre for Evidence-Based Medicine, Drug and Therapeutics Bulletin, Medical Letter and the Journal of Informed Pharmacotherapy. Most respondents were not aware of, and had no practical familiarity with, most of these resources (Table 2).
Residents were asked to identify factors that act as barriers to the incorporation of EBM into their training and practice of urology. Many respondents (52%) felt that there was a lack of time necessary to learn EBM strategies and 69% felt their programs lacked qualified EBM educators (Table 3).
Residents were questioned about 3 clinical scenarios with published guidelines or recommendations supported by a high level of evidence: 1) the use of Bacillus Calmette-Guerin (BCG) for high-risk non-muscle invasive transitional cell carcinoma (TCC), 2) adjuvant radiotherapy post-prostatectomy for positive surgical margins and 3) the role of neoadjuvant chemotherapy for high-risk muscle-invasive TCC. Residents were asked to comment on whether there was general consensus on the benefits of these clinical management options amongst staff, if these managements were commonly employed in their residency experience, and if they would apply these management options in their own practice.
Table 4 shows the results of all responses using the 5-point Likert scale. Grouping agreement responses, we found that 83% of residents thought there was consensus amongst attending staff for BCG use, 86% agreed that BCG was commonly given, and 93% planed to give BCG in high-risk non-muscle invasive TCC in their own practice. Although there appeared to be less agreement regarding the consensus (38%) and perceived use (52%) of postoperative adjuvant radiotherapy for positive margins at prostatectomy, a significantly higher proportion (p < 0.001), agreed that they would likely use adjuvant radiotherapy in their own practice. A similar statistically significant trend in responses is evident with the scenario regarding neoadjuvant chemotherapy for invasive TCC. Despite a clear lack of consensus amongst staff regarding potential benefit, many more residents agreed that they would likely suggest the use of neoadjuvant chemotherapy in their own practice (p < 0.001).
There have been criticisms of the EBM movement for ignoring clinical acumen as well as patient wishes, but more recent definitions of EBM promote the use of evidence and “informed opinion” in the context of the needs of the individual patient. As such, the need to incorporate such concepts into medical practice would appear undeniable. However, there continues to be much debate on how best to implement EBM into residency programs. Regarding the content issues of any EBM curricula, is it best to teach residents the ability to perform a systematic review of primary research, or facilitate residents learning to access already-prepared systematic reviews or guidelines? Furthermore, to fully realize the benefits of EBM, faculty and learners must model best evidence information when discussing patient care issues in real-life clinical experiences. Given these uncertainties regarding the best methods of teaching and evaluating EBM skills, it is understandable that previous reports have demonstrated ambiguous attitudes and experiences towards EBM in different residency programs.
To our knowledge, this is the first investigation of the attitudes, experiences and knowledge of EBM in urology training programs. In this contemporary self-report survey, it is remarkable that the vast majority of residents had a positive attitude towards the role of EBM in their residency training and most felt that formal EBM curricula should be mandatory in surgical training. In the 3 clinical scenarios presented in the survey that were felt to have a high level of evidence supporting recommended treatment options, residents consistently reported they were more likely to follow this best evidence compared to what they perceived was being done during their training. Despite this, and at least a decade of focus on the importance of EBM in medical practice, the reported experience of formal EBM teaching in urology residency appears to be highly variable between programs across Canada. Less than one third of respondents to this survey indicated that there was any formal, freestanding instruction of EBM throughout their residency and only 55% indicated that they had any formal EBM education in medical school.
This apparent lack of the more formal, content-related aspects of EBM in the training experience of the residents was reinforced by the responses to the questions regarding their familiarity with statistical and analytical terminology. Although 90% of respondents felt that they understood important concepts, such as relative risk, absolute risk and meta-analysis, only 46% of residents felt that they could explain these terms to others. These results are similar to a recent study of residents across different disciplines, suggesting the level of understanding of such concepts is limited in comparison to those with more formal training in EBM.16 Furthermore, the residents overwhelmingly felt that prepared reviews or guidelines are important in using best evidence into their practice, but the vast majority were not well-informed of the web-based EBM resources available to practitioners. Although this survey did not directly address the underlying reasons for this limited familiarity, Bhandari and colleagues, in a structured interview with surgical residents, did identify a lack of surgical EBM resources and information as a significant barrier.15
The respondents did suggest that journal clubs were frequently used as an opportunity to discuss best evidence in a clinically relevant arena, and it appeared that there was a correlation with consistently emphasizing critical appraisal skills in this venue and a feeling of appropriate evaluation of EBM skills throughout residency. The results of this survey do suggest that residents and faculty both use real-time clinical experiences to emphasize and discuss best evidence concepts, an important component of any EBM curricula in training. However, it appears that more formal, content-related components of EBM theory are lacking in a significant number of training programs across Canada. The barriers that were identified by the residents, such as time constraints and a perceived lack of appropriate educators, are very similar to previous reports in other residency programs in both the United States and Canada.15,17 Beyond journal clubs, online module activities or block rotations can be effective, efficient techniques for teaching the basics of EBM. It would appear appropriate for programs to review their own experience and access to such freestanding curricula available at their centres in order to reinforce these content components of EBM.
There are several limitations to this study that need to be considered. First, the results are derived from a self-report survey on the participation and attitudes towards EBM and independent verification of data was not possible. The present study was conducted with the participation of PGY-5 urology residents and this may not be a fully representative sample of postgraduate trainees. Also, this survey was conducted within several months of their Royal College of Physicians and Surgeons of Canada certification exams and perhaps affected their perception of their training in EBM. This may be countered, however, by the 100% response rate and by the fact that these residents have been a part of postgraduate training for more than 4 years, and, as such, have been maximally exposed to the range and scope of their disciplines training. A follow-up study exploring these issues is being planned for program directors and other key-stakeholders in urology training across Canada.
This self-report survey of urology chief residents identified the overwhelming acceptance of the importance of EBM in their training. Although best evidence practices appears to be addressed in journal clubs and in real-life clinical experiences, the lack of familiarity and understanding of EBM would suggest that individual program directors, or perhaps all Canadian program directors as a group, could review and implement a more formal, freestanding curricula. This would foster critical appraisal, as well as framing and testing good clinical questions, which ultimately guide practice and lifelong learning.
Competing interests: None declared.
This paper has been peer-reviewed.