Search tips
Search criteria 


Logo of jrsocmedLink to Publisher's site
J R Soc Med. 2010 July 1; 103(7): 288–294.
PMCID: PMC2895523

Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training



To evaluate the educational effectiveness of a clinically integrated e-learning course for teaching basic evidence-based medicine (EBM) among postgraduate medical trainees compared to a traditional lecture-based course of equivalent content.


We conducted a cluster randomized controlled trial to compare a clinically integrated e-learning EBM course (intervention) to a lecture-based course (control) among postgraduate trainees at foundation or internship level in seven teaching hospitals in the UK West Midlands region. Knowledge gain among participants was measured with a validated instrument using multiple choice questions. Change in knowledge was compared between groups taking into account the cluster design and adjusted for covariates at baseline using generalized estimating equations (GEE) model.


There were seven clusters involving teaching of 237 trainees (122 in the intervention and 115 in the control group). The total number of postgraduate trainees who completed the course was 88 in the intervention group and 72 in the control group. After adjusting for baseline knowledge, there was no difference in the amount of improvement in knowledge of EBM between the two groups. The adjusted post course difference between the intervention group and the control group was only 0.1 scoring points (95% CI −1.2–1.4).


An e-learning course in EBM was as effective in improving knowledge as a standard lecture-based course. The benefits of an e-learning approach need to be considered when planning EBM curricula as it allows standardization of teaching materials and is a potential cost-effective alternative to standard lecture-based teaching.


Acquiring knowledge, skills and attitudes to practice evidence-based medicine (EBM) is a core competence for all junior doctors.1 For EBM teaching and learning to be effective it should be clinically integrated.2 However, the majority of teaching of EBM and critical appraisal takes place in the classroom away from a clinical setting. Integrating EBM teaching into a clinical setting is not a straightforward task and risks distracting from service delivery as it can be time-consuming. The pool of teachers trained in providing face-to-face teaching on the foot is limited.3

To address the lack of time and capacity to deliver EBM teaching on the go, we developed an e-learning EBM course for just-in-time learning through on-the-job training for postgraduate medical trainees.4,5 Its feasibility was piloted in different languages and settings across five European countries in a before-and-after study. It showed that knowledge improved significantly and both trainees and tutors found the e-learning course acceptable.5 How would such a course fare against delivery of equivalent content in the traditional lecture mode during foundation (internship) training? We determined the effect of the e-EBM course on participant's knowledge of EBM in a randomized controlled trial.



We conducted a cluster randomized controlled trial6 to compare the clinically integrated e-learning EBM course to a lecture-based course between May and September 2007.


The trial was undertaken in seven teaching hospitals in the UK West Midlands region.


All of the participants in the trial were foundation year two doctors (interns). All trainees consented to be part of the trial.


The West Midlands Deanery leads the training programmes for all foundation-year doctors in the region. Approval was obtained from the Deanery, who granted us permission to undertake the trial. All of the hospitals in the region who were responsible for postgraduate medical education were recruited for the trial.


The teaching hospitals were randomized by means of computer programme into two groups: Group 1 – received the e-learning EBM teaching programme (experimental intervention) (four clusters, 122 participants); and Group 2 – received standard classroom-based standalone EBM teaching sessions of equivalent content (control group) (three clusters, 115 participants). Randomization was performed after recruitment and after consent was obtained.

Intervention and comparator

The teaching sessions (standalone and e-learning) were divided into three separate modules. The module contents covered the topics: asking and formulating clinical questions, searching for research evidence and critical appraisal of systematic reviews and their constituent studies ( Table 1). The development of the curriculum and the contents of the modules have been previously reported.4 The learning objectives and material contents for both the standalone and the e-learning group were the same.

Table 1
An overview of the clinically integrated e-learning course in evidence-based medicine (EBM)

For the standalone group, the teaching was delivered by a lecturer over one session consisting of three hours in each centre, following the standard format used in the educational programme devised for foundation training. To ensure equivalence of teaching content, we used PowerPoint slides identical to those deployed in the e-EBM course in the same sequence. The e-learning group were granted unlimited access for a period of six weeks to the e-learning materials via a project specific website ( ). Each participant was given their own personal log-in code and their log-in activities were monitored on a weekly basis to ensure compliance with the trial protocol.

Data collection

All participants completed validated multiple choice questions (MCQs)7 to assess EBM knowledge before accessing the e-learning materials and prior to the start of the teaching sessions. After completion of each module the participants completed the same MCQs relevant to that module again ( Figure 1). The MCQs were developed previously as part of a study to evaluate the outcomes of the EBM e-learning course in five European countries, which has been published elsewhere.5 The MCQs were linked to the learning objectives of the course to assure content validity.

Figure 1
Flow chart of participants in the trial


There was no a priori sample size calculation. However, post hoc statistical power computation shows that a study with seven clusters (mean cluster size of 22) and assuming an intraclass correlation coefficient (ICC) of 0.01, with a standard type I error rate (p 0.05) our sample had 80% power to detect a 3.5-point difference between groups in post course scores. Assuming a common standard deviation of 7 points, this represents a medium-effect size in terms of Cohen's d.8 If we assume an ICC of 0.02, the corresponding power would have been 74%.

Responses to the pre and post module MCQs were scored and comparisons between the e-learning and the standalone group were made. Missing data were not imputed and analyses were restricted only to those participants who answered both pre and post module MCQs. Analyses were done in an intention-to-treat basis (all participants were analysed in the group they were randomized in). Between arms comparison was undertaken by means of generalized estimating equations (GEE) model9 which permits the specification of a working correlation matrix to account for within-cluster (i.e. hospital) correlation of the outcome. In this case we assume that there is no order for participants within a hospital and then an exchangeable correlation matrix was specified. The model included teaching modality as the independent variable and baseline knowledge as a covariate.


A total of 91 postgraduate trainees completed the pre course MCQs in the e-learning (intervention) group and 79 in the standard classroom teaching (control) group. The number of participants who did not fully complete the course was 3 in the e-learning group and 7 in the standard classroom teaching group. The reasons for not completing included being called out of the teaching session to see patients, being on leave and leaving the training programme. The total number who completed the course and returned both pre and post course assessments was 88 in the e-learning group and 72 in the standard classroom teaching group (Figure 1).

Table 2 shows overall baseline knowledge scores and scores by teaching module for the two arms of the trial. The scores were generally high in both groups. Maximum attainable scores were 13, 7 and 18 points for modules 1, 2 and 3, respectively. Post course scores were improved in both groups compared to baseline (Table 2, Figure 2). After adjusting for baseline knowledge and taking into account the cluster nature of the trial design, there was no difference in the amount of improvement of knowledge between the two groups. The adjusted post course difference between the intervention group and the control group was only 0.1 scoring points (95% CI −1.2–1.4) (Figure 2).

Table 2
Knowledge (MCQ) scores after course delivery
Figure 2
Comparison of knowledge scores achieved through standard teaching versus e-learning (dots show data for individual participants; boxes with vertical whiskers show means and SD)


Main findings

The trial demonstrated that both the e-learning and the standard classroom-based teaching approaches lead to an overall improvement in EBM knowledge. The e-learning group achieved similar scores for overall knowledge gain compared to the control group. This raises the possibility of delivering this element of foundation (internship) curriculum through e-learning, replacing the traditional lecture-based method without loss of knowledge.

Comparison with other studies

There have been several other studies that have examined the effectiveness of computer-based teaching of EBM compared to standard lecture-based approach10–14 and our study also concurs with their findings. Our trial merits consideration as we undertook a randomized design to provide scientifically sound results. One of the strengths of the trial was that we were able to recruit more participants than previous trials10–12 and this increases the precision of our findings. Furthermore, a cluster randomized trial design was chosen to reduce the effect of contamination.15 This is a particular concern with educational research. Additionally, cross-over of participants and an inability to blind participants and the teachers can cause concern in educational research.11 These potential methodological flaws are addressed better with cluster randomized trials.16 Concerning randomization we ensured allocation concealment. Our statistical analysis of the data took into consideration the cluster design.9 Our results concur with a previous study examining the effectiveness of teaching EBM to postgraduate medical doctors in the obstetrics and gynaecology specialty by e-learning, which was conducted in the Netherlands and the UK.12

Methodological limitations

There are several limitations of our trial that warrant consideration in the interpretation of our findings. As previously found,12 trainees in both groups showed high baseline knowledge for all modules prior to the EBM teaching and this left little margin for improvement in the post course scores. This may be attributed to previous training in EBM within the deanery. Several questionnaire surveys to address junior doctors' knowledge and beliefs concerning EBM have been undertaken in the deanery17,18 which has fostered an educational environment that promotes the use of EBM in clinical practice. The participants may have been more aware and self-motivated than trainees generally. Furthermore, a small proportion of participants in both groups had lower scores post course compared to pre course. This has also been found in other studies that have used knowledge assessments and has been associated with cognitive test anxiety.12,19,20 Clinically integrated teaching of EBM can be expected to bring about changes in behaviour, practice and outcomes, but we were unable to measure these.

Implications for practice

As postgraduate medical training in the UK is currently evolving to meet future healthcare needs, it is imperative that it encompasses the challenges of teaching doctors in changing environments.21 The benefits of an e-learning approach to teaching should be considered as a viable way in meeting these challenges, as it can support a wide range of learning activities, which are readily accessible and can be tailor-made to meet specific learning objectives. As our trial implies, e-learning teaching provides knowledge gains equivalent to that of standard classroom-based teaching. Particularly, the benefits should be considered when planning EBM curricula as it allows standardization of teaching materials and is a potential cost-effective alternative to standard lecture-based sessions.



Competing interests None declared

Funding The project was funded by the European Union Leonardo da Vinci project (grant number UK/05/B/F/PP-162_349)

Ethical approval Not applicable

Guarantor JH

Contributorship The trial was conceived by KSK and the Leonardo project team. JH coordinated the project, delivered the teaching sessions and conducted the trial. GZ provided Internet-based data collection for the e-learning group. JZ performed the statistical analysis. All authors commented on the manuscript

Reviewer Elizabeth Murray


The authors would like to thank the West Midlands Deanery, in particular Robert Cragg, for his help and assistance


1. The Foundation Programme Committee Curriculum for the foundation years in postgraduate education and training. London: Department of Health; 2005 [PubMed]
2. Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine 2. BMC Med Educ 2006;6:59. [PMC free article] [PubMed]
3. Richardson WS. Teaching evidence-based practice on foot. Evid Based Med 2005;10:98–9a
4. Coppus SF, Emparanza JI, Hadley J, et al. A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project. BMC Med Educ 2007;7:46. [PMC free article] [PubMed]
5. Kulier R, Hadley J, Weinbrenner S, et al. Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Educ 2008;8:27. [PMC free article] [PubMed]
6. Donner A, Klar N. Cluster randomization trials in epidemiology: Theory and application. J Statistical Planning Inference 2008;42:37–56
7. Taylor R, Reeves B, Mears R, et al. Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching 85. Med Educ 2001;35:544–7 [PubMed]
8. Cohen J. A power primer. Psychological Bulletin 1992;112:155–9 [PubMed]
9. Liang K-Y, Zeger S. Longitudinal data analysis using generalized linear models. Biometrika 2008;73:13–22
10. Davis J, Crabb S, Rogers E, Zamora J, Khan KS. Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomized controlled trial. Med Teach 2008;30:302–7 [PubMed]
11. Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A. Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomised controlled trial. BMC Med Educ 2007;7:23. [PMC free article] [PubMed]
12. Kulier R, Coppus S, Zamora J, et al. The effectiveness of a clinically integrated e-learning course in evidence-based medicine: A cluster randomised controlled trial. BMC Medical Education 2009;9:21. [PMC free article] [PubMed]
13. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA 2008;300:1181–96 [PubMed]
14. Greenhalgh T. Computer assisted learning in undergraduate medical education. BMJ 2001;322:40–4 [PMC free article] [PubMed]
15. Klar N, Donner A. Current and future challenges in the design and analysis of cluster randomization trials. Statist Med 2001;20:3729–3740 [PubMed]
16. Hahn S, Puffer S, Torgerson D, Watson J. Methodological bias in cluster randomised trials. BMC Medical Research Methodology 2005;5:10. [PMC free article] [PubMed]
17. Hadley J, Wall D, Khan K. Learning needs analysis to guide teaching evidence-based medicine: knowledge and beliefs amongst trainees from various specialities. BMC Medical Education 2007;7:11. [PMC free article] [PubMed]
18. Awonuga AO, Dwarakanath LS, Khan KS, Taylor R. Post-graduate obstetrics and gynaecology trainees' views and understanding of Evidence-based Medicine. Medical Teacher 2000;22:27–9
19. Patel AA, Patel DD. Education through multimedia among Agricultural Diploma school students: An impact study. International Journal of Education and Development using Information and Communication Technology 2006;2:4–10
20. Cassady JC, Johnson RE. Cognitive Test Anxiety and Academic Performance. Contemporary Educational Psychology 2002;27:270–95
21. Postgraduate Medical Education Training Board The State of Postgraduate Medical Education and Training 2009: laying foundations for the future. London: PMETB; 2009

Articles from Journal of the Royal Society of Medicine are provided here courtesy of Royal Society of Medicine Press