|Home | About | Journals | Submit | Contact Us | Français|
To compare international medical graduates (IMGs) with Canadian medical school graduates in a family practice residency program.
Analysis of the results of the in-training evaluation reports (ITERs) and the Certification in Family Medicine (CCFP) examination results for 2 cohorts of IMGs and Canadian-trained graduates between the years 2006 and 2008.
St Paul’s Hospital (SPH) in Vancouver, BC, a training site of the University of British Columbia (UBC) Family Practice Residency Program.
In-training evaluation reports were examined for 12 first-year and 9 second-year Canadian-trained residents at the SPH site, and 12 first-year and 12 second-year IMG residents at the IMG site at SPH; CCFP examination results were reviewed for all UBC family practice residents who took the May 2008 examination and disclosed their results.
Pass or fail rates on the CCFP examination; proportions of evaluations in each group of residents given each of the following designations: exceeds expectations, meets expectations, or needs improvement. The May 2008 CCFP examination results were reviewed.
Compared with the second-year IMGs, the second-year SPH Canadian-trained residents had a greater proportion of exceeds expectations designations than the IMGs. For the first-year residents, both the SPH Canadian graduates and IMGs had similar results in all 3 categories. Combining the results of the 2 cohorts, the Canadian-trained residents had 310 (99%) ITERs that were designated as either exceeds expectations or meets expectations, and only 3 (1%) ITERs were in the needs improvement category. The IMG results were 362 (97.6%) ITERs in the exceeds expectations or meets expectations categories; 9 (2%) were in the needs improvement category. Statistically these are not significant differences. Seven of the 12 (58%) IMG candidates passed the CCFP examination compared with 59 of 62 (95%) of the UBC family practice residents.
The IMG residents compared favourably with their Canadian-trained colleagues when comparing ITERs but not in passing the CCFP examination. Further research is needed to elucidate these results.
Comparer les diplômés internationaux en médecine (DIM) inscrits dans un programme de résidence en médecine familiale à leurs collègues diplômés des facultés de médecine canadiennes.
Analyse des résultats de l’évaluation en cours de formation (REEF) et des résultats à l’examen de certification en médecine familiale (CMFC) de 2 cohortes de DIM et de diplômés formés au Canada entre 2006 et 2008.
St Paul’s Hospital (SPH) à Vancouver, C.-B., un centre de formation du programme de résidence en médecine familiale de l’University of British Columbia (UBC).
On a relevé les rapports d’évaluation effectués en cours de formation au SPH pour des résidents formés au Canada (12 de première année et 9 de deuxième) et ceux des DIM (12 de première année et 12 de deuxième); les résultats à l’examen du CMFC ont été vérifiés pour tous les résidents en médecine familiale de l’UBC qui ont fait l’examen de mai 2008 et en ont révélé les résultats.
Taux de réussite ou d’échec à l’examen du CMFC; pour chaque groupe de résidents, proportion de ceux ayant obtenu une des cotes suivantes : dépasse les attentes; répond aux attentes; ou nécessite amélioration. Les résultats de l’examen du CMFC de mai 2008 ont été examinés.
Par rapport aux DIM de deuxième année au SPH, ceux formés au Canada ont obtenu une plus forte proportion de cotes « dépasse les attentes ». Pour ceux de première année, les 2 groupes avaient des résultats semblables pour les 3 catégories de cotes. Si on combine les résultats des 2 cohortes, les résidents formés au Canada ont obtenu 310 REEF (99 %) indiquant « dépasse » ou « rencontre les attentes » et seulement 3 REEF (1 %) indiquant un besoin d’amélioration. Les DIM ont eu 362 REEF (97,6 %) indiquant « dépasse » ou « rencontre les attentes » et 9 (2 %) indiquant un besoin d’amélioration. Ces différences ne sont pas statistiquement significatives. Sur 12 candidats DIM à l’examen du CMFC, 7 (58 %) ont réussi, comparativement à 59 des 62 résidents en médecine familiale de l’UBC (95 %).
Les REEF des DIM se comparaient avantageusement à ceux de leurs collègues formés au Canada, mais non leurs résultats à l’examen du CMFC. Il faudra d’autres études pour éclaircir cette disparité.
For many years international medical graduates (IMGs) have made up approximately 25% of the physician work force in Canada. In British Columbia (BC), 22% of postgraduate physicians have been trained abroad.1–3
There is evidence in the literature that IMGs do not do as well as Canadian-trained residents in their residencies or on their final examinations; and there is a perception among program directors that IMGs do not perform as well.4–7
In 2005, the BC provincial government increased its funding for residency training for IMGs from 6 to 18 positions per year, with 12 positions in the specialty of family medicine and 6 in other specialties.8 A provincially appointed IMG task force recommended that IMG family practice residents be trained at a newly created site at St Paul’s Hospital (SPH) in Vancouver, BC, with the first cohort starting in July 2006. St Paul’s Hospital has had an existing site for family practice residency training since 1994, and the attending clinicians have had extensive experience of family practice teaching as well as assessing IMGs since 1992.9 The existing SPH family practice residency site continued to recruit residents through the Canadian Residency Matching Service and matched only medical students from Canadian (or US) medical schools in the first iteration. The IMG residents were ranked through Canadian Residency Matching Service only if they had completed the BC IMG clinical assessment program.
The 2 residency programs (ie, SPH site and IMG site) would exist side by side and share many of the same teaching resources and some of the same rotations. Most IMGs in Canada are simply integrated into existing residency programs and their particular needs as IMGs are not examined in any depth.
The faculty members that were appointed to oversee the new IMG site had experience in teaching Canadian-trained graduates as well as long-standing exposure to and a specific interest in teaching IMGs.10 The anticipation was that by creating a specific training site for IMGs, the challenges associated with the various backgrounds and training of IMGs could be addressed—even though the 2 programs would be integrated educationally to a certain extent. In particular, the residents at the IMG site were given more extensive exposure to ethical, cultural, and behavioural medicine issues than their Canadian counterparts were. More time was spent analyzing doctor-patient relationships and communication issues.
To our knowledge (University of British Columbia [UBC] Family Practice Residency), this is the first time in North America that a training site has been created specifically for IMGs, with the added advantage of working alongside Canadian-trained residents. We believed that this IMG site provided a unique opportunity to monitor the educational progress of the IMGs in this setting by comparing them with their Canadian-trained colleagues.
Every family practice resident in BC is continuously assessed by means of a Web-based evaluation, which is based on the 4 principles of family medicine. Preceptors complete the evaluation at the end of every rotation; a rotation can last for as little as 2 weeks for electives to as long as 8 weeks for most of the core rotations. A review was conducted of all the in-training evaluation reports (ITERs) for both the IMGs and the Canadian-trained residents, as well as the results of the Certification in Family Medicine (CCFP) examination, which is taken by our residents in May of the second year of the 2-year residency. Ethics approval was obtained from the UBC Providence Health Care Research Ethics Board.
The residents from the IMG and SPH sites were asked for their written consent for their ITERs to be collated and reviewed by the author with the individual evaluations being blinded before review. Consent was requested by the program coordinators, and any identifying details in the evaluations were removed.
Each ITER was graded initially as either pass or fail. Additionally, each evaluation was given one of the following designations: exceeds expectations, meets expectations, or needs improvement. The ITERs also contained the evaluators’ written comments but these were not seen by the author.
The CCFP examination results were obtained from the central office of the UBC Family Practice Residency Program with the residents’ permission.
All of the residents at the 2 sites gave written permission to have their ITERs reviewed. Residents who had gaps in their training for more than 3 months (all were on maternity leave) were excluded from analysis. Three second-year residents and 1 first-year resident from the SPH site (SPH Canadian graduates) were excluded, and no residents from the IMG site were excluded. Participants included 12 first-year and 9 second-year residents from the SPH site (SPH Canadian graduates), and 12 first-year and 12 second-year IMG residents from the IMG site. The ITERs from the cohorts between 2006 and 2008 (second-year residents) and between 2007 and 2008 (first-year residents) were analyzed to see if the SPH Canadian graduates and the IMGs showed significant differences. Results are presented in Tables 1 and and22.
In the 2 first-year resident groups (ie, SPH Canadian graduates and IMGs), SPH Canadian graduates and IMGs have very similar ratios in the categories exceeds expectations, meets expectations, and needs improvement. In particular, Fisher exact tests (2-tailed) indicated that the rate of evaluations in the needs improvement category versus the other 2 categories was comparable in the 2 groups, P = .68. It remains to be seen if this is mirrored in the second year of their residencies.
A different pattern occurred between the second-year SPH Canadian graduates and IMGs, with a wider gap between the 2 groups: SPH Canadian graduate residents consistently had a larger ratio of the exceeds expectations designation. Remarkably, no second-year Canadian-trained resident had a single needs improvement designation in the whole 2 years. The rate of the needs improvement designation was significantly higher for the IMGs, P = .02 (2-tailed Fisher exact test).
However, when looking at the total number of needs improvement designations of the first- and second-year residents’ rotations, there is little difference between the 2 groups, with very few needs improvement designations (9 for IMGs and 3 for SPH Canadian graduates) being seen as weak in a rotation yet still meriting a pass. This difference was within the variation of results expected by chance, P = .21 (2-tailed Fisher exact test). Out of 684 evaluations given, only 1 was rated as a fail. Of the SPH Canadian graduate residents, 3 (1%) evaluations were given a needs improvement rating compared with 9 (2%) evaluations of the IMGs.
There were, however, considerable differences between IMGs and the Canadian graduates across all of the UBC family practice sites in the CCFP examination results. Of the 12 IMGs who completed the examination in May 2008, 7 passed. The percentage of passes in this group (58%) is in keeping with the historical national pass rate for IMGs (Brailovski C, oral communication, September 2008). In comparison, of the 62 Canadian-trained UBC Family Practice residents who revealed their results to the program, 59 passed—for a pass rate of 95%.
There could be any number of explanations for these results, especially drawing comparisons between a well established family practice residency site (ie, SPH site) and a fledgling site (ie, IMG site) unlike any other in the country. The average age of the IMG residents was 40 years and that of the SPH residents was younger than 30 years. Most of the IMGs also had family responsibilities, with children ranging from newborns to university students. English was a second language to all 24 IMGs.10 There are also some considerable differences between the 2 sites, especially in the second year of residency, with the IMG site having a more traditional rotation-based schedule compared with the SPH site’s horizontal schedule.
It could be argued that the ITERs are not a good method of assessing readiness for practice. Although ITERs are extensively used in Canadian residency programs as well as in other parts of the world, there continues to be criticism in the literature as to their reliability in forecasting the competence of medical residents11,12; however, ITERs continue to be widely used and are constantly reviewed to improve their ability to predict performance.13,14 Program directors express the need for ongoing assessments during residency training in addition to a final examination mandated by the Royal College of Physicians and Surgeons of Canada or the College of Family Physicians of Canada.15 If, however, the written comments of the evaluators had been assessed, it is possible that more differences might have been found rather than only relying on the limited categories reviewed.
The outcome for IMGs in the CCFP examination is disappointing, especially considering that very knowledgeable faculty members, many of whom have served as examiners or on the examination committee at the College of Family Physicians of Canada, worked very diligently with the IMG candidates to thoroughly familiarize them with the examination requirements. Specifically, each resident had the opportunity to learn about how simulated office orals are constructed and marked, and he or she was given at least 3 practice simulated office orals before sitting the CCFP examination.
This result is at odds with the perception that the IMG residents performed very adequately during their residency training and compared favourably with their Canadian-trained colleagues.
The numbers in this analysis are relatively small and time alone will tell if the results in future will be different. Analyses at other locations where larger numbers are incorporated will be needed before definitive conclusions can be drawn.
This analysis does appear to support the conclusion that even when IMGs train in a dedicated site with highly experienced teachers, they still struggle to match their Canadian counterparts in passing their final examinations. One explanation is that, in some way, IMGs are disadvantaged by the format of the CCFP examination. It could be cultural; it could be the subtle nuances of the English language; it could be the more direct approach that some IMGs have in eliciting medical information; or it could even be that some examiners lack experience with IMGs.16,17
The contrast between the results of the examination and the results of the ITERs might lead to the opinion that perhaps more weight should be given to the ITERs in determining a resident’s fitness for practice rather than relying solely on a terminal examination.
This analysis reveals that IMGs continue to have difficulties with passing the CCFP examination even when they have been carefully selected and have obtained residency training in a dedicated site. The results of the ITERs, however, seem to indicate that IMGs are seen by their teachers to be competent physicians who are ready for practice after 2 years of family practice residency training.
I thank Ms Karen Brown, Ms Sharon Hall, Ms Eva Chan, and Ms Anna Needs for their help in obtaining consents and collating the in-training evaluation reports from the 2 family practice residency sites. I also acknowledge the helpful comments from my colleagues Dr Jill Kernahan and Dr Betty Calam, and the statistical expertise of Dr George Pachev.
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.