|Home | About | Journals | Submit | Contact Us | Français|
To review the success of international medical graduates (IMGs) who are pursuing or have completed a Quebec residency training program and examinations.
We retrospectively reviewed IMGs’ success rates on the pre-residency Collège des médecins du Québec medical clinical sciences written examination and objective structured clinical examination, as well as on the post-residency Certification Examination in Family Medicine.
All IMGs taking their examinations between 2001 and 2008, inclusive, and Canadian and American graduates taking their examinations during this same period.
Success rates for IMGs on the pre-residency and post-residency examinations, compared with success rates for Canadian and American graduates.
Success rates on the pre-residency clinical examinations remained below 50% from 2001 to 2008 for IMGs. Similarly, during the same period, the average success rate on the Certification examination was 56.0% for IMGs, compared with 93.5% for Canadian and American medical graduates.
Despite pre-residency competency screening and in-program orientation and supports, a substantial number of IMGs in Quebec are not passing their Certification examinations. Another study is under way to analyze reasons for some IMGs’ lack of success and to find ways to help IMGs complete residency training successfully and pass the Certification examination.
Faire le point sur la réussite des diplômés internationaux en médecine (DIM) qui poursuivent ou ont complété un programme québécois de résidence ainsi que les examens.
Étude rétrospective des taux de réussite des DIM à l’examen écrit pré-résidence du Collège des médecins du Québec portant sur les sciences médicales cliniques de même qu’à l’examen de certification post-résidence en médecine familiale.
Tous les DIM ayant passé leurs examens entre 2001 et 2008 inclusivement et les diplômés canadiens et américains qui ont passé leurs examens durant la même période.
Taux de réussite des DIM aux examens avant et après la résidence comparativement aux taux de réussite des diplômés canadiens et américains.
Le taux de réussite des DIM aux examens cliniques pré-résidence sont demeurés sous les 50 % entre 2001 et 2008. De même, durant la même période, le taux de réussite des DIM aux examens de certification était de 56 %, comparativement à 93,5 % pour les diplômés canadiens et américains.
Malgré une vérification pré-résidence des compétences et la disponibilité d’un programme d’orientation et de support, un nombre appréciable de DIM au Québec ne réussissent pas aux examens de certification. Une étude est actuellement en cours pour comprendre les raisons de l’échec de certains DIM et trouver des moyens de les aider à réussir leur résidence et les examens de certification.
Like other Canadian provinces, Quebec faces an increasing shortage of family physicians. In addition to finding ways to encourage more Canadian medical graduates (CMGs) to enter family medicine training and practice, preparing international medical graduates (IMGs) for successful clinical practice is a possible solution to this shortage. In the past several years, substantial financial and human resources have been mobilized to support IMGs in Quebec so that once they are admitted to family medicine or specialty residency programs they can complete their training successfully. Ko et al, for example, found that IMGs and CMGs provided similar quality of care in the management of acute myocardial infarction: “the evidence supports the notion that carefully selected IMGs can provide quality health care to [acute myocardial infarction] patients, after passing national certification examinations and having time to assimilate into practice in Canada.”1
International medical graduates’ ability to provide high-quality care, not only in terms of skills but in terms of their relationships with patients, depends largely upon their training and examination performance. Part and Markert concluded that recent clinical experience and performance on standardized examinations were the 2 criteria most predictive of foreign-born IMGs’ first-year performance as internal medicine residents.2 Similarly, Tamblyn et al have highlighted the association between family medicine examination scores and the quality of medical practice.3
The Collège des médecins du Québec (CMQ) is the regulatory authority in Quebec that protects the public, promotes quality medicine, and grants permits to practise medicine in Quebec. The CMQ has evaluated IMGs for more than 25 years; from the early 1980s on, the CMQ has searched for reliable and valid assessment tools that can assist in comparing IMGs’ competencies with those of medical students graduating in Quebec.4 Medical graduates from schools that are not accredited by the Liaison Committee on Medical Education or the Committee on the Accreditation of Canadian Medical Schools are considered to be IMGs. Admittedly, this definition is broad, and IMGs are a heterogeneous group with varied academic backgrounds. Nevertheless, regardless of background or nation of training, all who fall into this category must complete a multi-step process to achieve recognition of the equivalency of their degrees.4 (Note that graduates of American medical schools are not considered to be IMGs, and so they do not need to go through this process.) The steps that IMGs must take to pursue residency in Quebec are as follows:
Those IMGs whose degrees are recognized through this process can apply to 1 of the 4 Quebec faculties of medicine for family medicine or specialty residency positions if they meet Quebec government requirements for funding eligibility. All residency programs in Quebec are accredited through the same process and with the same standards used in the rest of Canada.
International medical graduates who complete a family medicine residency program are eligible to take the family medicine Certification examination in Quebec in order to obtain a permit to practise. The examination comprises the College of Family Physicians of Canada (CFPC) Certification examination and the MCCQE Part II. Candidates who have already passed the MCCQE Part II do not have to take it again. (Until October 2006, the Quebec Certification examination also included the CMQ family medicine OSCE.)
An important first step in evaluating and improving the performance of IMGs who are pursuing or have completed a Quebec residency program is reviewing their success on examinations. In this study, we evaluated and compared IMGs’ and CMGs’ results on pre-residency and family medicine Certification examinations. The next step, which is under way, is to analyze variables from the files of IMGs to see if predictive success factors can be identified.
After approval by the CMQ Ethics Committee on Research, a retrospective evaluation was conducted of IMGs’ examination performances. This evaluation involved reviewing IMGs’ results on the CMQ medical clinical sciences examination (written examination and OSCE) from 2001 to 2008, as well as comparing IMGs’ results with those of Canadian and American physicians on CMQ family medicine Certification examinations from 2001 to 2008. All IMGs and CMGs taking the examinations in those years were included. For the purpose of the study, the term CMGs includes graduates of American medical schools accredited by the Liaison Committee on Medical Education. Independent-samples t tests were used to compare the mean scores of CMGs with those of IMGs.
The OSCE passing score is 60%, which is the same as the one used at the Laval University Faculty of Medicine for this type of examination.
The reliability of scores for the different OSCEs was assessed using Cronbach α internal consistency coefficients. We used standardized scores to compare the results from the different examinations, an approach that Tamblyn and colleagues have described.3 For the analyses, we used averaged standardized scores so that we were able to compare the different groups’ performances on examinations by putting the results on the same scale.3
Generalizability studies were performed to analyze whether the circuits in which candidates sat their CMQ OSCE examinations had an effect on their scores. Candidates were nested in circuits by cases (Ca:Ci × Cases) as described by Brailovsky et al.5
Table 1 outlines IMG candidates’ success rates for the CMQ medical clinical sciences written examination and IMG and Laval University CMG candidates’ success rates for the CMQ OSCE. Candidates are allowed 3 attempts at each of these examinations. The success rates described do not differentiate between first-time examination takers and repeaters, although determining whether repeaters pass on subsequent attempts or consistently do poorly on examinations would be valuable.
The IMGs’ success rate for the written examination is higher than that for the clinical OSCE. Scores for written examinations are known to correlate poorly with scores obtained in examinations that measure clinical competence.6,7 Clinical competence examinations require other skill sets, such as history-taking and patient management skills. The success rate for the CMQ OSCE examination remains below 50% for IMGs, even though preexamination OSCE workshops have been offered for the past 5 years. Laval University students completing their final year of undergraduate medical studies have an average success rate of about 98% when they take this exit OSCE before receiving their medical degrees.8
Cronbach α coefficients varied from 0.784 to 0.878 for the different examination sessions. The case-to-case homogeneity to measure the construct of clinical competence in each examination was supported through the item-item total correlations (> 0.300 in 93% of cases). The calculation of the coefficient “α minus the item” for each case demonstrated that more than 94% of cases contributed positively to the reliability of the examinations.
Our generalizability studies indicated that the percentage of variance explained by circuits was almost nil, and the percentage of variance explained by cases by circuits was very low (< 3% in all examinations). This result was considered sufficient indication that candidates were not penalized or advantaged by the examination circuit where they sat the examination.5
When we analyzed candidates’ performance on the different components of the OSCE, we found that IMGs perform poorly in history-taking, investigation, differential diagnosis, and treatment. When we used standardized scores to compare IMGs’ and CMGs’ performances, the IMGs’ z scores were approximately 1.5 to 2 standard deviations lower than CMGs’ scores on the same skills. The IMGs’ mean aggregate global scores were 15% to 20% lower than CMGs’ scores.
Table 2 shows that, from 2001 to 2008, the success rates of IMGs in Quebec were significantly lower than those of CMGs graduating from Quebec faculties of medicine. Again, the success rates described in Table 2 do not differentiate between first-time examination takers and repeaters. The table also shows that the removal of the additional Quebec family medicine examination in 2007 and 2008 did not substantially improve the success rates for IMGs. Between 2001 and 2008, the average success rate for IMGs was 56.0%. Quebec medical school graduates who completed the same family medicine residency programs had an average success rate of 93.5% on the same Certification examinations during this period (Table 2).
While the Quebec IMG numbers are small, the findings are confirmed by results on the CFPC examination for all Canadian family medicine residency programs (Tables 3 and and4).4). In 2007, CMGs’ overall success rate on the CFPC examination was 90.4%, which is much higher than the 66.0% overall success rate for IMGs (Table 3). The discrepancy in success rates was confirmed in 2008.
In spite of the pre-residency assessment tools used in Quebec, including the assessment of medical knowledge and clinical skills, a substantial number of IMGs accepted into residency training have difficulty completing their programs and passing the Certification Examination in Family Medicine. Data from the Conférence des recteurs et des principaux des universités du Québec, available on the CMQ website, indicate that at least 10% of IMG residents leave their family medicine residency programs without completing them, and that 55% require additional training.9
One hypothesis, which Bates and Andrew proposed in 2001, is that unsuccessful IMGs have been unable to integrate into the Canadian health system.10 The authors discuss the challenges that IMGs face when they enter the North American postgraduate educational system: difficulties with English; differences in medical education; length of time since medical school graduation and clinical experience; financial and family obligations; traumatic experiences; and different cultural beliefs about gender roles, communication, and similar issues.10 In that article and another on the 7-year experience of the program to assess IMGs in British Columbia,11 they therefore recommend strategies such as an evaluation process that assesses attitudes and behaviour, clear articulation of the program’s expectations and North American expectations for physicians, and experienced faculty who can explore the root causes of academic problems.10,11
The IMGs in our study, however, had had some time to adapt to the Canadian system, although transitioning from one professional and personal culture to another is, of course, an ongoing, long-term process. Our IMGs passed the screening competency examinations leading to recognition of the equivalency of their diplomas, and they spent at least 24 months in a Canadian family medicine residency program. In addition, since 2004, the 4 Quebec family medicine programs have organized a pre–family medicine residency orientation program for IMGs to help them integrate into the Canadian health care system and prepare for their residencies. During residency, family medicine tutors and clinical preceptors devote additional time to helping these physicians integrate and acquire the competencies needed for independent practice.
We offer some other tentative explanations. Possibly the poorer performance is not attributable to a lack of medical knowledge and competency in clinical skills, but rather to how and when an IMG learns to translate the knowledge and integrate it with clinical decision making—an ability that CMGs acquire during their clerkships, if not earlier in their training. Recently, Palés et al surveyed medical graduates in Spain about their acquisition of learning outcomes.12 Eleven domains were examined. Students reported that they felt competent in 8 domains—clinical skills, practical procedures, patient investigation, health promotion, communication skills, attitudes and ethical and legal responsibilities, role of the doctor, and personal development—but only partially competent in 3 domains: patient management, medical information search skills, and decision-making skills and clinical reasoning and judgment.12 On the other hand, data from an ongoing study by Brailovsky and MacLellan suggest that IMGs who return to medical school, usually at a clerkship level, complete their family medicine residency and Certification examinations in a fashion comparable to that of CMGs.
Another hypothesis is that, unlike CMGs’ accredited, homogeneous medical education, IMGs’ undergraduate medical education is variable, and they are a diverse group. For some IMGs, therefore, the educational process might not provide the foundation in knowledge, skills, and attitudes they need to succeed in Canadian residency training and Certification examinations. For example, van Zanten and Boulet’s recent study revealed considerable variability in undergraduate medical education and performance on the United States Medical Licensing Examination among students from medical schools in different Caribbean nations.13 This variability was evident on the 3 examination components (basic science, clinical knowledge, and clinical skills and clinical skills assessment), although, interestingly, the pass rates were generally higher for the clinical skills component.
The main limitation of our study is a lack of specific information on which IMGs succeed in passing examinations. The study also did not reveal predictors of success. However, the preliminary finding that fewer IMGs than CMGs pass examinations of clinical skills will guide our future research on improving IMGs’ examination performance and ensuring their successful entry into practice.
In Quebec, IMGs have a much lower success rate on the Certification Examination in Family Medicine than CMGs do. This is the case even though the IMGs have passed several screening competency examinations and have successfully completed a 2-year accredited family medicine residency program.
Many questions remain about the reasons for family medicine IMGs’ lack of success on examinations. A clearer understanding of which IMGs are more likely to fail or pass might be achieved by exploring differences between candidates who grew up in Canada but attended international medical schools and candidates raised in different nations. In addition, a closer look could be taken at whether candidates’ performance differs depending on whether or not they attended medical schools in nations where accreditation criteria are similar to those used by American and Canadian medical schools.14
Currently, we are conducting a retrospective analysis of several variables in the 2000 to 2007 files of IMGs who requested that the CMQ recognize the equivalency of their medical diplomas. Data concerning IMGs who complete Quebec clerkships are also being analyzed. One goal of this new study is to explore why many IMGs who pass the pre-residency screening examinations fail the post-residency Certification examination. Another goal is to identify factors that can help IMGs complete their residency training successfully, pass their Certification examinations, and enter family practice in Quebec.
Funding for this study was provided by the Collège des médecins du Québec and by the Ministère de la santé et des services sociaux du Québec. We thank the Collège des médecins du Québec Medical Education Division staff, particularly Ginette Plante-Lauzier. Alison Harkin provided editorial assistance with the manuscript.
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
All the authors contributed to concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.