|Home | About | Journals | Submit | Contact Us | Français|
Surveys have suggested one of the most important determinants of orthopaedic resident selection is completion of an orthopaedic clerkship at the program director’s institution. The purpose of this study was to further elucidate the significance of visiting externships on the resident selection process. We retrospectively reviewed data for all medical students applying for orthopaedic surgery residency from six medical schools between 2006 and 2008, for a total of 143 applicants. Univariate and multivariate regression analyses were used to compare students who matched successfully versus those who did not in terms of number of away rotations, United States Medical Licensing Examination® scores, class rank, and other objective factors. Of the 143 medical students, 19 did not match in orthopaedics (13.3%), whereas the remaining 124 matched. On multiple logistic regression analysis, whether a student did more than one home rotation, how many away rotations a student performed, and United States Medical Licensing Examination® Step 1 score were factors in the odds of match success. Orthopaedic surgery is one of the most competitive specialties in medicine; the away rotation remains an important factor in match success.
Orthopaedic surgery continues to be one of the most competitive residency training programs in which to match. In more recent years, the specialty has filled greater than 99% of Postgraduate Year 1 positions through the National Residency Matching Program, annually ranking at the top in terms of percentage of Postgraduate Year 1 positions filled with graduating US senior medical students [6, 8, 9]. Another marker of the exclusivity of orthopaedics is that the number of applicants greatly outnumbers the number of available positions, often with almost two applicants competing for every one available Postgraduate Year 1 position . In 2007, for example, there were 1.4 total applicants per available position and 577 of 719 (80.3%) US seniors who applied to orthopaedic surgery successfully matched .
Despite this competitiveness, there is relatively little information available in the orthopaedic literature regarding which criteria are weighed most heavily by residency programs in selecting residents. Most of the information available to medical students and their advisors consists of surveys of residency program directors [1, 2]. To our knowledge, only one previous study compared the attributes of matched applicants against applicants that did not successfully match in orthopaedic surgery . This study, however, did not consider the importance of visiting rotations on match success. The visiting rotation more recently has been rated as the most important resident selection criterion in surveys of orthopaedic residency program directors and faculty, although to our knowledge there are no data available regarding how successful students are in matching at the particular programs at which they rotated [1, 2]. The importance of the visiting rotation is noteworthy because, if significant, it could affect when, where, and how many orthopaedic electives each applicant decides to complete before applying for residency.
Another study showed competitive specialties rely heavily on objective academic criteria when selecting residents . Although little objective data exist on the subject, anecdotal evidence strongly suggests competitive specialties such as orthopaedic surgery may use objective criteria such as United States Medical Licensing Examination® (USMLE) scores and Alpha Omega Alpha (AOA) membership to screen students during initial narrowing of large applicant pools. Although the USMLE is not validated for scores other than pass/fail, there is at least anecdotal evidence that programs use this score as a way of screening students. This helps mitigate the increasing burden on individual programs to sort through hundreds of applications for a small number of available positions each year. These objective criteria also serve as a proxy for a cognitive skill set to help fill the absence of validated measures of resident competence; the only current objective measure for accreditation by the Residency Review Committee is a 75% pass rate by residents on Part I of the American Board of Orthopaedic Surgery examination .
This shifts the burden to the applicant, who, with little reliable information available to base choices, increasingly feels the pressure to make early career decisions, focus on the numbers’ game of top board scores and high class ranks, and apply to a burdensome number of programs. In a yearly survey during the late 1990s, the lowest number of programs applied to by any student was approximately 20, with the occasional student applying to more than 80 programs . Such an environment makes advising and mentoring medical students interested in orthopaedic surgery exceedingly difficult and provides a major foundation for the current study.
The purpose of this study was to further elucidate the significance of objective criteria commonly cited as important in the resident selection process by comparing medical students who matched in orthopaedic surgery against those who did not match, specifically to elucidate the role of the away rotation on match success. The four factors used for comparison were the number of rotations each applicant completed in orthopaedic surgery, at their home medical school and at other institutions, USMLE Step 1 score, rank in medical school, and AOA membership. In surveys, these attributes consistently rank as the most important objective criteria for choosing orthopaedic residents [1, 2]. It is our hypothesis that successful candidates will have completed more orthopaedic clerkships at more institutions than students who were unsuccessful in the match.
As our primary outcome of interest was measured by our multiple logistic regression model, we did a power analysis using the method described by Hsieh et al.  to calculate sample size for multiple logistic regression. The calculation assumes the following parameters: alpha of 0.05, power of 0.8, overall probability of an event (matching) of 80% , probability of matching having done one or more away rotations of 90%, and probability of matching without doing an away rotation of 60%. We also assumed 75% of students would have done away rotations. Using this calculation for a univariate model, our sample size would have to be 73 students. Assuming a variance inflation factor of 75% owing to confounders in the model, we would need 128 students in our study. We exceeded this number.
We emailed US medical schools to determine if they would be willing to participate in our study and allow us to have information about students’ records who had applied to orthopaedic surgery residency programs in the last three match cycles. Schools who gave us a positive response were included in the study. We retrospectively reviewed data for all medical students applying for residency in orthopaedic surgery from those six medical schools between 2006 and 2008, for a total of 143 applicants. All six schools were US allopathic medical schools, two private in an urban setting, one private in a suburban setting, one private in a rural setting, and two public in an urban setting. Students first were grouped according to whether they successfully matched into an orthopaedic residency program. Any student who did not successfully match in orthopaedic surgery but either successfully scrambled into another field or took a year off and reapplied to orthopaedics we considered as having not matched in orthopaedics and did not include any subsequent years of application data for that student. USMLE Step 1 score, AOA status, and class rank data were recorded for each student. All six schools in this study use the AOA honor society as a way to distinguish excellence in each class. For purposes of this study, each student’s class rank was broken down into five groups: top 10 percentile, 11th to 25th percentiles, 26th to 50th percentiles, 51st to 75th percentiles, and bottom 25th percentile. The number of orthopaedic clerkships each student completed at their home medical school and at other medical schools also was recorded. Some schools offer short, less than 4-week, introductory rotations in orthopaedic surgery during the required clerkship year. These shorter rotations were not included in this study; we counted only full, month-long rotations in orthopaedics completed by each student.
Statistical analysis was performed using SPSS® 15.0 data analysis software (SPSS Inc, Chicago, IL). Differences in USMLE Step 1 scores between groups were determined with Student’s t test for independent samples, equal variances not assumed. Differences between groups in class rank categories were determined with the chi square test with Yates correction. Differences between groups in AOA membership and whether a student did more than one rotation at home were performed with Fisher’s exact test. Following this, univariate logistic regression analysis was performed for all of the above variables, along with the number of visiting rotations each student completed. A subgroup analysis comparing students who did two rotations at outside institutions against those who completed three or more away rotations was performed using Student’s t test for independent samples and Fisher’s exact test. All variables were entered using the backward stepwise method of multivariate binary logistic regression to assess the binary outcome of whether the student matched into orthopaedic surgery, using the likelihood ratio statistic for selection of variables with criteria 0.10 for removal. Variables entered were USMLE Step 1 score (continuous), AOA status, class rank (categorical), whether a student did zero, one, or more than one home rotation, and number of away rotations. A binary grouping based on USMLE Step 1 score would have been arbitrary; therefore, the parameter was made continuous for the purposes of regression analysis.
USMLE scores, AOA status, class rank, and number of home rotations were not independently predictive of whether students successfully matched into an orthopaedic residency. Of the 143 medical students applying to orthopaedics, 19 did not match (13.3%), whereas the remaining 124 matched (86.7%). The results of univariate analysis found USMLE Step 1 scores (p = 0.190), AOA status (p = 0.637), class rank (p = 0.823), and whether a student did more than one rotation at their home institution (p = 0.239) were not different between the two groups (Table 1).
Students who performed two or three or more away rotations had better odds of matching at an orthopaedic program (p < 0.001 and p = 0.035, respectively). None of the above-mentioned confounders were factors in univariate binary logistic regression (Table 2). In univariate analysis, completing one away rotation was not a factor in the probability of matching; however, performing two or three or more away rotations increased (p < 0.001 and p = 0.035, respectively) the odds of a student matching. Students who did two away rotations tended to be stronger academically, as they were more likely to be in AOA (p = 0.040), to be in the top 10% of their class (p = 0.023), and to have higher USMLE Step 1 scores (p < 0.001) than students who did three or more away rotations (Table 3).
Whether a student did more than one home rotation (p < 0.001), how many away rotations a student did (p = 0.011, p < 0.001, and p < 0.001 for one, two, and three or more away rotations, respectively), and USMLE Step 1 score (p = 0.004) were factors in the odds of match success (Table 4). The Hosmer-Lemenshow goodness-of-fit test statistic was 1.077 (p = 0.998), which indicates our model fit the data well. The model did have a substantial degree of variability owing to the small number of students who did not match. The Naeglekerke R2 value for the final model was 0.623, and although this is not completely analogous to the R2 used in linear regression, it suggests the model does not explain all of the variability in the sample.
Orthopaedic surgery is consistently one of the most competitive residency fields in which to match for US senior medical students. The rationale for performing our study was to provide the incoming third- or fourth-year medical student with objective information regarding what criteria are important to match success. Specifically, we wished to delineate the role of the away rotation in match success. Doing away rotations has the potential to help an applicant in numerous ways. First, it allows applicants the opportunity to impress residents and attendings with their work ethic and ability to fit into the team. Second, it allows the student to get letters of recommendation from the orthopaedic staff at that institution, which, if written in strong support of the student, could help the student’s chance of matching not only at that school but at other programs as well. Third, it may allow the applicant to get involved in research projects or mentorship that they otherwise would not have had access to.
Strengths of our study included a representative sample of public and private medical schools in urban, suburban, and rural settings, thus providing increased external validity of our results. We were able to provide quantitative data for factors that have helped to determine match success in the past several match cycles. We have uncovered several interesting findings that may help medical students organize their fourth-year medical school schedules to provide a more comprehensive strategy to match into orthopaedic surgery and provide valuable data for mentors to use. Weaknesses include an inability to quantitatively model certain factors that certainly play a role in match success such as interviews, attending-to-attending conversations on an applicant’s behalf, and letters of recommendation [1–3]. Another limitation of this study is the limited number of students studied. The variability of the study was large because the unmatched group contained only 19 students; as such, the odds ratios and confidence intervals found in the multivariate analysis are large and should be interpreted with caution. However, because the match rate of US seniors in 2007 was 80.3% and the match rate in our group was only slightly higher, we believe our sample is representative of what might be found throughout the United States. The lack of demographic data recorded for the applicants is also a limitation. This is especially true in light of the fact that the only other previous study comparing matched and unmatched students showed younger age to be a significant difference between the two groups . It is possible some of the distinction between the two groups could have been explained by demographic differences, such as age, gender, or reputation of medical school attended. These possible confounding variables are not accounted for in this study. The study also could not take into account some other factors that may play a critical role in the selection process, such as the interview day, research, letters of recommendation, personal statement, or curriculum vitae. Many of these factors may be difficult to model in a quantitative fashion, and surely other individual factors play a role in the match process. Some selection bias is likely present, because schools were selected based on their willingness to participate. Some schools would not participate because they thought the data were too sensitive, despite the fact that it was blinded to review and analysis. We did not compare match success of individual schools for two reasons. The quality of the medical school (eg, its reputation) may have no correlation to the ability of the orthopaedic department to assist their students matching. Second, owing to the number of applicants per school per year, we were unable to statistically correlate match success based on medical school attended. Likewise, we did not investigate the likelihood of a student matching at a program where they performed a rotation. There are numerous factors that may impact rotators matching at an institution at which they did a rotation or rotations. Our data were blinded to the student and, therefore, this information was not available.
Our study has numerous interesting findings. First, we found away rotations were a strong independent predictor of match success. At first, it seemed paradoxical to us on univariate analysis that students who did two away rotations would fare better than those who did three or greater. This was reconciled when we analyzed the subgroup data, which indicated students who did three or greater rotations had lower USMLE Step 1 scores, were less likely to be AOA members, and were less likely to be in the top 10% of their class. This was further clarified when our multivariate analysis showed USMLE Step 1 scores were important to match success, and the 95% confidence intervals of students who did two away rotations and those who did three or greater rotations indicated these factors were not significantly different. This suggests USMLE Step 1 score may confound the relationship an away rotation has on match success. In addition, among other factors, because Electronic Residency Application Service only allows a certain number of letters of recommendation, the marginal impact of increasing the number of away rotations may diminish after two. Therefore, it is possible, beyond two rotations, the student may only help chances of matching at the specific program where they have rotated. The home rotation is another interesting entity identified in this study. We consider this rotation to be of special significance, because it potentially could allow a student to gain increased exposure to individual attendings and residents whom they are likely to see more often than students rotating from outside institutions. This may afford them opportunities such as research and mentoring, which otherwise may not have been available to them by not being at the institution (the way an away rotator would). We considered this type of rotation separately for this reason. Bajaj and Carmichael  found residency directors thought performance in an externship, class rank, and interview performance were the most important factors in match success; whereas applicants thought USMLE scores, externships, and letters of recommendation were the most important factors. Another study by Clark et al.  suggested board scores and medical school grades could affect eventual match success. An analysis by Bernstein et al.  suggested program directors believed performing an away rotation at the program director’s institution, USMLE Step 1 scores, and rank in medical school were the most important characteristics medical students can have. Interestingly, when asked what the most important characteristic a letter of recommendation could have, the respondents answered “the letter is written by someone I know” . This suggests a potential avenue medical students can use to their advantage when selecting away rotations and requesting letters of recommendation. In contrast to the studies by Clark et al.  and Bajaj and Carmichael , we found class rank and medical school grades had little effect on match success. This also could be a testimony to the strength of the applicant pool, or it could be a result of varied sampling techniques between our study and others of this type. We found USMLE scores and away rotations both are important when considered as part of the total portfolio, which confirms data from the studies by Clark et al. , Bajaj and Carmichael , and Bernstein et al. .
The away rotation is an important factor in selection of candidates for residency in orthopaedic surgery. It seems apparent students who are weaker academically try to bolster their chances of match success by performing more away rotations. This is somewhat concerning because students will have 5 or more years to learn orthopaedic surgery and only the last remaining months of medical school to learn other subspecialties in medicine. In addition, our data indicate performing more than two away rotations may not improve a student’s chances of matching. That is not to say, if a student puts in a stellar performance on a third away rotation, his or her chances of matching at that institution will not improve. In general, however, it seems the third away rotation will not improve overall chances of matching. Currently in our profession, it is unclear how to optimally evaluate prospective residents. This is particularly true because of the excellent quality of applicants orthopaedic programs have received. Away rotations provide programs with a firsthand look at the affective domain of an applicant that may set them apart from other applicants. Validated metrics for future resident performance need to be defined to facilitate the resident selection process Our data do not allow us to determine what specifically helps a student match following an away rotation Additional studies are necessary to determine whether mentorship, letters of recommendation, or the actual audition is the most important factor in match success.
We thank all of the medical schools who participated in our survey.
Each author certifies that he or she has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.