All data used in the analyses were abstracted in anonymous form from the combined Universities and Colleges Admissions Service (UCAS) and UKCAT database. Figure 1​1 depicts the data included in the final modelling process. The structure of offer data is complex in that information relates to both individual candidates and specific applications to each institution. The dependent variable was the outcome (offer or rejection) of an “application event” involving one candidate and one specific medical school. To each of these application events we attributed a unique identifier. We included both conditional (dependent on attainment of A level or equivalent examinations) and unconditional offers (academic requirements for entry fulfilled) as a positive outcome and analysed these separately, because unconditional offers also depended on a candidate with a conditional offer obtaining the required grades at A level or equivalent, and therefore these outcomes were likely to differ in their predictors. We summed and converted the UKCAT subtest scores to standardised z scores to create an overall metric of UKCAT test ability, in comparison with a student’s peers. Table 1​1 summarises the independent variables utilised in the modelling process. Male sex was also included as a disadvantaged category for two reasons; there is clear evidence that females are over-represented in medical school intakes compared with the UK population,^{20 21} and univariate analysis showed that male sex was associated with a reduced odds of an offer. We used both standardised UKCAT and A level (or equivalent) tariff scores in both a continuous and a dichotomous form, although not in the same analyses owing to problems of multicollinearity. All data management and analyses were carried out using STATA MP version 12.²²

For the statistical modelling we utilised observations with complete data only on all variables (fig 1). Tables 3​3 and 4​4 depict, respectively, the results of multilevel logistic univariate regressions for the probability of conditional and unconditional offers. Even after controlling for candidate level effects, the unadjusted odds of receiving a conditional offer varied significantly between all three groups of medical schools for most widening participation categories; age and sex being the only exceptions (table 3). The intraclass correlation in this case represents the residual of the latent response variable of each individual candidate; each candidate is conceptualised as having a certain level of a latent trait (an unobserved normally distributed variable). The level of this trait would be manifest in the likelihood of the candidate receiving an offer, thus this trait could be conceptualised as “offerability.” In a multiple logistic regression, the intraclass correlation is the correlation within each individual candidate, once fixed effects are controlled for, such as academic attainment. Thus intraclass correlations of 0.3 to 0.5 (table 3) would be consistent with moderate individual level effects for candidates (as opposed to the effect of group membership of the medical schools applied to). As the odds ratios are raw, such individual level effects would include UKCAT and predicted or actual A level (or equivalent) performance. On average the intraclass correlations were highest for medical schools using UKCAT as a threshold and lowest for those using UKCAT in borderline cases, with all between group differences being significant at the P<0.05 level according to a Wilcoxon test (table 3). This would suggest that individual candidate effects and performance are more predictive of the provision of a conditional offer in those institutions with a stronger use of the UKCAT. A similar pattern was observed for the raw odds of obtaining an unconditional offer, with the association between applicant age and sex weaker than for the other variables. No significant association was observed with applicant age once only unconditional offers were considered. The intraclass correlations were then smaller in magnitude, reflecting weaker candidate level effects. The intraclass correlations for medical schools using UKCAT as a factor were, on average, significantly larger than for the other medical school groups (P<0.001), reflecting the increased predictive power of individual compared with group or unmodelled effects. A plausible reason for this may be a greater emphasis, for example, on A level performance when providing an offer, as some medical schools may insist on the original conditions of an offer being met, whereas other may allow a certain latitude, especially if the number of applicants accepting offers falls short of expectation.

Tables 5​5 and 6​6 show, respectively, the results for the separate multilevel multiple logistic regressions for the probability of conditional and unconditional offers. The results in table 5 depict three varying models for the prediction of a conditional offer. For those applications relating to medical schools using UKCAT as borderline, six independent and statistically significant predictor variables existed, including sex, socioeconomic status, ethnicity, type of school attended, and academic attainment, with academic achievement being most strongly predictive of an offer. The UKCAT score was a relatively weak independent predictor in applications relating to this group of medical schools in that the odds of receiving a conditional offer were only increased by roughly 20% for every standard deviation of standardised score above the mean that was scored (odds ratio 1.23, 95% confidence interval 1.11 to 1.37). The model for applications to medical schools that used UKCAT in borderline cases had three statistically significant interaction terms; age and academic attainment, sex and school type attended, and UKCAT score and English as a second language. The first of these indicates that for candidates aged more than 20 at the time of application, A level (or equivalent) attainment conferred relatively less advantage when a conditional offer was made compared with younger applicants. This is highly plausible given that many older applicants will be applying on the basis of a university degree rather than on A level attainment. The interaction term between male sex and school type attended implies that the relative disadvantage of having attended a state school was reduced for men compared with women. Lastly, the interaction between UKCAT score and English as a second language suggests that an increased UKCAT score conferred a greater advantage for those for whom English was a second language compared with native English speakers. Medical schools using the UKCAT score as a factor in the admissions process had five significant and independent predictors of the probability of a conditional offer. In contrast with the medical schools using UKCAT score in borderline cases, the strongest of these was standardised UKCAT score, with an increase in 1 SD above the mean more than doubling the odds of an offer, all other factors being equal (odds ratio 2.31, 95% confidence interval 2.07 to 2.58). As in the model with medical schools that used the UKCAT score in borderline cases, interactions were significant between academic attainment and age and UKCAT score and English as a second language. In addition the interaction term for academic attainment and UKCAT score was significant, indicating some degree of synergy between A level (or equivalent) grades and UKCAT score when predicting the probability of a conditional offer. For medical schools with UKCAT used as a threshold score in the admissions process the only significant independent predictors were UKCAT score (odds ratio 8.59, 95% confidence interval 6.96 to 10.62) and, to a lesser extent, academic attainment (1.63, 1.46 to 1.82). As with the other two categories of university the interaction between UKCAT score and English as second language was significant. Two other interaction terms were also significant. Firstly, the interaction between sex and UKCAT score, suggesting that for males an increase in UKCAT score conferred less of an advantage than for females when seeking a conditional place offer. This is interesting given that, on average, men outperform women on the UKCAT.¹⁶ Secondly, the interaction between socioeconomic status and English as a second language was significant and suggested that those from a lower socioeconomic background who were not native English speakers were relatively disadvantaged compared with candidates who did not belong to this subgroup.

In total, data related to 4456 medical school entrants to the 22 universities analysed. Table 9​9 and figure 2​2 depict the breakdown of the proportion of each medical school intake by widening participation status. Some categories of institutions appeared fairly homogenous in relation to the proportion of candidates of widening participation status taking up places, whereas others were relatively diverse. For example, medical schools using UKCAT as a threshold consistently admitted more than around 55% of their medical students from state schools, whereas more variation was observed in medical schools using UKCAT as a factor in the admissions process (fig 2).

To ensure nesting, only observations from 2679 entrants with complete data were utilised, out of the total of 4456 individuals (60%). Table 11​11 shows the results for sex, ethnicity, English as second language, and age. The manner in which UKCAT was used was entered as a three level factor variable. Unless stated otherwise, the group of medical schools that used UKCAT as a factor was used as the baseline category. This simplified the presentation of the results because although all the combinations of interactions were explored, those that were significant (P<0.05) mainly involved this group of medical schools as one of the comparators. Several of the effects observed in the raw univariate single level analysis (table 9) were attenuated or not observed after controlling for individual institutional effects and potential confounding. For example, entrants to medical schools that used UKCAT as a factor or as a threshold continued to be more likely to be male than those admitted to medical schools that used UKCAT in borderline cases. In contrast, the trend for entrants to medical schools that used UKCAT as a factor or threshold to be over 20 years at application was no longer apparent after controlling for individual institutional and the effect other sociodemographic variables. The intraclass correlation (0.22) was larger than for the other widening participation categories depicted in table 11. This would suggest a relatively large institutional level effect compared, for example, with that for the association with an entrant’s sex (intraclass correlation=0.002). This is supported by the descriptive statistics observed in table 9 and figure 2 where the relatively large variation between institutions in the same UKCAT usage category for intake of older students could be contrasted with the relative homogeneity within groups for the proportion of male entrants. These findings would suggest that the trend for mature entrants reflects one or more individual universities taking in either large or small proportions of mature students, rather than an effect associated with type of UKCAT usage (table 9). However, not all the trends observed were attenuated after controlling for institutional effects and potential confounding variables; the difference between the odds of an entrant to a medical school using UKCAT in borderline cases compared with one from those using UKCAT as a factor having English as a second language actually increased in significance (P value from 0.2 to 0.04) once this more complex modelling was performed.

Of 16435 total applicants, 1189 were initially given 1507 unconditional offers, having already achieved the required academic conditions at the time of application. A further 5368 candidates were initially provided with 7363 conditional offers (an average of 1.4 offers per successful candidate). Of these individuals, 4470 (83%) eventually obtained an unconditional offer, having obtained sufficient academic qualifications to satisfy the relevant institutions that entry was merited. In most, but not all cases, this would involve achieving the initial A level (or equivalent) grades set as part of the conditions of the provisional offer. To explore the predictors of an application being converted from a conditional to an unconditional offer data were analysed from 3290 applications relating to 2803 candidates holding conditional offers where complete data were available. A multilevel multiple logistic regression model was built up in a stepwise fashion. A random intercept for each candidate was introduced, although it was not possible to model the random effects associated with individual institutions simultaneously. All possible combinations of interactions were explored and those found to be significant and independent predictors of successfully converting a conditional offer to an unconditional one were included in the final model (table 13​13).). The strongest predictor of success was academic attainment (odds ratio 10.35, 95% confidence interval 8.01 to 13.27, P<0.001), in these cases measured using a standardised UCAS tariff. Indeed, the only other significant and independent predictor was the application being associated with a medical school using UKCAT in borderline cases or a factor in the admissions process (compared with the remaining group). However, a trend of borderline statistical significance was observed for conditional offers associated with candidates of low socioeconomic status to have less probability of being converted to unconditional ones (0.42, 0.18 to 0.99, P=0.05). Two interactions were also significant. Firstly, an interaction between mature applicant status and academic achievement was observed (0.11, 0.07 to 0.17, P<0.001). This is interpreted as increased A level achievement being relatively less of an advantage to those applying to holding conditional offers and over 20 years old, compared with younger applicants. In practice many mature candidates will be applying on the basis of a university degree, either achieved or pending, rather than A level achievement, and therefore this finding is readily understood in this context. Perhaps a less anticipated interaction was that observed for low socioeconomic status and non-white ethnicity (8.56, 1.81 to 40.43, P=0.007). This implied that those applicants holding conditional offers from low socioeconomic backgrounds were more likely to convert them to unconditional ones if they were of non-white than of white ethnicity.

After controlling for factors such as UKCAT and academic performance we estimated that non-white applicants were still around 40% less likely to receive a conditional offer from a medical school using UKCAT in borderline cases than a candidate reporting their ethnicity as white (table 5); an almost identical finding to that reported by a study of ethnic minority applicants to medical school in 1990.²⁷ A subsequent study from candidates for admission from 1996 and 1997 also reported a relative disadvantage for applicants from ethnic minority groups, although the authors noted variation between individual medical schools unrelated to the overall number of non-white applicants to each institution. The present results from the separate models were largely echoed by the pooled analysis, which directly compared the three types of medical school. Some authors have cited such findings as evidence of institutional discrimination.²⁸ Although UCAS does not pass on information of ethnicity to universities it may be apparent from a candidate’s name or subsequently at interview. Evidence of racial discrimination has been uncovered previously within a medical school setting and as such cannot be entirely ruled out, deserving further exploration.²⁹ It has been previously highlighted that racism is defined as an attitude, and thus studies, such as the present one, that simply examine the probabilities of an offer to an applicant are unable to provide direct evidence of this.³⁰ Moreover, there may be more innocent, unmodelled effects that may explain some of these observations. For example, it may be that candidates from certain under-represented groups are less likely to have a relative in the medical profession and therefore find it more difficult to gain relevant work experience that would bolster their application and improve their performance at interview. One possibility is that universities using the UKCAT as a threshold may place less emphasis on personal statements or the interview in deciding on offers and mitigating against this effect during the offer process (tables 7 and 8). Owing to the nature of the structure of offer data, individual institutional effects could not be controlled for at the same time as within candidate effects. However, whether these observations are associated with a group effect or individual universities they deserve further exploration.

This is the first study to report in detail on the impact of the UKCAT on medical school selection. Missing data were sparse except for two predictor variables: socioeconomic background and school qualifications. Therefore some caution must be exercised when drawing conclusions about the role of these predictors. However, the findings from the sensitivity analysis using multiple imputation suggested that the data were largely missing at random and thus should not affect the conclusions drawn from the multilevel multiple logistic regression models predicting offers. The case of missing advanced qualification data in relation to the prediction of conditional offers could be considered something of an exception to this; when these missing data were imputed the findings showed a relatively greater degree of variation compared with the results for the non-imputed data. Although graduate and other widening access courses make up a small fraction of overall UK medical school intake, these findings are unlikely to generalise to this population of applicants, who were excluded from the analysis.