Descriptive statistics: applications
Overall, the percentages of each category of candidate (where the specific sociodemographic variable was available) were: 44.8% male sex, 39.8% self reported ethnicity as non-white, 21.1% English as a second language, 22.8% aged over 20 years at the time of application, 64.5% attended a non-fee paying or non-grammar school, 5.4% reported being from a non-professional social background, and 40.5% reported academic attainment below the level of AAB at A level (or equivalent). Table 1 shows the proportion of each category of candidate applying to the different categories of medical school along with the standard deviation of the percentages across individual institutions within each group. There was a number of modest, albeit statistically significant, differences between the proportions of widening participation candidates applying to each category of medical school (table 1). The most noticeable difference was between medical schools that used the UKCAT score in borderline cases and those that used it as a factor in admissions, with candidates who achieved relatively low grades at A level or equivalent being more likely to apply to the institutions that used UKCAT as a factor. In general, the variation between proportions of applications to institutions in the same category of UKCAT usage was modest, at around 10% or less, with the exception of ethnicity, which showed more variation within groups.
The overall proportion of missing data for each sociodemographic variable did not significantly vary between groups (table 1), with the exception of the data on advanced qualifications. Such data were missing for 26% of applicants to medical schools using UKCAT in borderline cases, 29% using UKCAT as a factor, and 28% using UKCAT as a threshold (P<0.001 in all cases).
At the time of application, candidates were aware at least of basic admissions policies in relation to the UKCAT and also knew their own test scores. We therefore anticipated that applications to universities with more robust use of the UKCAT would, on average, be made by candidates with higher test scores. Table 2 shows the average performances of candidates applying to each of the three groups of medical schools and advanced level qualifications. Again, there were modest absolute but statistically significant differences between the performances of candidates associated with applications to the three types of institution. In general, candidates applying to medical schools that used UKCAT as a threshold tended to achieve higher UKCAT scores compared with candidates applying to the other two groups of medical schools. Candidates applying to medical schools using UKCAT as a factor tended to go on to obtain higher grades at advanced qualifications, and this was the case whether only the “best of three” A level (or equivalent) grades were counted or whether a total UCAS tariff score was calculated from grades of all examinations taken (with the exception of resits and general studies).
Univariate analysis: applications
For the statistical modelling we utilised observations with complete data only on all variables (fig 1). Tables 3 and 4 depict, respectively, the results of multilevel logistic univariate regressions for the probability of conditional and unconditional offers. Even after controlling for candidate level effects, the unadjusted odds of receiving a conditional offer varied significantly between all three groups of medical schools for most widening participation categories; age and sex being the only exceptions (table 3). The intraclass correlation in this case represents the residual of the latent response variable of each individual candidate; each candidate is conceptualised as having a certain level of a latent trait (an unobserved normally distributed variable). The level of this trait would be manifest in the likelihood of the candidate receiving an offer, thus this trait could be conceptualised as “offerability.” In a multiple logistic regression, the intraclass correlation is the correlation within each individual candidate, once fixed effects are controlled for, such as academic attainment. Thus intraclass correlations of 0.3 to 0.5 (table 3) would be consistent with moderate individual level effects for candidates (as opposed to the effect of group membership of the medical schools applied to). As the odds ratios are raw, such individual level effects would include UKCAT and predicted or actual A level (or equivalent) performance. On average the intraclass correlations were highest for medical schools using UKCAT as a threshold and lowest for those using UKCAT in borderline cases, with all between group differences being significant at the P<0.05 level according to a Wilcoxon test (table 3). This would suggest that individual candidate effects and performance are more predictive of the provision of a conditional offer in those institutions with a stronger use of the UKCAT. A similar pattern was observed for the raw odds of obtaining an unconditional offer, with the association between applicant age and sex weaker than for the other variables. No significant association was observed with applicant age once only unconditional offers were considered. The intraclass correlations were then smaller in magnitude, reflecting weaker candidate level effects. The intraclass correlations for medical schools using UKCAT as a factor were, on average, significantly larger than for the other medical school groups (P<0.001), reflecting the increased predictive power of individual compared with group or unmodelled effects. A plausible reason for this may be a greater emphasis, for example, on A level performance when providing an offer, as some medical schools may insist on the original conditions of an offer being met, whereas other may allow a certain latitude, especially if the number of applicants accepting offers falls short of expectation.
Table 3 Odds of an application resulting in a conditional offer of a place according to widening participation sociodemographic group and type of medical school’s use of UK clinical aptitude test (UKCAT)
Table 4 Unadjusted (raw) odds of an application resulting in an unconditional offer of a place according to widening participation sociodemographic group and use of UKCAT by medical school
Multilevel multiple logistic regression: applications
Tables 5 and 6 show, respectively, the results for the separate multilevel multiple logistic regressions for the probability of conditional and unconditional offers. The results in table 5 depict three varying models for the prediction of a conditional offer. For those applications relating to medical schools using UKCAT as borderline, six independent and statistically significant predictor variables existed, including sex, socioeconomic status, ethnicity, type of school attended, and academic attainment, with academic achievement being most strongly predictive of an offer. The UKCAT score was a relatively weak independent predictor in applications relating to this group of medical schools in that the odds of receiving a conditional offer were only increased by roughly 20% for every standard deviation of standardised score above the mean that was scored (odds ratio 1.23, 95% confidence interval 1.11 to 1.37). The model for applications to medical schools that used UKCAT in borderline cases had three statistically significant interaction terms; age and academic attainment, sex and school type attended, and UKCAT score and English as a second language. The first of these indicates that for candidates aged more than 20 at the time of application, A level (or equivalent) attainment conferred relatively less advantage when a conditional offer was made compared with younger applicants. This is highly plausible given that many older applicants will be applying on the basis of a university degree rather than on A level attainment. The interaction term between male sex and school type attended implies that the relative disadvantage of having attended a state school was reduced for men compared with women. Lastly, the interaction between UKCAT score and English as a second language suggests that an increased UKCAT score conferred a greater advantage for those for whom English was a second language compared with native English speakers. Medical schools using the UKCAT score as a factor in the admissions process had five significant and independent predictors of the probability of a conditional offer. In contrast with the medical schools using UKCAT score in borderline cases, the strongest of these was standardised UKCAT score, with an increase in 1 SD above the mean more than doubling the odds of an offer, all other factors being equal (odds ratio 2.31, 95% confidence interval 2.07 to 2.58). As in the model with medical schools that used the UKCAT score in borderline cases, interactions were significant between academic attainment and age and UKCAT score and English as a second language. In addition the interaction term for academic attainment and UKCAT score was significant, indicating some degree of synergy between A level (or equivalent) grades and UKCAT score when predicting the probability of a conditional offer. For medical schools with UKCAT used as a threshold score in the admissions process the only significant independent predictors were UKCAT score (odds ratio 8.59, 95% confidence interval 6.96 to 10.62) and, to a lesser extent, academic attainment (1.63, 1.46 to 1.82). As with the other two categories of university the interaction between UKCAT score and English as second language was significant. Two other interaction terms were also significant. Firstly, the interaction between sex and UKCAT score, suggesting that for males an increase in UKCAT score conferred less of an advantage than for females when seeking a conditional place offer. This is interesting given that, on average, men outperform women on the UKCAT.16
Secondly, the interaction between socioeconomic status and English as a second language was significant and suggested that those from a lower socioeconomic background who were not native English speakers were relatively disadvantaged compared with candidates who did not belong to this subgroup.
Table 5 Results of multilevel multiple logistic regression analyses with outcome variable a conditional offer of a place at medical school
Table 6 Results of multilevel multiple logistic regression analyses with outcome variable an unconditional offer of a place at medical school
The results in table 6 depict the three models for the prediction of an unconditional offer, according to category of medical school. The results were broadly similar to those in table 5, according to the probability of conditional place offers. Medical school that used UKCAT scores in borderline cases and as a factor had six independent and statistically significant principal predictor variables: sex, socioeconomic status, ethnicity, type of school attended, UKCAT score, and academic attainment. For medical schools using UKCAT as a threshold, the only sociodemographic variables that were independent and significant predictors of an offer were UKCAT score and academic attainment. In addition there was a trend of borderline statistical significance for older candidates to more likely receive an unconditional offer than those under 20 at application (odds ratio 1.53, 95% confidence interval 0.99 to 2.38, P=0.06). This observation is unsurprising as many older candidates have already obtained the required qualifications for entry (although specific graduate entry courses were excluded from the analysis). Each model also included between two and four statistically significant interaction terms. In addition to the interaction terms that were significant in the results outlined in table 5 the model for medical schools using UKCAT as a factor relating to the probability of an unconditional model also showed a significant interaction between ethnicity and English as a second language (2.10, 1.09 to 4.05, P=0.03). This implied that those who reported their ethnicity as non-white and were not native English speakers were relatively less disadvantaged compared with white non-native English speakers when obtaining an unconditional offer. This interaction term seemed to be the consequence of a small number of white non-native English speaking candidates, relatively few of whom obtained unconditional offers.
Tables 7 and 8 present the results of the combined multilevel multiple logistic regression. The models contained interaction terms concerning medical school group (entered as a factor variable), where the baseline category was varied. Therefore it was necessary to depict the results for the logistic regression models where medical schools using UKCAT as a threshold (table 7) and using it as a factor (table 8) had been used as the base categories separately. Consequently the odds ratios and P values for all statistically significant combinations of interaction terms are presented as pairwise comparisons (for example, borderline group versus threshold group by age interaction) rather than overall interaction effects. This seemed more appropriate given that UKCAT usage was a factor variable with more than two levels but not ordinal (ordered categorical) in nature—that is, failing to fulfil the “proportional odds” assumption. The odds of an applicant receiving a conditional offer did not significantly vary between the different categories of universities once the effects of other predictors and interaction terms were controlled for (tables 7 and 8). However, this was not true for unconditional offers, where it was less likely that a candidate applying to a medical school that used UKCAT as a threshold received an unconditional offer compared with the other two categories of medical school, implying that, on average, relatively fewer applicants to the medical schools using UKCAT as a threshold were satisfying the academic conditions set as part of their conditional offers. The interaction terms involving the group of medical school applied to largely mirrored the findings from the separate models (tables 5 and 6); the interaction terms highlighted intergroup differences in relation to UKCAT score, academic attainment, ethnicity, age, school type attended, and socioeconomic status. Interaction terms involving group and English as second language were not statistically significant at the P<0.05 level for either conditional or unconditional offers. Likewise, significant interactions between sex and type of university applied to were not observed, although an interaction between an application to a medical school that used UKCAT scores in borderline cases (versus those using threshold scores) and sex was of marginal statistical significance (odds ratio 0.83, 95% confidence interval 0.68 to 1.01, P=0.06). This indicated a trend towards male applicants being slightly less likely to receive an offer from a medical school that used the UKCAT in borderline cases as opposed to in a threshold manner. Not all the intergroup differences were present or in the direction originally hypothesised (tales 7 and 8). For example, interaction terms suggested that an applicant from a state school (in this case excluding grammar schools) was less likely to receive a conditional offer from a medical school that used UKCAT scores in borderline cases compared with one using threshold scores (0.74, 0.60 to 0.90, P=0.004) but were more likely to receive such an offer than from a medical school that used UKCAT as a factor (1.47, 1.21 to 1.80, P<0.001). Nevertheless this finding was consistent with the results from the separate models for the prediction of conditional offer provision (table 5) in that medical schools using UKCAT as a factor in the admissions process showed a relatively high degree of disadvantage for state school students (0.52, 0.45 to 0.60, P<0.001) even after controlling for the effects of other, potentially confounding, variables and interactions. Medical schools that used UKCAT as a threshold were also less likely to provide a conditional offer to an older applicant compared with the other two categories of medical school.
Table 7 Results of multilevel multiple logistic regression analyses with outcome variable as conditional or unconditional offer of a place to study medicine. Values are odds ratios (95% confidence intervals) unless stated otherwise
Table 8 Results of multilevel multiple logistic regression analyses with outcome variable conditional or unconditional offer of a place at medical school. Values are odds ratios (95% confidence intervals) unless stated otherwise
Descriptive statistics and univariate analysis: medical school entrants
In total, data related to 4456 medical school entrants to the 22 universities analysed. Table 9 and figure 2 depict the breakdown of the proportion of each medical school intake by widening participation status. Some categories of institutions appeared fairly homogenous in relation to the proportion of candidates of widening participation status taking up places, whereas others were relatively diverse. For example, medical schools using UKCAT as a threshold consistently admitted more than around 55% of their medical students from state schools, whereas more variation was observed in medical schools using UKCAT as a factor in the admissions process (fig 2).
Table 9 Sociodemographic and educational characteristics of medical school entrants according to type of medical school (use of UK clinical aptitude test (UKCAT)) to which they were admitted. Values are mean (SD) percentage of entrants from widening (more ...)
Fig 2Percentage of medical school entrants who were male, reported non-white ethnicity, reported speaking English as a second language, were aged over 21 years at application, had not attended an independent or grammar school, reported a non-professional (more ...)
The proportion of missing data did not vary significantly between groups (table 9), with the exception of advanced qualifications; this variable was missing for 26% of entrants to medical schools that used UKCAT scores in borderline cases, 29% for those that used UKCAT as a factor in the admissions process, and 28% of those that used UKCAT as a threshold, with the difference between factor and threshold groups and the borderline group being significant (P<0.01 in all cases).
Table 10 illustrates the relative performance of entrants on the UKCAT and at A level examinations (or equivalent), the latter in terms of UCAS tariff. In table 10 the UCAS tariff is presented in two ways; as absolute points from all advanced level examinations (excepting resits and general studies), and as a standardised z score calculated on the basis of the “best of three” (or equivalent number) of advanced level grades. Those entrants to medical schools that used the UKCAT as a threshold had significantly higher total scores, on average, than those entrants to medical schools that used UKCAT as a factor, who, in turn, had on average higher scores than entrants to medical schools that used the UKCAT in borderline cases (P<0.001 in all cases). Both “uncensored” (no maximum set) and censored (for example, “best of three”) standardised UCAS tariff scores were significantly higher (P<0.05 and P<0.001, respectively) in entrants to medial schools that used UKCAT as a factor compared with the other two categories of medical school.
Table 10 Average performance of applicants on UK clinical aptitude test (UKCAT) and advanced educational qualifications (according to total Universities and Colleges Admissions Service (UCAS) tariff score) by group of medical schools applied to according (more ...)
Multilevel multiple logistic regression: medical school entrants
To ensure nesting, only observations from 2679 entrants with complete data were utilised, out of the total of 4456 individuals (60%). Table 11 shows the results for sex, ethnicity, English as second language, and age. The manner in which UKCAT was used was entered as a three level factor variable. Unless stated otherwise, the group of medical schools that used UKCAT as a factor was used as the baseline category. This simplified the presentation of the results because although all the combinations of interactions were explored, those that were significant (P<0.05) mainly involved this group of medical schools as one of the comparators. Several of the effects observed in the raw univariate single level analysis (table 9) were attenuated or not observed after controlling for individual institutional effects and potential confounding. For example, entrants to medical schools that used UKCAT as a factor or as a threshold continued to be more likely to be male than those admitted to medical schools that used UKCAT in borderline cases. In contrast, the trend for entrants to medical schools that used UKCAT as a factor or threshold to be over 20 years at application was no longer apparent after controlling for individual institutional and the effect other sociodemographic variables. The intraclass correlation (0.22) was larger than for the other widening participation categories depicted in table 11. This would suggest a relatively large institutional level effect compared, for example, with that for the association with an entrant’s sex (intraclass correlation=0.002). This is supported by the descriptive statistics observed in table 9 and figure 2 where the relatively large variation between institutions in the same UKCAT usage category for intake of older students could be contrasted with the relative homogeneity within groups for the proportion of male entrants. These findings would suggest that the trend for mature entrants reflects one or more individual universities taking in either large or small proportions of mature students, rather than an effect associated with type of UKCAT usage (table 9). However, not all the trends observed were attenuated after controlling for institutional effects and potential confounding variables; the difference between the odds of an entrant to a medical school using UKCAT in borderline cases compared with one from those using UKCAT as a factor having English as a second language actually increased in significance (P value from 0.2 to 0.04) once this more complex modelling was performed.
Table 11 Results of multilevel multiple logistic regression analyses for prediction of sex, ethnicity, English as second language status, and age (>21 years at application) of entrants to 22 participating medical schools with complete data available (more ...)
Table 12 depicts the results of multilevel multiple logistic regression for the prediction of widening participation status of entrants for school type attended, socioeconomic status, and achieving below average academic attainment (for this cohort of medical school applicants). Several trends that were highly statistically significant on estimation of the raw odds ratio were now absent or of only borderline significance. Notably, the intergroup differences in the probability of an entrant being from a state school (as opposed to independent or grammar school) attenuated to a difference of only borderline significance (odds ratio 1.60, 95% confidence interval 0.97 to 2.62, P=0.06) for medical schools using UKCAT as a threshold compared with those using UKCAT in borderline cases. In contrast, the faint trends for entrants to medical schools using UKCAT as a factor and as a threshold to be more likely to be from low socioeconomic backgrounds were now of borderline significance (P=0.05 and P=0.06, respectively). The relatively high proportions of entrants to medical schools using UKCAT as a factor with below average academic attainment compared with the other categories of medical schools remained statistically significant, even after adjustment for institutional and potential confounding effects.
Table 12 Results of multilevel multiple logistic regression analyses for prediction of school type attended, socioeconomic status, and relatively low academic attainment (less than AAB or equivalent at A level) of entrants to 22 participating medical (more ...)
Modelling the prediction of achieving conditions for medical school entry
435 total applicants, 1189 were initially given 1507 unconditional offers, having already achieved the required academic conditions at the time of application. A further 5368 candidates were initially provided with 7363 conditional offers (an average of 1.4 offers per successful candidate). Of these individuals, 4470 (83%) eventually obtained an unconditional offer, having obtained sufficient academic qualifications to satisfy the relevant institutions that entry was merited. In most, but not all cases, this would involve achieving the initial A level (or equivalent) grades set as part of the conditions of the provisional offer. To explore the predictors of an application being converted from a conditional to an unconditional offer data were analysed from 3290 applications relating to 2803 candidates holding conditional offers where complete data were available. A multilevel multiple logistic regression model was built up in a stepwise fashion. A random intercept for each candidate was introduced, although it was not possible to model the random effects associated with individual institutions simultaneously. All possible combinations of interactions were explored and those found to be significant and independent predictors of successfully converting a conditional offer to an unconditional one were included in the final model (table 13). The strongest predictor of success was academic attainment (odds ratio 10.35, 95% confidence interval 8.01 to 13.27, P<0.001), in these cases measured using a standardised UCAS tariff. Indeed, the only other significant and independent predictor was the application being associated with a medical school using UKCAT in borderline cases or a factor in the admissions process (compared with the remaining group). However, a trend of borderline statistical significance was observed for conditional offers associated with candidates of low socioeconomic status to have less probability of being converted to unconditional ones (0.42, 0.18 to 0.99, P=0.05). Two interactions were also significant. Firstly, an interaction between mature applicant status and academic achievement was observed (0.11, 0.07 to 0.17, P<0.001). This is interpreted as increased A level achievement being relatively less of an advantage to those applying to holding conditional offers and over 20 years old, compared with younger applicants. In practice many mature candidates will be applying on the basis of a university degree, either achieved or pending, rather than A level achievement, and therefore this finding is readily understood in this context. Perhaps a less anticipated interaction was that observed for low socioeconomic status and non-white ethnicity (8.56, 1.81 to 40.43, P=0.007). This implied that those applicants holding conditional offers from low socioeconomic backgrounds were more likely to convert them to unconditional ones if they were of non-white than of white ethnicity.
Table 13 Results of multilevel multiple logistic regression analysis for the prediction of achievement of grades or degree required for a conditional offer to be converted to an unconditional offer. Data from 3290 applications relating to 2803 candidates (more ...)
Missing data analysis
Those applications with no data on socioeconomic background of the candidate (n=12
562) were significantly less likely to result in an offer (0.81, 0.76 to 0.85, P<0.001). Candidates with missing socioeconomic status (n=4605) were significantly more likely (P<00.01) to be male (1.13, 1.05 to 1.21), have attended a private or grammar school (1.13, 1.05 to 1.21), be over 21 years at application (1.33, 1.23 to 1.44), be of non-white ethnicity (1.73, 1.62 to 1.86), speak English as a second language (1.39, 1.28 to 1.51), and have below average academic attainment (1.27, 1.17 to 1.39) and UKCAT score when compared with those with socioeconomic background reported.
Likewise, those applications with no attainment data on A levels, higher, or Irish leaving certificate (n=11
949) were significantly less likely to result in an offer (0.58, 0.54 to 0.61, P<0.001). Candidates with missing or non-standard advanced qualification data (n=4917) were significantly more likely (P<0.01) to have missing data on socioeconomic status (1.26, 0.17 to 1.36), be female (1.10, 1.03 to 1.18), have attended a state school (3.65, 3.36 to 3.96), be over 21 years at application (45.28, 40.61 to 50.49), be of non-white ethnicity (1.42, 1.32 to 1.52), speak English as a second language (2.15, 1.99 to 2.33), and have a below average UKCAT score when compared with those with socioeconomic background reported (1.86, 1.74 to 1.99). Thus, according to missing data theory, the missing socioeconomic status and advanced qualification data seemed to be not missing completely at random—that is, the value of the missing variable and the probability that it is missing is unrelated to observed variables.
The results of the sensitivity analyses using multiple imputed data generally indicated that the missing values for advance qualification as socioeconomic status were missing at random. On average the odds ratio recovered from the analyses of the imputed data only varied by around 5-10% compared with those obtained for the non-imputed data. The only exception to this was the findings for the prediction of conditional offers using imputed advanced qualification values. These results differed by a slightly larger degree, with, on average, odds ratio differing by an average of 12%. A fuller report of these results is available from the lead author’s website (www.dur.ac.uk/p.a.tiffin/fps