This paper reports the first two academic years’ progress of the 146 students who, in September 2007, commenced year 1 of the five-year medical course at Hull York Medical School (HYMS). All were invited to participate in the study, which had ethics approval from HYMS’ Medical Education Ethics Committee (study ref 0701). Data were collected from the following measures. Figure shows when each set of measurements was undertaken.
Schedule of testing from pre-application to end of academic year two. Timing of cognitive and non-cognitive qualities tests, tutor assessments and end of year examinations.
1. The UKCAT test, which all students were required to have taken in 2006 before applying to medical school, provided four cognitive skills subtest scores: verbal reasoning (VR), numerical reasoning (NR), abstract reasoning (AR)
and decision analysis (DA)
]. HYMS did not preselect students on the basis of any minimum or specified range of UKCAT performance. UKCAT results were available for 131 students; all but two of the remaining students had applied a year earlier, before the UKCAT was introduced.
2. The non-cognitive personal qualities assessments were three paper-based tests delivered under examination conditions at the University of Hull and the University of York in October 2007. The tests, which have also been part of the UKCAT since 2007, were:
a) The Interpersonal Traits Questionnaire (ITQ), which measures narcissism, aloofness, confidence
(in dealing with people) and empathy
and produces a summary score for INVOLVEMENT
(versus detachment) in which confidence
are positive, narcissism
b) The Interpersonal Values Questionnaire (IVQ), which measures the extent to which the respondent favours individual freedoms (
versus societal rules)
as a basis for making moral decisions [9
c) The Self-Appraisal Inventory (SAI) [11
], which measures the domains of (mental) RESILIENCE
(comprising scales measuring anxiety, moodiness, neuroticism
and irrational thinking
) and SELF-CONTROL (
versus risk taking tendency) using the scales of restraint
and anti-social tendencies
. SAI also contains a Lie scale.
3. Tutor assessment (TA) data are collected routinely about all students, from each problem-based learning (PBL) tutor. Groups of 8 students meet with the same tutor twice a week for a 1.5
h problem-based learning (PBL) tutorial throughout year 1, and again (in different groups with different tutors) throughout year 2. These tutors, all clinicians, are also the personal mentor for their students, and consider each individual’s assessment data in formative one-to-one interviews between tutor and student. These data comprised:
a. Assessment of a set of specified behavioural items, adapted from the Peer assessment of professional behaviours form
created by Gary Butler, University of Wollongong, and used with permission. The items were suggested by the requirements for medical school curricula [3
], taking into consideration the ability of tutors to observe student behaviour. The assessments were undertaken by their PBL tutor for each student once in year 1 (in May 2008) and twice in year 2 (January and May 2009). The assessment form was being developed over this period; it was piloted in May 2008, revised in January 2009 and finalised in May 2009, so there was slight variation in the items on each occasion, with the tutors assessing 17 behaviours in May 2008 and May 2009, but only 14 behaviours in January 2009. See Additional file 1
for an example of each form. The final three items (18 to 20) on the May 2009 form are excluded from the analysis because these items relate to a different context (clinical placements) and were assessed by other tutors whom the students met irregularly.
b. In addition, as part of this research, the PBL tutors were asked to make an overall assessment of each student in May each year, by rating them as either ‘problematic’, ‘average’ or ‘particularly promising’.
4. The marks from end of academic year summative examinations, held in June 2008 and June 2009, which are allocated to one of three HYMS themes: Theme A (Life sciences
and Clinical sciences
), Theme B (Clinical techniques and skills
and Person-centred care
) and Theme C (Evidence-based decision-making, Population health and medicine
and Managing resources for quality and efficiency
). Theme A tests largely knowledge recall, with some interpretation. Theme B tests interpersonal understanding and communication and practical skills by a written paper (30% of marks) and an OSCE (70%). The OSCE included equal numbers of five minute stations covering practical skills and communication skills; for a detailed list of the stations, see Additional file 2
. Theme C tests not only knowledge, but also analytic and numerical evaluation skills.
Tutor assessments and examination data were available for the whole cohort of 146 students. 143 agreed to take part in the non-cognitive study but only 137 completed all three non-cognitive tests, of whom 122 had also completed the UKCAT cognitive tests (see Additional file 3
The data were entered into SPSS for analysis and screened for abnormal distributions. As this was an exploratory rather than a hypothesis driven study, structured methods such as regression analysis and adjustment of significance levels for repeated comparisons were eschewed in favour of describing basic relationships between variables, and the effects of collinearity were not taken into account. The boundary provided by statistical significance was taken as a guide to which relationships are reported as important.
Pearson product–moment correlations were computed within and between the groups of measures given above, and significant results (p
.05, 2-tailed test) tabulated. Although only four rating categories (Unsatisfactory, Borderline, Satisfactory and Excellent) were available to tutors, these scales were assumed to approximate interval measurements and to be suitable for parametric statistical tests. Correlations involving variables with some missing data were automatically adjusted on a case-by-case basis. To assess relationships with high and low medical school performance, scores on non-cognitive tests, tutor assessment (TA) items, overall tutor rating and OSCEs were compared for students obtaining the top and bottom 20% of the summative examination results, using one-way Analysis of Variance to produce an F value.