PMCC PMCC

Search tips
Search criteria

Advanced
Results 1-6 (6)
 

Clipboard (0)
None
Journals
Authors
more »
Year of Publication
Document Types
1.  The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations 
BMC Medical Education  2010;10:40.
Background
Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.
Methods
a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.
Results
The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.
Conclusions
An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use.
doi:10.1186/1472-6920-10-40
PMCID: PMC2893515  PMID: 20525220
2.  The educational background and qualifications of UK medical students from ethnic minorities 
Background
UK medical students and doctors from ethnic minorities underperform in undergraduate and postgraduate examinations. Although it is assumed that white (W) and non-white (NW) students enter medical school with similar qualifications, neither the qualifications of NW students, nor their educational background have been looked at in detail. This study uses two large-scale databases to examine the educational attainment of W and NW students.
Methods
Attainment at GCSE and A level, and selection for medical school in relation to ethnicity, were analysed in two separate databases. The 10th cohort of the Youth Cohort Study provided data on 13,698 students taking GCSEs in 1999 in England and Wales, and their subsequent progression to A level. UCAS provided data for 1,484,650 applicants applying for admission to UK universities and colleges in 2003, 2004 and 2005, of whom 52,557 applied to medical school, and 23,443 were accepted.
Results
NW students achieve lower grades at GCSE overall, although achievement at the highest grades was similar to that of W students. NW students have higher educational aspirations, being more likely to go on to take A levels, especially in science and particularly chemistry, despite relatively lower achievement at GCSE. As a result, NW students perform less well at A level than W students, and hence NW students applying to university also have lower A-level grades than W students, both generally, and for medical school applicants. NW medical school entrants have lower A level grades than W entrants, with an effect size of about -0.10.
Conclusion
The effect size for the difference between white and non-white medical school entrants is about B0.10, which would mean that for a typical medical school examination there might be about 5 NW failures for each 4 W failures. However, this effect can only explain a portion of the overall effect size found in undergraduate and postgraduate examinations of about -0.32.
doi:10.1186/1472-6920-8-21
PMCID: PMC2359745  PMID: 18416818
3.  Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling 
Background
A potential problem of clinical examinations is known as the hawk-dove problem, some examiners being more stringent and requiring a higher performance than other examiners who are more lenient. Although the problem has been known qualitatively for at least a century, we know of no previous statistical estimation of the size of the effect in a large-scale, high-stakes examination. Here we use FACETS to carry out a multi-facet Rasch modelling of the paired judgements made by examiners in the clinical examination (PACES) of MRCP(UK), where identical candidates were assessed in identical situations, allowing calculation of examiner stringency.
Methods
Data were analysed from the first nine diets of PACES, which were taken between June 2001 and March 2004 by 10,145 candidates. Each candidate was assessed by two examiners on each of seven separate tasks. with the candidates assessed by a total of 1,259 examiners, resulting in a total of 142,030 marks. Examiner demographics were described in terms of age, sex, ethnicity, and total number of candidates examined.
Results
FACETS suggested that about 87% of main effect variance was due to candidate differences, 1% due to station differences, and 12% due to differences between examiners in leniency-stringency. Multiple regression suggested that greater examiner stringency was associated with greater examiner experience and being from an ethnic minority. Male and female examiners showed no overall difference in stringency. Examination scores were adjusted for examiner stringency and it was shown that for the present pass mark, the outcome for 95.9% of candidates would be unchanged using adjusted marks, whereas 2.6% of candidates would have passed, even though they had failed on the basis of raw marks, and 1.5% of candidates would have failed, despite passing on the basis of raw marks.
Conclusion
Examiners do differ in their leniency or stringency, and the effect can be estimated using Rasch modelling. The reasons for differences are not clear, but there are some demographic correlates, and the effects appear to be reliable across time. Account can be taken of differences, either by adjusting marks or, perhaps more effectively and more justifiably, by pairing high and low stringency examiners, so that raw marks can be used in the determination of pass and fail.
doi:10.1186/1472-6920-6-42
PMCID: PMC1569374  PMID: 16919156
4.  The attractions of medicine: the generic motivations of medical school applicants in relation to demography, personality and achievement 
Background
The motivational and other factors used by medical students in making their career choices for specific medical specialities have been looked at in a number of studies in the literature. There are however few studies that assess the generic factors which make medicine itself of interest to medical students and to potential medical students. This study describes a novel questionnaire that assesses the interests and attractions of different aspects of medical practice in a varied range of medical scenarios, and relates them to demographic, academic, personality and learning style measures in a large group of individuals considering applying to medical school.
Methods
A questionnaire study was conducted among those attending Medlink, a two-day conference for individuals considering applying to medical school for a career in medicine. The main outcome measure was the Medical Situations Questionnaire, in which individuals ranked the attraction of three different aspects of medical practise in each of nine detailed, realistic medical scenarios in a wide range of medical specialities. As well as requiring clear choices, the questionnaire was also designed so that all of the possible answers were attractive and positive, thereby helping to eliminate social demand characteristics. Factor analysis of the responses found four generic motivational dimensions, which we labelled Indispensability, Helping People, Respect and Science. Background factors assessed included sex, ethnicity, class, medical parents, GCSE academic achievement, the 'Big Five' personality factors, empathy, learning styles, and a social desirability scale.
Results
2867 individuals, broadly representative of applicants to medical schools, completed the questionnaire. The four generic motivational factors correlated with a range of background factors. These correlations were explored by multiple regression, and by path analysis, using LISREL to assess direct and indirect effects upon the factors. Helping People was particularly related to agreeableness; Indispensability to a strategic approach to learning; Respect to a surface approach to learning; and Science to openness to experience. Sex had many indirect influences upon generic motivations. Ethnic origin also had indirect influences via neuroticism and surface learning, and social class only had indirect influences via lower academic achievement. Coming from a medical family had no influence upon generic motivations.
Conclusion
Generic motivations for medicine as a career can be assessed using the Medical Situations Questionnaire, without undue response bias due to demand characteristics. The validity of the motivational factors is suggested by the meaningful and interpretable correlations with background factors such as demographics, personality, and learning styles. Further development of the questionnaire is needed if it is to be used at an individual level, either for counselling or for student selection.
doi:10.1186/1472-6920-6-11
PMCID: PMC1397825  PMID: 16504048
5.  Unhappiness and dissatisfaction in doctors cannot be predicted by selectors from medical school application forms: A prospective, longitudinal study 
Background
Personal statements and referees' reports are widely used on medical school application forms, particularly in the UK, to assess the suitability of candidates for a career in medicine. However there are few studies which assess the validity of such information for predicting unhappiness or dissatisfaction with a career in medicine. Here we combine data from a long-term prospective study of medical student selection and training, with an experimental approach in which a large number of assessors used a paired comparison technique to predict outcome.
Methods
Data from a large-scale prospective study of students applying to UK medical schools in 1990 were used to identify 40 pairs of doctors, matched by sex, for whom personal statements and referees' reports were available, and who in a 2002/3 follow-up study, one pair member was very satisfied and the other very dissatisfied with medicine as a career. In 2005, 96 assessors, who were experienced medical school selectors, doctors, medical students or psychology students, used information from the doctors' original applications to judge which member of each pair of doctors was the happier, more satisfied doctor.
Results
None of the groups of assessors were significantly different from chance expectations in using applicants' personal statements and the referees' reports to predict actual future satisfaction or dissatisfaction, the distribution being similar to binomial expectations. However judgements of pairs of application forms from pairs of doctors showed a non-binomial distribution, indicating consensus among assessors as to which doctor would be the happy doctor (although the consensus was wrong in half the cases). Assessors taking longer to do the task concurred more. Consensus judgements seem mainly to be based on referees' predictions of academic achievement (even though academic achievement is not actually a valid predictor of happiness or satisfaction).
Conclusion
Although widely used in medical student selection to assess motivation, interest and commitment to a medical career, the personal statement and the referee's report cannot validly be used by assessors, including experienced medical school selectors, to identify doctors who will subsequently be dissatisfied with a medical career.
doi:10.1186/1472-6920-5-38
PMCID: PMC1325230  PMID: 16351718
6.  Mapping medical careers: Questionnaire assessment of career preferences in medical school applicants and final-year students 
Background
The medical specialities chosen by doctors for their careers play an important part in the workforce planning of health-care services. However, there is little theoretical understanding of how different medical specialities are perceived or how choices are made, despite there being much work in general on this topic in occupational psychology, which is influenced by Holland's RIASEC (Realistic-Investigative-Artistic-Social-Enterprising-Conventional) typology of careers, and Gottfredson's model of circumscription and compromise. In this study, we use three large-scale cohorts of medical students to produce maps of medical careers.
Methods
Information on between 24 and 28 specialities was collected in three UK cohorts of medical students (1981, 1986 and 1991 entry), in applicants (1981 and 1986 cohorts, N = 1135 and 2032) or entrants (1991 cohort, N = 2973) and in final-year students (N = 330, 376, and 1437). Mapping used Individual Differences Scaling (INDSCAL) on sub-groups broken down by age and sex. The method was validated in a population sample using a full range of careers, and demonstrating that the RIASEC structure could be extracted.
Results
Medical specialities in each cohort, at application and in the final-year, were well represented by a two-dimensional space. The representations showed a close similarity to Holland's RIASEC typology, with the main orthogonal dimensions appearing similar to Prediger's derived orthogonal dimensions of 'Things-People' and 'Data-Ideas'.
Conclusions
There are close parallels between Holland's general typology of careers, and the structure we have found in medical careers. Medical specialities typical of Holland's six RIASEC categories are Surgery (Realistic), Hospital Medicine (Investigative), Psychiatry (Artistic), Public Health (Social), Administrative Medicine (Enterprising), and Laboratory Medicine (Conventional). The homology between medical careers and RIASEC may mean that the map can be used as the basis for understanding career choice, and for providing career counselling.
doi:10.1186/1472-6920-4-18
PMCID: PMC524180  PMID: 15461786

Results 1-6 (6)