|Home | About | Journals | Submit | Contact Us | Français|
“Although there is over a century's literature on selection methodologies, rarely has any process provoked such fierce debate as MTAS. Many facts surrounding specialty selection, and our role within it, are not well understood. This is illustrated by Dr Nachev's personal view (BMJ 2007;335:615 doi: 10.1136/bmj.39342.515961.59), which contains two fundamental factual inaccuracies. The problems surrounding selection methodology are multi-faceted and multi-dimensional, well beyond disregarding CVs, relying on ‘white space' questions and poor IT delivery.”
For the distress caused, I share the anger of the profession. The inquiry led by Sir John Tooke details the facts about our role in the process (www.mmcinquiry.org.uk). Here, I highlight critical issues to encourage close scrutiny of facts surrounding principles, methodology, and context underlying MTAS, so that there is learning for the future.
Clearly, in the past, selection practices in medicine have been effective. Although few would deny there is scope for improvement, the CV and interview process has generally worked well. So why change? MTAS was devised alongside the MMC (Modernising Medical Careers) programme introduced by the Department of Health, where the fundamental principles underpinning gateways to progression were changed overnight. Consequently this changed the rules governing selection. Traditionally, medicine has relied on robust indicators of attainment on the CV, such as work experience and Royal College examinations for selection decisions. The MMC pathway relied on the belief that doctors could be selected to specialties without previous experience in that specialty. The selection methodology in MTAS was designed for ST1 (the first level of specialty training), not for thousands of doctors already working in specialties. Under MMC principles and in compliance with Postgraduate Medical Education and Training Board (PMETB) (www.pmetb.org.uk/index.php?id=456), we were advised that work experience and exams could not be scored or used to rank applicants. The introduction of run-through ST1 posts was new.
I have worked on selection methodology in medicine for over 12 years. This work informed the development of selection centres and the introduction of new shortlisting tests in general practice. In 2007, all deaneries worked together through a GP national office, where thousands of doctors are successfully appointed using this process. Since 2002, in partnership with doctors we have developed selection methodology for many secondary care specialties and for graduate entry into medical school. Although commonality exists across all specialties and levels, selection criteria for each are distinct, with evidence supporting different priorities between specialties.
Having completed this work, in 2004, I was invited to meet the MMC team to advise on selection methodology into specialty training. I recommended developing a national test for shortlisting (supported by early evidence from general practice) and validated selection centres with full involvement of the royal colleges and with large scale consultation. Following this meeting, I received no further correspondence from the MMC team and no pilots were put in place. In May 2006 we won an open competition tender organised by the Department of Health. Our work included advising on selection methodology for foundation programmes and the general practice selection process. For specialty selection, the scope of work states; “The number of applicants expected to apply for entry into Specialty Training is approximately 6,000 and that applications will be via a single electronic national portal entry system (separate project) the working assumption for the closing date will be 5th January 2007.” At the outset we were asked to advise on selection methodology for ST1. We were not asked to deliver selection methodology for doctors in “transition” via ST2, ST3, ST4, and FTSTA (fixed term specialist training appointment) posts, nor academic posts. We believed these arrangements would be delivered via local processes.
The rules and conditions governing MTAS were defined by MMC, based on PMETB principles, and, via the Conference of Postgraduate Medical Deans steering group, they represented all stakeholders. Given the time scale (less than 16 weeks) there was no option but to use materials from existing application forms used (over several years) for entry into specialist training. By contrast, in collaboration with the general practice national office, my team designed the shortlisting test with general practitioners, which has shown to work well. For the future, the general practice model has been identified as best practice.
However, this model cannot be transferred into all specialties. Medicine is a broad discipline and secondary care is significantly different from general practice, requiring bespoke selection methodologies. Different selection ratios for specialties and for locations add complexity; one size cannot fit all. Some believe selection practices in other professions can be readily transferred, but medicine in the UK is unique. Few of those deciding policy understand what a clinician does on a daily basis. An important challenge is to translate the needs of the profession to policy makers.
In the past 12 years, advising on selection methodology in all sectors, I have learnt more from collaborating with the medical profession than from any other. Unlike selection approaches used by some organisations, I applaud the focus on psychometric scrutiny, the need for validatory evidence, and the demand to treat human beings with respect and dignity in the process. The fact is, MTAS was not designed by psychologists. Without a full understanding of the issues, we cannot hope to navigate the future, which looks yet more challenging.
Having investigated the points raised by Professor Patterson, we have concluded that two statements in “MTAS or a tale of evidence heedless medicine” (BMJ 2007;335:615 doi: 10.1136/bmj.39342.515961.59) were factually inaccurate.
The first was that “The criteria and procedure for selection in MTAS were principally designed by a handful of organisational psychologists engaged through their consulting firm, Work Psychology Partnership.” In fact, the criteria and procedure for selection in MTAS were principally designed by the MMC team, Department of Health personnel, the UK strategy group, and the COPMeD steering group for recruitment and selection (as set out in the Tooke report, Aspiring to Excellence). The second was that “The selection methods they have developed have never been used to select specialist trainees.” In fact, Professor Patterson and her team have researched and implemented selection methods for trainees entering general practice, paediatrics, obstetrics and gynaecology, and surgery. A substantial portion of this work has been published in the BMJ.
We're happy to set the record straight.