|Home | About | Journals | Submit | Contact Us | Français|
Objectives To develop a robust valid and exportable appraisal and assessment process for doctors in training which is portfolio based and works at all hospitals within the deanery. It is called the personal development portfolio.
Design For every senior house officer, there was a recorded meeting with his or her supervisor, at the beginning, midterm and at the end of the post. An outside assessor witnessed the exit assessment meeting.
Setting The Wirral Hospital, a District General Hospital with 72 senior house officers in 10 different specialties was used as a pilot site to develop the process. Then the process was exported and implemented at the other 12 trusts of the deanery.
Main outcome measures Records were kept of the induction, midterm and exit assessment meetings. A record was kept of the number of senior house officers succeeding and failing at their exit assessments. Also, the number promoted to the specialist registrar grade was recorded.
Results The process was performed every 6 months on 11 occasions between 2000 and 2005. It involved 72 senior house officers in 10 different specialties. On each occasion, participation usually exceeded 70%: 623 were appraised and assessed and 609 of them (97.8%) had satisfactory exit assessments. For 14 doctors (2.2%), the process identified a cause for concern, which was usually accepted by the doctor and sometimes allowed remedial action to be taken. Twenty-six (4.2%) were promoted to the specialist registrar grade in this period. The process also identified the strengths and weaknesses of the senior house officer posts in the 10 different specialties that had such posts, and was used to encourage good medical teaching practice in them. Over 4 years, we exported the process to all the other 12 Trusts in the Mersey Deanery. Once established, the process was easy to use for both trainees and trainers, although it was time consuming.
Conclusions It was possible to develop and implement a portfolio based appraisal and assessment process, which was accepted by senior house officers and their trainers in all specialties at all hospitals within the deanery. Now that the senior house officer grade has been superceded by the Foundation and the training grade years, the principles of the personal development portfolio are being used to appraise and assess doctors in these grades too.
The Mersey Deanery has developed an appraisal and assessment process for doctors in training which is portfolio based and which works in all the hospitals within the deanery. It is called the ‘personal development portfolio’. We describe how the process was set up and the steps taken to ensure that it is fair, valid, useful and user friendly.
The personal development portfolio was developed for the senior house officer grade. However, the principles of the personal development portfolio are now being used for doctors in the Foundation Year 2 grade, and will be used in due course for the run through training grade also.
The government document Modernising Medical Careers addressed the unique features of the senior house officer grade, and the constraints under which they worked. It recognized the need to reform the grade.1-3
In August 2007, the implementation of Modernizing Medical Careers will create a run through training grade to assimilate the senior house officers and the specialist registrar grades. The document recognizes that regular appraisals and assessments are essential components of the run-through training scheme, to ensure that the doctor is making satisfactory progress at every stage.2 Regular appraisal and assessment are required of all doctors working in the National Health Service, and the guidelines for this process laid down in the General Medical Council document Good Medical Practice.4
Mersey Deanery began a system of regular appraisal and assessment for all senior house officers in the deanery in 1990. The personal development portfolio system evolved from 15 years of practical experience of appraisal and assessment. It has incorporated the General Medical Council guidelines of appraisal and assessment.
Now that the senior house officer grade is being phased out, the basic principles of this portfolio system have been adapted for use with the F2 grade, and will also be used with doctors in the run through training grade when this commences in August 2007.
Table 1 shows how the process works. When the trainee doctors start at each post, there is a hospital induction day, where the purposes of appraisal and assessment are explained to them. The doctors are instructed how to obtain the portfolio documents required, by downloading them from this website [http://www.merseydeanery.ac.uk/SHO/appraisal and assessment].
The process implements successive cycles of appraisal and assessment for the trainee, involving the active participation of both the trainee and the trainers at every post that the trainee works in. At the start of each post, there is an induction meeting when the supervisor and trainee learn about each other [Table 1 (1)]. At the mid-term appraisal they meet again to document a review of progress, making adjustments to learning targets where appropriate [Table 1 (2)]. At the end of the post, the summative assessment is made, in the presence of the external assessor [Table 1 (3)].
Every stage of the doctors' training is witnessed and documented, and as the trainee doctors ascend through the grade, they accumulate a growing personal development portfolio of their previous and current experience, to go with them in their next training post.5
The record of in-training assessment is awarded after the summative assessment meeting [Table 1 (RITA)]. It is witnessed and validated by the external assessor and is based on the summative assessment of the senior house officer's overall performance in that post. The trainee receives a RITA ‘C’, ‘D’ or ‘E’. This terminology is the same as used for the assessment of specialist registrars.7,8
If the performance has been satisfactory, a RITA ‘C’ is awarded. If there has been a problem, it is identified and documented. If it is relatively minor and correctable, a RITA ‘D’ is given and the trainee and the supervisor at the next post are informed of the need to correct the problem.
If the problem is more major, the trainee receives a RITA ‘E’, and is referred to the postgraduate dean, who determines what remedial action needs to be taken, and organizes it. In this way the deanery is aware of the small number of doctors with problems. For all the other doctors in training, the completed documented evidence of induction, appraisal and assessment are evidence that the proper educational guidance has been given.
As a separate part of the process, the external assessor also examines whether the senior house officer post is suitable for training doctors. The external assessor interviews both the supervisor and the trainee separately, using the 10 parameters in the post assessment form. These include assessment of hotel services in the post, the provision of supervised training, evidence of support for the trainee, in-service lecture programmes, availability of study leave and evidence of appraisal and assessment. The post needs to be suitable for teaching and training.6
The departmental secretaries make a copy of the exit assessment documents, with copies for the doctor, the doctor's supervisor, the external assessor and the clinical tutor at the postgraduate centre in the hospital. The clinical tutor sends copies of all assessments and the post assessments to the deanery. There were 700 senior house officers in our deanery, in 13 hospital trusts.
The attendance was rarely less than 70%. For the early exercises, the attendance rate for the induction appraisals was unsatisfactory; but it improved from 2003 onwards, perhaps with familiarity with the process. From that time, many senior house officers also had the midterm appraisal (Table 2).
In 609 of the 623 (97.8%) assessments, the senior house officer was judged to have passed the assessment. Fourteen (2.2%) were awarded a RITA D or E, and 26 (4.2%) were promoted from that post to specialist registrar grade (Table 2).
A problem was identified in 14 doctors (2.2%) which could be described using the documentation. This judgement was almost always accepted by the senior house officer (Table 3).
Four doctors showed signs of stress at work, which they nearly always admitted to. However, the process failed to detect any abnormality in one before the individual committed suicide.
In two cases, doctors from overseas were having difficulties in communication. Another four were failing to undertake audit projects set by their consultants, in contrast with their more successful colleagues, in highly competitive specialties such as orthopaedics.
Except in pathology and general practice, the rate of promotion from senior house officer to specialist registrar grade was very low. During the period 2000-2005, focused and talented career obstetric senior house officers started getting promoted, after 3 to 4 years in post. Overall, only 26 doctors (4.2%) were promoted from their current post to the specialist registrar grade (Table 4).
Table 5 shows how the process has spread through the deanery, from one hospital in January 2000, to all 13 hospital trusts in the deanery from January 2003 onwards.
Bayley and Higgins introduced the appraisal and assessment process for senior house officers in the Mersey Deanery in 1990. By 1999, 70% were having one review, and 30% were having two. However, the process was not obligatory, and unsatisfactory senior house officers could evade detection.9 It was suspected that some did this deliberately. At the same time, it appeared that some hospital departments, although happy to have deanery senior house officers rotated to them, were reluctant to have their training methods subjected to scrutiny.
The aim of the first formal pilot in January 2000 was to make the process obligatory at one hospital in the Mersey Deanery. Each senior house officer was to be seen at the end of the post by their educational supervisor, in the presence of an external assessor. Their progress would be reviewed on the lines of a review form. From that one hospital, the process would be spread from hospital to hospital within the deanery.
In the 11 exercises at the Wirral Hospital since January 2000, the overall attendance for the exit assessments was rarely less than 70% (Table 3). At first, when setting up the process, there was reluctance from some departments and some senior house officers to participate. Both employed various excuses to avoid taking part. However, once attendance got up to 70%, it became, in effect, obligatory because there was peer pressure both for all the departments to organize the process, and for all the senior house officers to attend.
For the small percentage who did not attend, the reasons for this could be explored and the candidate could be assessed later. On most occasions, there were legitimate reasons, such as annual or study leave. However a few failing candidates did find the exit assessment very threatening. One turned up inebriated, another failed to attend twice, and one was threatening towards the assessors. Clearly such behaviour is not compatible with the General Medical Council's concept of a good doctor, and needed to be detected and acted on.
There have to be three meetings. In the case of a failing trainee, the doctor could complain legitimately that, if there have been no obligatory induction and midterm appraisals, with a caution if necessary, they could not be judged fairly at the end.
The process did not examine competencies. It checked that assessments on competency were or were not made. Modernising Medical Careers has identified that a doctor in training needs to acquire various competencies, which then need to be assessed. However there is a considerable problem in doing this. Modular training and the European Working Time Directive have reduced the senior house officers' exposure to experience, as it did for the specialist registrars.10 Many colleges are trying to address the acquisition and assessment of competency using log books. The process works in tandem with the specialty assessments and assesses whether the senior house officer also acquired the generic competencies required of a good doctor and demonstrated good medical practice as required by the General Medical Council guidelines.4
Career guidance was given as part of the process. The candidates' career aspirations could be explored during the interviews. Their aspirations needed to be to be realistic. The harsh facts of Table 4 show the fierce competition for career progression in many hospital specialties, and provided a useful guide for both trainer and trainee.
Career guidance may become more difficult after the implementation the F2 training posts which will lead on to the run through training grades. Doctors develop at different rates, and not all are ready to make a career choice so early in their careers.
For 14 doctors (2.2% of 623 assessments) the process identified cause for concern (Table 3). The causes for concern were made plain by the supervisor in the presence of the external assessor. In most cases, the complaint was judged to be correctable (RITA ‘D’); but referral to the postgraduate dean for targeted training was used occasionally (RITA ‘E’). In either case, the next supervisor has been informed of the issue so that this may receive attention.
In a large survey of assessments for specialist registrars in the Oxford Deanery, the incidence of RITA ‘D’ and ‘E’ assessments was 3%, which is very similar to our figure.7
It can be very difficult to improve the manual skills in a doctor in a craft specialty if they do not have those skills innately. For any change to take place, the senior house officer must understand the need for change, and to be capable of making it.
Alternatively, the need to change specialties may be recognized and accepted. In almost all the 14 cases the judgement was accepted, if reluctantly. None contested the verdict legally.
The problem doctors are problems, both for themselves, and for their employers. Has their assessment been fair? Can they or will they correct their faults? Can they be employed safely? Are their career aspirations realistic? We feel that the process makes a fair assessment, but it cannot always correct the fault in the trainee.
When a RITA ‘D’ or ‘E’ is given, although it is not intended to be punitive, the trainee may feel threatened. A RITA ‘D’ or ‘E’ will have an adverse effect on career prospects of that trainee in a competitive specialty.
However, 97.8% of trainees received a satisfactory RITA ‘C’ and, for them, the process was productive and not threatening.
The external assessors are usually consultants in the same specialty from a neighbouring hospital—often the college tutor of that specialty. They are essential to maintain fair play and transparency between the participants. Their presence helps prevent the occasional maverick consultant passing an unfair judgement on the trainee, and helps prevent the manipulative trainee from avoiding a fair assessment. The assessor acts as a witness to the exit assessment, listening to the discussion between the senior house officer and the consultant, and, when occasionally necessary, helping bad news to be imparted. Finally, they are a witness to validate the award of the RITA grade.
The external assessors may also be able to detect the strengths and weaknesses of the posts by seeing the consultant supervisor and senior house officer separately, and noting the scoring of the post according to the parameters of the post assessment form.
Candidates are often very unwilling to make a complaint about their posts lest they may be seen as trouble makers. However, the factual nature of the questions of the post assessment form may allow weaknesses to be unearthed and explored by the external assessor with the consultant supervisor. At the Wirral Hospital, the personal development portfolio process detected unsatisfactory catering provisions at night, and these have since been improved.
These questions about the training posts have great significance now that the working time directive mandates the implementation of limited junior doctors' hours. This has reduced the exposure of the trainee to training, and has tended to separate the trainee from the trainer, so that opportunities for supervision and experiential training may be compromised.10
The final role of the external assessor has been to organize the same process in their own hospitals, with the college tutors from this hospital reciprocating as their external assessors. The portfolio process for senior house officers has spread from hospital to hospital within our deanery, so that after four years, 13 out of 13 hospital trusts in the deanery perform the exercise (Table 5).
The process is led from the deanery and coordinated in the hospitals by the postgraduate education department manager and the clinical tutor. It is organized within the specialties by the specialty college tutor, with the help of their secretaries. Often the exit assessments are on the hospital audit day, the process being an audit of trainee doctor performance. The external assessors for each specialty claims a half day of professional leave from their hospital. Most seem to enjoy the visit, and the college tutor from this hospital reciprocates as the external assessor for the other hospital. (In this exercise, we learned how very useful medical secretaries are in implementing the process.)
Mersey Deanery implemented the F2 grade in August 2004. Many of the doctors were former Mersey Deanery preregistration house officers (PRHOs), who were used to a PRHO portfolio process already. The personal development portfolio process has been modified and implemented for the F2 grade. When in post, the F2 doctor has to achieve core competencies. They have to undertake clinical evaluation exercises, direct observation of procedural skill, case-based discussion and peer assessments. In all these exercises, they are seen, appraised and assessed by several members of their team, for example their consultant, other consultants, other more senior doctors in training, and the ward sisters, in interviews which are documented. These exercises are a de facto 360 degree appraisal in which it becomes clear to the doctor in training and several of their more senior colleagues as well as the supervisor what are the strengths and weaknesses of the trainee and whether they are making satisfactory progress or not.
We also plan to use the principles behind the personal development portfolio process for the run through training grade when these posts are implemented in 2007.
Finally, an unexpected effect of the portfolio development process has been to augment a formal dress code of senior house officers and their supervisors. Both parties nearly always attend the interviews smartly attired (Figure 1).
Acknowledgments We are grateful to Dr S J Harper, for appraising and assessing the script, and for formatting the tables, and to Mrs Karen Burrows for formatting the script.
Competing interests None declared.