|Home | About | Journals | Submit | Contact Us | Français|
GP appraisal is currently considered inadequate because it lacks robustness. Objective assessment of appraisal evidence is needed to enable judgements on professional performance to be made.
To determine GP appraisers' views of the acceptability, feasibility, and educational impact of external peer feedback received on three core appraisal activities undertaken as part of this study.
Independent peer review and cross-sectional postal questionnaire study.
One of three core appraisal activities (criterion audit, significant event analysis, or video of consultations) was undertaken by GP appraisers and subjected to peer review by trained colleagues. A follow-up postal questionnaire elicited participants' views on the potential acceptability, feasibility, and educational impact of this approach.
Of 164 appraisers, 80 agreed to participate; 67/80 (84%) submitted one of three appraisal materials for peer review and returned completed questionnaires. For significant event analyses (n = 44), most responders believed the peer feedback method was feasible (100%) and fair (92.5%) and would add value to appraisal (95.5%). Peer feedback on criterion audits (n = 15) was believed to be acceptable and fair (93.3%) and it was thought it would be a useful educational tool (100%). Completing a consultation video (n = 8) was perceived to be feasible as part of normal general practice (n = 5). It was unanimously agreed that assessment of videos by peers has educational impact and would help improve appraisal.
This group of GP appraisers strongly supported the role of external and independent feedback by trained peers as one approach to strengthening the existing appraisal process.
Appraisal for GPs in NHS Scotland was introduced in 2003.1 This ‘non-judgemental’ process helps GPs to reflect on their working practices with a peer — a trained GP appraiser colleague. One of five core educational activities (Table 1) — each linked to the principles of Good Medical Practice,2 — is discussed in depth annually. Although appraisal has gained acceptance,3–5 the benefits and impact are limited for many GPs.6,7 When introduced, it was envisaged that the combined evidence from five appraisals would inform a proposed system of medical revalidation in the UK.8 The recent white paper makes clear, however, that the existing appraisal model is inadequate for this purpose because it lacks robustness and objectivity.9 Addressing this will require objective assessment of evidence to be included to allow judgements on professional performance.10
In the west of Scotland, a well-established educational model aims to provide GPs with informed, objective, confidential, and independent feedback from trained peers on the quality of three core appraisal activities: videoed consultations, criterion-based audit, and significant event analysis (SEA).11–13 The model is coordinated by NHS Education for Scotland (NES) and has been used on a voluntary basis by GPs as part of regional arrangements for continuing professional development. Although the concept of ‘peer review’ is well established in medicine, the novel aspect of this model is that it appears to be the only one reported in the literature that utilises GP peers who are specifically trained to review and provide developmental feedback on the educational activities outlined. To date, around 40% of west of Scotland GP principals have submitted at least one piece of work for peer review since its inception.
The underlying principles of the model are based on an adaptation of cognitive continuum theory.14 This framework aids understanding of the thinking used in performing a range of tasks. The aim is to improve the quality of reflection on particular tasks. How this is done is described in one of six ‘modes of practice’ ranging from the highly structured scientific experiment (mode 1) to intuitive judgements (mode 6). Peer review sits between modes 4 (system-aided judgement) and 5 (peer-aided judgement) and is designed to minimise the probability of a mode 6 judgement (self-assessment) leading to invalid conclusions on decisions made.
The model described is underpinned by a research base.11–13,15–23 However, further evidence from a range of GP groups as to its overall utility is necessary to inform future development.24,25 GP appraisers are a key group with important professional insights into the strengths and weakness of the appraisal system and how it may be enhanced. Against this background, this study aimed to evaluate GP appraisers' views of the acceptability, feasibility, and educational impact of independent, external peer feedback received on one of three core appraisal activities they undertook as part of this study. In this way, GP appraisers would gain experience of receiving feedback on specific aspects of their own appraisal activities as part of the NES model described. Appraisers could then judge the usefulness of this approach as a support mechanism for the GP appraisal system in NHS Scotland.
All 164 GP appraisers in NHS Scotland were randomly split into three groups, and one of three appraisal activities (SEA, criterion audit, or videotaped consultations) was allocated to each. To maximise fairness for participants and encourage engagement with the study, group numbers were adjusted to account for the effort needed to perform each activity (for example, fewer individuals were allocated to video consultation than SEA because the former is much more onerous). Appraisers were invited to undertake the allocated activity within a 4-month timeframe and submit this to the NES peer feedback model. Informed consent was obtained by providing a full written description and purpose of the evaluation. A standardised re-imbursement fee was offered by the host organisation (NES) to cover back-fill costs for GP participation time.
GP appraisal is considered inadequate and needs to be more robust and objective. The quality of core appraisal activities undertaken by GPs is also known to be variable. Independent feedback by trained peers on these activities is proposed as a potential educational solution. This study provides evidence that GP appraisers who received peer feedback on appraisal activities found it acceptable and educationally useful. Furthermore, they agreed it would ‘add value’ in improving the quality of appraisal evidence.
The model involves the application of formative assessment instruments,11,15,16 by trained GP peers to facilitate developmental feedback for participants. For audit and SEA submissions, two GP peers (from a group of 20) are allocated to independently review submitted reports, with a single reviewer used for videos of consultations (from a group of 20). A GP peer review coordinator then collates this written feedback on performance, provides their own input, and compiles a short report which is sent to participants.
A questionnaire was developed to examine attitudes towards the feasibility, acceptability, and educational impact of external peer feedback on the relevant appraisal activity undertaken. The questionnaire was pre-tested with four GP appraiser colleagues. The survey took place in March 2008, 3 days after peer feedback reports were sent to participants. Non-responders were emailed reminders until all questionnaires were returned. Levels of agreement with attitudinal statements were indicated on a 7-point rating scale.
Data were entered into a Microsoft Excel spreadsheet. The number of rating scores ≥4 was calculated. A Wilcoxon signed-rank test and 95% confidence interval (CI) for estimated median differences were performed to compare before and after rating scores, which were calculated using Minitab (version 15).
A total of 80/164 GP appraisers agreed to participate (49%). Of the 80, 67 (84%) submitted appraisal materials for external peer review. All 67 participants returned completed questionnaires (100%).
Table 2 outlines statistical differences in participants' reported levels of apprehension about submitting SEA (P = 0.52), audit (P = 0.14), and video (P = 0.04) for external peer feedback before and after the study. The differences in appraisers' estimated knowledge and skills levels before and after receiving feedback on SEA (P<0.01), audit (P = 0.06) and video consultations (P = 0.18) are also detailed in Table 2.
A majority of potential participants allocated to the SEA group (91.2%; 44/48) submitted completed reports (Table 3). There was unanimous agreement (n = 44) that submitting one report every 5 years for external peer review would be feasible for most GPs. Responders believed the peer feedback received was fair (n = 42); would facilitate improvement in their SEA practice (n = 32); and would be acceptable to most colleagues (n = 42). The majority agreed (n = 42) that external feedback on SEA would add value to the existing appraisal system.
Fifteen of 18 expected criterion audit reports (83.3%) were received (Table 4). Most responders were unconcerned about their audit report being reviewed by external peers (n = 14) and believed this would also be acceptable to GP colleagues (n = 15). Feedback received provided specific advice on how to improve audit skills (n = 15) and would facilitate improvements in related practices for the majority (n = 12). A majority agreed (n = 14) that the existing appraisal system would be enhanced by external feedback on criterion audits.
A total of 8/14 appraisers (57.1%) submitted videos of consultations for review (Table 5). Five responders believed completing a consultation video was feasible as part of normal general practice. Peer feedback on consulting skills was perceived by all participant appraisers to be fair and educationally useful and to have the potential to have an impact on the consulting skills of most GP colleagues. There was a unanimous perception that peer-reviewed video consultations would add value to the appraisal process.
This study found strong support from participants for the feasibility, acceptability, and educational gain associated with the external and independent review of three core appraisal activities by trained colleagues. Many expressed a certain level of apprehension about receiving external feedback on their appraisal activities, but this decreased slightly after participation, particularly for those submitting a video of consultations. Additionally, there was a general perception that review of core appraisal activities by trained colleagues would ‘add value’ to the current appraisal system in Scotland.
Peer feedback received on SEA and criterion audit reports helped responders to understand more about these techniques, and engendered a reported commitment to practise it more effectively. There was a perceived increase in estimated knowledge and skills levels associated with each activity after peer feedback had been received, particularly for SEA. However, a large minority experienced low-to-moderate education gains from participation. This could be explained by the fact that most GP appraisers should be experienced in SEA and audit. Related evidence and knowledge are also required for many educational and governance reasons,1 including training practice accreditation and to benefit from the Quality and Outcomes Framework.25 Additional knowledge will also be gleaned by appraisers when reflecting on SEA and audit reports with appraisees. It should also be acknowledged that a proportion of GPs will be highly effective in applying both techniques, and that peer feedback, while reinforcing excellent performance in these cases, may offer minimal educational gain. Given this background, there is a strong perception from appraisers that other GP colleagues would benefit from developmental feedback by trained peers on SEA and audit attempts, which reflects previous research evidence.17–20,22
A comparatively small number of appraisers submitted consultation videos for peer review. Although there was strong agreement that other GPs would benefit from this method of receiving feedback on their consultations skills, there was less agreement on whether this would be acceptable to most GPs. Indeed previous work has demonstrated that the process can be effectively undertaken by a GP without previous experience in this area,21 however, peer review of video consultation is often perceived to be much more challenging and stressful than participating in audit or SEA.26 The process can often take longer and be technically difficult to organise and execute, while patient consent is also required. Perhaps most importantly it is also perceived to be more professionally threatening than the other two activities.26 However, there was strong positive agreement on the potential value of external peer review of video consultations educationally and when linked to the GP appraisal system.
A large minority of all Scottish appraisers participated. However, the study population was self-selecting and potentially biased as the views and experiences of non-participating appraisers and other GP groups may differ from those reported. The study context is specific to NHS Scotland, but the underlying principle linking external peer review and appraisal is potentially relevant to other medical specialties and healthcare systems. Most appraisers who agreed to participate did so, but a substantial minority failed to submit video consultations, which points to a potential difficulty in linking this activity with the peer review model as part of normal general practice. Written feedback was provided by peer reviewers. However, in some situations, face-to-face feedback might be more desirable but this also creates a feasibility issue.
Given the increasing importance being accorded to the links between appraisal and revalidation,8–10 more attention will inevitably be paid to the standard of evidence being submitted. Ultimately, someone has to make a judgement as to whether it is good enough. Appraisers have consistently voiced their discomfort at being given this task.6,7 This external peer review system allows the judgements to be made prior to an appraisal with a built-in sampling process of ‘quality control’ ensuring fairness across the country, that is, a national standard. However, the feasibility of a national external peer review system to support appraisal is open to question. For such a system to function, additional resources will be required. One option is to consider the vision that all GPs are potential peer reviewers — with appropriate training — with local ‘experts’ taking on the role of quality assuring the process. If it is to be practicable, much wider discussion is needed. The status quo, however, may not be an option.
The literature on the limited value of self-assessment is clear.27 The benefit, therefore, of providing additional peer review should be not only desirable but — given the stakes — essential. The adaptation of the Hammond model of Cognitive Continuum Theory underpins this justification by encouraging a more rational (rather than merely intuitive) approach to decision making on material submitted for appraisal and inclusion in a revalidation folder.
A call for external verification of evidence will not be straightforward — GP appraisers are in the frontline and, in their opinion in this study, those who took part felt the advantages outweighed the disadvantages by suggesting it gave added value to the appraisal process. In their proposal to improve the appraisal process, the Royal College of General Practitioners suggests that over a 5-year period GPs should present evidence of a specific number of SEA and completed audit cycle reports for discussion during appraisals.28 This represents an increase in the quantity of evidence that is currently required. However, it is unclear how this overcomes the perceived inadequacy of the present approach, which is that the appraisal system lacks a robust method to verify the quality of evidence that is already known to be variable.17–20,22
Future research on this model should focus on shedding light on the acceptability and feasibility of peer reviewing videos of consultations on a large-scale basis. More confirmatory work on strengthening the inter-rater reliability of video consultation peer review is also required. Overall, there is a need to explore with decision makers if and how independent peer review can make a meaningful contribution to improving the robustness of evidence required for appraisal in support of revalidation.
We wish to thank all GP appraisers and GP peer reviewers in NHS Scotland who participated. The following individuals helped to coordinate the study and we extend our grateful thanks to them, also: Marjorie McArthur, Kirsten Macdonald, Dr Ed Russell-Smith and Dr Gordon McLeay. We are also grateful to Dr Lilian Murray, University of Glasgow, for statistical advice.
This study was funded by NHS Education for Scotland
The study was judged by the authors to be service evaluation
The authors' employing organisation is responsible for managing the national GP appraisal scheme and also manages the peer review model outlined in the study. The views expressed reflect those of the authors and not NES