|Home | About | Journals | Submit | Contact Us | Français|
Making health care safer is a key policy priority worldwide. In specialty training, medical educators may unintentionally impact on patient safety e.g. through failures of supervision; providing limited feedback on performance; and letting poorly developed behaviours continue unchecked. Doctors-in-training are also known to be susceptible to medical error. Ensuring that all essential educational issues are addressed during training is problematic given the scale of the tasks to be undertaken. Human error and the reliability of local systems may increase the risk of safety-critical topics being inadequately covered. However adherence to a checklist reminder may improve the reliability of task delivery and maximise harm reduction. We aimed to prioritise the most safety-critical issues to be addressed in the first 12-weeks of specialty training in the general practice environment and validate a related checklist reminder.
We used mixed methods with different groups of GP educators (n=127) and specialty trainees (n=9) in two Scottish regions to prioritise, develop and validate checklist content. Generation and refinement of checklist themes and items were undertaken on an iterative basis using a range of methods including small group work in dedicated workshops; a modified-Delphi process; and telephone interviews. The relevance of potential checklist items was rated using a 4-point scale content validity index to inform final inclusion.
14 themes (e.g. prescribing safely; dealing with medical emergency; implications of poor record keeping; and effective & safe communication) and 47 related items (e.g. how to safety-net face-to-face or over the telephone; knowledge of practice systems for results handling; recognition of harm in children) were judged to be essential safety-critical educational issues to be covered. The mean content validity index ratio was 0.98.
A checklist was developed and validated for educational supervisors to assist in the reliable delivery of safety-critical educational issues in the opening 12-week period of training, and aligned with national curriculum competencies. The tool can also be adapted for use as a self-assessment instrument by trainees to guide patient safety-related learning needs. Dissemination and implementation of the checklist and self-rating scale are proceeding on a national, voluntary basis with plans to evaluate its feasibility and educational impact.
Patients worldwide are unintentionally but avoidably harmed as a result of their interactions with health care . A global campaign to highlight the patient safety problem and recommend potential solutions is being led by the World Health Organisation (WHO) . In the United Kingdom (UK), making patient care safer has been an explicit policy priority in the National Health Service (NHS) for over a decade . Until recently the predominant safety improvement focus was on acute hospital settings where there is well-established evidence of the scale and consequences of adverse health care events . In UK general medical practice comparable research and knowledge of the nature and impact of avoidable harm is growing, but is methodologically limited compared with secondary care [5,6]. However, the emerging evidence from a range of international sources suggests that the safety of general practice may be compromised in a significant minority of cases [7-9]. Specialty trainees are known to be involved in a proportion of these incidents largely because of a range of systems, knowledge, cognitive, training and behaviour based reasons.
In the NHS, a raft of interventions designed to make patient care safer are being implemented or are under development. For example: multi-centre collaborative programmes of safety improvement that aim to reduce harm in specific areas of acute hospital care and, more recently, primary care are currently underway ; a high-profile and long-running campaign to minimize healthcare acquired infections is ongoing ; while a concerted attempt to improve the reporting and learning from patient safety incidents has also taken place .
From an educational perspective, it is now recognised that there is a need for patient safety education to be more explicitly integrated and prominently positioned within existing undergraduate and postgraduate training curricula for all health care professions . The UK Royal College of General Practitioners (RCGP) - which has responsibility for the content of the specialty training curriculum - has responded by developing a curriculum statement on ‘patient safety’  and defining specific learning objectives (Table1). However, it is left to individual GP educational supervisors to determine how the RCGP curriculum is best delivered and the related learning needs of trainees are identified and acted upon during the training period. The role of the postgraduate deaneries in this regard is to verify that the evidence provided by training practices to support the delivery of the patient safety element of the curriculum is of an adequate standard.
Ensuring that all essential educational issues are identified, prioritised and satisfactorily covered during training is not straightforward given the complexity of the tasks to be undertaken and the high volume of topics to be addressed. Within the framework of the RCGP curriculum, GP educational supervisors currently guide the activities undertaken by the trainee using a combination of locally developed induction packs, nationally promoted learning interventions and assessments, and professional experience in the workplace to match across to RCGP curriculum competencies (Table2). However, it is currently unclear how the patient safety-related learning needs of trainees are specifically addressed when in the training environment. Given the lack of standardized guidance on how and what specific learning issues are to be covered, it is possible there is variation in GP educational supervisors’ interpretation and delivery of this safety-critical element of the curriculum at the ‘sharp end’ of frontline educational practice.
Overall there appears to be a paucity of evidence on how postgraduate training in general practice (or lack of) may have a visible and negative impact on the safety of patients – arguably it may be impossible to ever demonstrate clear causation. However, we know that a range of issues connected with the postgraduate training environment may act as proxy indicators of the safety of patient care being compromised unnecessarily and avoidably. For instance, medical educators may be involved in failures of, or inadequate, clinical supervision ; or fail to respond appropriately to trainees’ seeking professional guidance ; or conduct insufficient joint reviews of the management of complex clinical cases ; or provide limited feedback on drug prescribing performance ; and may let poorly developed attitudes and behaviour (such as lack of insight) continue unchecked . Other salient issues that are potentially safety-critical include trainees’ possessing different levels of clinical knowledge  and an inability to prioritise their clinical workloads and manage time . The quality of the learning environment in which trainees are based may also affect the safety of patient care , while doctors-in-training are known to be susceptible to medical errors [6,22].
Given what is known about human error theory in the healthcare workplace , and the marked differences in local safety cultures  and the reliability of practice systems [5-9], it is inevitable that variation in the quality of training provision exists and that some issues will be inadequately covered or even overlooked completely. If this happens with fundamental training topics which are considered to be safety-critical then there is a likelihood that the risk of patients being harmed and other quality of care issues arising could potentially increase.
Evidence is accumulating from healthcare safety and improvement initiatives that routine adherence to the adoption and use of checklist reminders can improve the overall reliability with which care processes and tasks are undertaken and so potentially mitigate future risks [25-28]. Against this background, we aimed to identify and prioritise the most safety-critical issues to be addressed by educational supervisors and trainees in the first 12-weeks of specialty training in the general practice environment – the training period judged to be specifically high risk and beyond which trainees are given greater clinical freedom. In doing this we may help maximize early opportunities to address safety-critical issues proactively via a checklist reminder, which may lead to a reduced risk of patients being unintentionally harmed during and after the training period.
We used a six-stage mixed methods approach to develop and content validate the checklist for educational supervisors (Table3).
As a first step we held three 90-minute workshops in April 2010 with a total of 72 experienced GP educational supervisors (known previously as ‘trainers’) at the south east of Scotland annual GP training conference. They were asked to prioritise their perceptions of the six most critical threats to patient safety in the GP specialty training (GPST) environment (beyond specific clinical topics). In each workshop 4 or 5 groups of supervisors reflected on this task for around 45 minutes, provided feedback on their findings (via a flip chart) and engaged in open discussion about how this information could be of value in the training environment. The workshop facilitator (PB) made contemporaneous field notes and retained all the flip chart data with consent from participants. The authors then independently reviewed each chart and coded the text before generating a preliminary list of safety-critical themes and items (specific educational and training issues) of relevance to early GPST. Differences in theme formulation and item nesting were jointly negotiated until agreement was reached.
We then identified and examined a small number of published reviews and grey literature on the key threats to patient safety in primary care internationally [5-9,29-33]. The information collated was evaluated by the authors and a small number of safety-critical themes and related items of specific relevance to GPST were highlighted. We augmented our aforementioned list with these data to generate an updated preliminary checklist consisting of 15 safety-critical themes and 67 related items.
We invited and recruited a convenience sample of six highly experienced GP specialty training programme directors and course organisers based in the west of Scotland to participate in a modified-Delphi process (three rounds by electronic mail) to gain consensus on the checklist domains and items that were judged to be ‘essential’ to the first 12-weeks of specialty training in the general practice setting. This further iterative refinement resulted in the checklist being reduced to 14 safety-critical themes with 55 related items.
We invited a random sample of 25 West of Scotland regional GP educational supervisors by electronic mail to reflect on the relevance of each of the 55 items to the first 12-weeks of training in the GP environment and then attend and participate in a follow-up 3-hours workshop to further review and endorse the checklist content. A total of 21/25 (84%) completed both tasks leading to further refinement and updating of content themes (n=14) and items (n=47). Discussion with this group also resulted in the proposed final checklist being adapted as a self-assessment tool for specialty trainees.
To further triangulate our findings, we sent the checklist content for review to nine volunteer specialty trainees who had recently completed their initial training period in the GP environment. We held a follow-up 2-hour workshop (n=3) and telephone discussions (n=6) with the trainees who further validated the content. Additional issues were raised by them which were judged by the authors to be relevant to the quality of the learning environment trainees were based in, rather than being deemed essential safety-critical issues in the first 12-weeks of training.
Finally, a further 24 GP educational supervisors from a single geographical district in the west of Scotland undertook a CVI exercise. The relevance of each retained checklist item was assessed by asking the supervisors to rate them using a validated  4-point scale: (1=not relevant, 2=somewhat relevant, 3-quite relevant and 4=very relevant). Twenty out of 24 supervisors were required to endorse each item by assigning a rating of at least 3 out of 4, to establish content validity beyond the 0.05 level of significance. This was determined to provide sufficient evidence for inclusion of each item as part of the final checklist. Supervisors were also asked to identify any missing items that they deemed important for inclusion, but after reflective consideration and open discussions no more items were added, deleted or amended.
The study proposal was pre-screened by the west of Scotland research ethics committee but judged to be a service development not requiring ethical approval.
The study identified 14 safety-critical domains and 47 related items that were judged to be essential components of specialty training in general practice to be addressed by educational supervisors and trainees in the first 12-week period of training in the general practice setting. Each item was also aligned with one or more of the 12 RCGP curriculum competencies to assist and guide supervisors in delivering the curriculum and collecting supporting evidence (Table4). All checklist items were endorsed by a minimum of 20 of 24 GP educational supervisors who rated each item ≥3 on the 4-point scale. The overall content validity index ratio for the tool was 0.98 (Table5).
We identified and prioritised a whole range of safety-critical issues that were judged to be ‘essential’ as part of the high risk 12-week phase of specialty training in the general practice environment. We demonstrated that the content validity of the checklist tool is adequate, providing the first steps towards validating this approach as a method for educational supervisors to guide and support safety-critical education interventions at this phase of training and potentially in future training periods (Additional file 1). At the suggestion of supervisors, the checklist was also adapted to enable trainees to self-assess safety-related learning needs early in their specialty training and monitor future progress (Additional file 2).
At a fundamental level, the domains and related items generated in this study have the potential to be used by GP Educational Supervisors and General Practice Managers to modernize existing Induction Packs, or help inform the development of new induction processes. More specifically, the tool can be applied by supervisors as a checklist prompt or reminder to ensure that the most safety-critical educational tasks are actually carried out and are done so timeously, efficiently, and without ambiguity. The potential to improve the reliability of educational provision in these important areas of specialty training and health care practice should be evident. Completion of the checklist could also be used as supporting evidence to the postgraduate deanery (e.g. during training accreditation visits) that the core patient safety element of the training curriculum is being proactively considered and delivered.
All of these factors are of prime importance because of the potential medico-legal implications for the GP educational supervisor if a trainee is inadequately tutored and supervised and patient safety is subsequently compromised. GP educators need to have mechanisms in place to make an early assessment of a trainee’s competencies and to undertake regular performance reviews and feedback sessions as part of an overall strategy to minimise risks. It should be noted that as trainees’ progress through training they are exposed to a more ‘positive’ and less ‘reductionist’ view of care quality in general practice via engagement with a range of improvement methods and exposure to the Quality & Outcomes Framework as part of routine clinical practice.
Theory suggests that because human memory and attention are imperfect , one way to mitigate errors or lapses in the execution of tasks is through the use of procedural checklists to help us perform more reliably. Compliance checklists to improve the safety or reliability of processes are standard practice in high-reliability industries [35,36], most notably in commercial aviation and petro-chemical organisations. In health care checklist use is well documented, particularly in surgical , obstetric  and intensive care  settings where their implementation is associated with improved clinical outcomes, team working and productivity efficiencies.
However, despite these positive reports there is some scepticism over whether the introduction of a checklist on its own is the single greatest factor attributable to improvement success. Bosk et al.  suggest that the widespread deployment of a checklist may not be successful without a clear understanding that it is a ‘technical’ solution being introduced to a complex dynamic working environment where pre-existing ‘socio-cultural’ issues may need to be contained or resolved. Clinicians may resist or feel threatened by checklist implementation because they perceive it to impinge on their expertise, interfere with their decision-making or over-simplify the working environment. We also know that there is variation in perceptions of local safety climate within and between general practice teams and that some may unintentionally inflate this measure because of inexperience with, or lack of knowledge of, local safety concerns . Similar variation exists in the strength of team working , the maturity of the learning environment  and commitment to quality improvement . Attending to these types of socio-cultural issues is, therefore, as equally important as supporting efforts into checklist development and implementation if momentum is to be gained and a successful impact made.
Adaptation of the checklist as a self-assessment tool for trainees was an unexpected study development. Self-assessment is a method of measuring and interpreting one’s own performance and is a well-established educational intervention amongst all health care professions and as part of specialty training . The use of self-assessment tools by specialty trainees to identify learning needs is routine in medicine [45,46], but does not appear to be well developed in terms of highlighting specific educational interventions related to patient safety. The activity takes on a greater significance when the performance focus is related to the identification of learning need associated with safety-critical education and training. Although the limitations of self-assessmentis well established , it is still promoted as a valuable educational activity because it can be used as a baseline measure which prompts joint discussion and monitoring of performance over time by the trainee and supervisor.
A key strength is that the study idea and subsequent tool development were driven by the identification of a need for a patient safety education intervention by experienced frontline medical educators. We used mixed methods to explore and capture in-depth qualitative feedback and quantitative measures to further confirm validation of tool content which strengthened the rigour of this methodological process. For instance, the final CVI ratio was very high which may be a reflection of the deep consideration of checklist items by a large number of informed educators.
Limitations included the potential for bias through the pragmatic use of convenience samples of volunteer participants. Issues around multi-ethnicity were not raised by our participants but we recognise that in other areas of the UK and beyond, particularly in larger cities, this may be recognised as a potential safety-critical problem. Demographic data on participants was not collected as we judged that this would not necessarily add value to study findings and may hinder recruitment. Our key interest was in tapping into trainers’ knowledge, experiences and perceptions of the safety-critical elements of training.
We only recruited a small number of trainees to validate both the checklist and self-rating scale (which they would use) as it was important to triangulate our findings and get their perspective on safety-critical issues. This was useful in terms of confirming rather than adding to our findings, but exploring these issues with more trainees may have resulted in deeper insights.
Although content validation processes were robust, the tools have yet to be adequately tested and evaluated for evidence of acceptability, feasibility and educational gain. Determining how the successful implementation and wide-spread use of the checklist would directly impact on safety outcomes may not be possible, which means we may need to define proxy measures of success.
The dissemination and implementation of the checklist and rating scale are proceeding on a national, voluntary basis in Scottish GP specialty training practices with plans in place to evaluate their feasibility and educational impact in the short-term. There is clear potential for patients to be unintentionally and avoidably harmed because of what happens (or does not happen) educationally in the GP training environment. The checklist concept is but one small (albeit untested) intervention to help minimise the related risks to patients and the potential medico-legal consequences for supervisor and trainee alike . The use of a checklist measure to ensure the reliable delivery of ‘essential’ patient safety education has potential relevance for all medical specialties and other clinical professions with similar training arrangements in the UK and internationally.
The authors declare that there are no competing financial or non-financial interests.
PB conceived the study idea, acquired funding, led the study design, data collection, analysis and interpretation, and drafted the initial manuscript. JM contributed to study design and data collection, and the content and critical review of the manuscript. MK contributed to study design, and the content and critical review of the manuscript. All authors read and approved the final manuscript.
The pre-publication history for this paper can be accessed here:
GPST Safety Checklist.
GPST Safety Self-Rating Scale.
We offer sincere thanks to all GP educational supervisors, programme directors, course organisers and specialty trainees for their support and contributions to checklist development and validation. We also thank Drs Winnie Weir, Rhona McMillan, Rhoda Abel, Linsey Semple, Ken Lee, Alison Garvie and Murray Lough, and Ms Marion McLeod for their specific and valued contributions to this study.
The study was funded by the authors’ employing organisation, NHS Education for Scotland, which is a special health authority with responsibility for the education, training and life-long development of the healthcare workforce in Scotland.