|Home | About | Journals | Submit | Contact Us | Français|
Preparation of case reports during student attachments has the attraction of reflecting real life clinical practice, but lacks standardisation when used in summative assessment. This study examined the occurrence and nature of feedback after the introduction of a new system of formative case reports in Third Year clinical attachments. Quantitative and qualitative methods were used to compare the new system to previous practice. Comparison of questionnaire responses demonstrated more and earlier feedback in the New Third Year, which was likely to be delivered at a meeting rather than as written comment. In the New Third Year, the quality of feedback was better and several markers of high quality feedback were rated more highly. There was no difference, however, in students’ confidence in their ability to assess patients. The qualitative data from the New Third Year documented much excellent feedback but also examples of poor practice as well as inconsistency of advice. In conclusion, a relatively simple intervention effected radical changes to feedback practice and attitudes, although it is not known if the clinical skills of students improved.
Preparation of case reports is well established at medical schools and involves skills and behaviours that are recognised attributes of established practitioners. Using case reports in assessment has been questioned on the grounds that grading does not correlate with overall assessment of clinical performance.1 , 2 In recent years, assessment of clinical competence has shifted from summative assessment to formative learning events.3 , 4 Formative assessment has been described as any assessment designed specifically to provide feedback5 and has the potential to create more effective learners. Feedback can become a tool to encourage teaching and learning, and is positively correlated with achievement.6
This study focussed on the introduction of a new case report system in Third Year medicine at Queen’s University Belfast requiring students to complete fewer cases but with greater emphasis on formative assessment and feedback. Unlike previous practice no summative mark was recorded.7 We examined the effect of these changes on the nature and occurrence of feedback as well as the attitudes of students and teachers. Comparison was made with the previous system, which was used mainly to contribute to summative assessment.
The comparison between the New Third Year case report system and the previous system was made using qualitative and quantitative methods involving both students and teachers.
All Fourth Year medical students at Queen’s University Belfast (Third Year during 2012-13, referred to as “Old”) and all current (2013-14, referred to as “New”) Third Year students were sent the questionnaire.
One of the current (2013-14) Third Year groups was chosen as a focus group. Given that groups were randomly selected it was anticipated this would be representative. From a group of 14 students eight agreed to attend.
Semi-structured interview candidates were selected from current active teachers (2 male, 3 female). One had a university appointment and was also a module organiser. Five semi-structured interviews were completed by which point it was concluded that little new material was being obtained.
A brief questionnaire was developed and piloted on a group of third year students undertaking a summer elective in Medical Education. Most questions were closed requiring a response on a Likert scale. Questions examining the nature of feedback were adapted from the principles of feedback outlined by Nicol and Macfarlane-Dick.8 The revised covering letter and questionnaire were loaded onto SurveyMonkey and circulated to current Fourth Year students (Old Third Year) in October 2013. The same questionnaire was sent to all Third Year medical students in March 2014. By this time all had experienced the new system.
The focus group was held in the Royal Victoria Hospital with the principal researcher acting as moderator. The group discussion was recorded discreetly for later transcription.
A guide for semi-structured interviews with teachers was formulated and piloted. Interviews were conducted in the offices of either the researcher or the teacher along the lines suggested by Bryman.9
Statistical comparisons between years were based on a χ2 distribution with one degree of freedom. When cells contained less than 5, adjoining cells were merged to avoid distortion of data. The 5% level of significance for two-sided tests was applied throughout. Percentages quoted in results section illustrate key contributors to any differences.
The open ended questionnaire responses were analysed using the Framework method of thematic analysis as were the focus group and semi-structured interviews.10
The protocol was approved by the Joint Research Ethics Committee of the School of Medicine, Dentistry and Biomedical Sciences, Queen’s University Belfast. Return of the e-mailed questionnaire was taken as implying consent. All focus group and semi-structured interview participants were given information sheets and signed a consent form. Data were collected and stored anonymously as required by University regulations and the Data Protection Act.
Overall response rates were 33.5% in the New Third Year and 27.0% in the Old Third Year.
In deciding (Figure 1) the main purpose of case reports there was a change in the New compared to Old towards “ feedback to improve clinical skills” (New v Old: 20.9 v 9.2%) and away from “ contributing to overall mark” (New v Old: 1.1 v 13.8%).
Asked if the purpose of doing case reports was made clear, there was no difference between New and Old (Table 1). There was also no difference in the ability of students to gain access to patients, nor in the perception of whether patients were representative of clinical practice. There was a highly statistically significant change towards agreement that feedback always occurred in New compared with Old.
The mechanism by which feedback was received moved (Figure 2) in the New Third Year towards group (New v Old: 29.9 v 6.8%), individual (New v Old: 14.9 v 3.4%) or combinations of written and tutor meetings (New v Old: 26.4 v 10.2%) and away from purely written feedback (New v Old: 27.6 v 78.0%). The timing of feedback (Figure 3) also changed towards earlier feedback with much less occurring 4 weeks after the end of attachment (New v Old: 2.3 v 20.3%).
The New Third Year was more likely to consider the quality of feedback was excellent (strongly agree/agree, New v Old: 52.9 v 15.3%). Students were asked about markers of high quality feedback and agreed (Table 2) that feedback encouraged self-reflection and helped to clarify a good performance to a greater extent in the New compared with Old. The New Third Year believed that feedback encouraged dialogue with teachers but not with peers. They considered that feedback helped to close the gap between current and desired performance, and to identify specific actions to improve performance.
When asked (Figure 4) if at the end of clinical attachments students were confident in their ability to assess patients there was no difference in response between New and Old.
Quoted questionnaire responses are identified by student year (New or Old), and respondent (R1, R2 etc). These were considered with the focus group (identified F1 … F8) and semi-structured interviews (identified Interview 1 … 5) under three major themes.
There was consensus amongst staff and students from both New and Old that the most important function of cases was to improve clinical skills:
“ it’s about getting them into the practice of taking a patient history and recording it in an accurate way and being able to set up the basic skills” (Interview 4).
Also seen in a positive light was experience navigating the patient record and hospital information systems. This allowed students to see the pattern and pace of investigation and management:
“Learnt about the structure and format of medical notes and how to extract the relevant information” (New, R3).
A downside was regurgitation of information copied from patient notes:
“made me speak to patients more, but not necessarily examine them as I just copied from the notes” (New, R45).
The potential to enhance patient contact was recognised in both student year groups. There was a perception amongst students and staff that this needed to be encouraged:
“I think it got some students actually seeing patients, especially those who may not have been too keen to be involved” (Old, R4).
An additional purpose was to learn about disease, not just from the student’s own case, but also those of fellow students. A perceived downside to detailed study of one patient was the time taken up with the commentary especially in shorter attachments:
“Cases in shorter placements made me focus on just one patient and not enough in the whole spectrum of illness” (New, R14).
Cases were seen by teachers as a discriminating method of assessment, but there was concern that students put in less effort in the new system:
“ students probably rightly think that they can get away with less work around the case” (Interview 1).
Furthermore there was a view that students needed some degree of summative assessment to give an idea of their ranking within the year even if the mark was not centrally retained.
Students and staff were positive about feedback and believed the absence of summative marks allowed them to focus on the attachment as a whole:
“ it enabled me to focus on the patient more, their history, story and the impact of disease on their quality of life” (New, R33).
“We now actually have to sit down and give the student the 10 minutes” (Interview 2).
Students and staff saw the benefits of developing skills through an iterative process, to which feedback contributed, often with reflection on a good or ideal example:
“they … look at the feedback before they then do the second and third cases because I see the whole purpose of this is to try and make their second case better and the third case even better” (Interview 2).
Students appreciated those teachers who took time to deliver feedback especially where it identified specific failings or recommendations for improvement.
Different formats to deliver feedback were described. Group work with oral presentation allowed common themes to be identified, but there was concern that feedback on written clinical notes was neglected. When group work went well it facilitated dialogue not just with the teacher but also amongst students.
The quality of feedback attracted criticism, especially on the written case commentary:
“Sometimes just written comments which were often brief, illegible or there was no comment left at all ” (Old, R38).
This appeared to contribute significantly to a negative view of that part of the case report system:
“The discussion although interesting doesn’t add anything to improving your clinical examination of the patient” (New, R5).
There was divergence amongst teachers as to whether feedback had improved in the new system. Two teachers, who already appeared to be delivering formative feedback at a high level, did not think the new system changed anything:
“I think the feedback arrangement is much the same. The only thing is I don’t put a mark down anymore” (Interview 3).
Not surprisingly some deficiencies were highlighted. Finding cases caused anxiety amongst students and there were complaints that some patients were unsuitable. Students acknowledged they acquired more confidence approaching patients later in the year.
Teacher time was a limiting factor in providing adequate feedback, which depended heavily on key members of staff. There was a perceived trade-off between giving feedback and other forms of bedside teaching. Providing good feedback depended critically on the assessor knowing the allocated case:
“they should know the case well and then they can give relevant feedback” (Interview 5).
A perception of poor standardisation of marking and inconsistency of advice about how to do cases was a frequent criticism across both year groups:
“one doctor wanted it done one way, another a different way” (New, R47).
Students, as well as teachers, recognised the importance of cases in learning how to assess patients. There were striking comments about the need for students to cross the threshold from tutorial room to ward. There was less support for the commentary, which many found time consuming and burdensome especially within shorter attachments. Teachers expressed concern that students were not spending enough time on the ward, and by implication that more time with patients would increase competence. The limited evidence available does not support the contention that more clinical encounters alone improves clerkship performance.11
Students were pleased that cases no longer counted towards a summative mark, but teachers were concerned that the absence of a summative mark might lead to less effort by students. Perhaps eliminating all summative elements from clinical clerkships should occur only if and when both students and teachers embrace an approach to learning which is performance rather than goal orientated.12 It is worth remembering that there is no reason why good feedback cannot be given after summative assessment,13 nor are there practical reasons not to provide feedback rapidly to large numbers of students.14
Comparison with Mini-CEX15-17 highlights a limitation of the case report system in that inferences were drawn from written cases and presentations and students were not observed performing clinically. There is good evidence that students particularly value direct observation and feedback at the bedside.18 Training in direct observation can increase the comfort of tutors in this type of activity.19
The questionnaire pointed to a major change in the quality of feedback in the New Third Year. There were also dramatic alterations in indicators of high quality feedback, such as encouraging self-reflection, encouraging dialogue with teachers, and identifying specific actions to improve performance.8 The qualitative data highlighted examples of good feedback practice and identified development of feedback seeking behaviour amongst students.20 , 21
On the other hand, given the evidence that students do not always recognise feedback,22 , 23 it is possible some differences were more perceived than real. In other words, the New Third Year students, having been told that there was to be greater emphasis on feedback, were better able to recognise it.
This study provides evidence that the practice of delivering feedback can be altered relatively quickly after a single year induction session. The majority of teachers had not received formal instruction in how to deliver feedback. It would appear wise to continue to develop the attitudes of students and staff to feedback.22 , 23 It may be useful to investigate feedback practice in weaker students who might need it most.24 , 25
An important consideration is whether the structure of clinical attachments supports good feedback practice. Teherani and colleagues studied a longitudinally integrated model and compared it to traditional discipline-specific block clerkships.26 The longitudinally integrated model was rated more favourably, however, formal opportunities to interact with faculty, peers and patients were essential.27
Notwithstanding the progress made, there were difficulties with the case report system. The Old Third Year believed the marking system was inconsistent, and, even in the New Third Year, students found some advice and feedback confusing. It is hard to know in a course that spans many specialties as well as different sites, how uniform processes can be made. Nevertheless, certain shortcomings would be helped by clearer central direction. It was also apparent that “formal” feedback had been introduced at the expense of time otherwise spent on bedside skills, which presumably in previous years included activity not recognised as feedback.28 , 29
An overall limitation of this work is that it included no objective assessment of student performance. We do not know if students in the New Third Year were more or less competent, but there was no difference in the students’ confidence in their ability to assess patients. Many studies demonstrate that instruction in feedback improves reflective practice and approaches to learning.23 , 30 , 31 There is good evidence that the clinical performance of trained physicians is improved by feedback,32 but within clinical clerkships there is little hard evidence that clinical competence is improved.33 A recent study within surgical clerkships showed benefits in knowledge and skill acquisition in those receiving feedback compared to a control arm receiving compliments, even though students found it hard to distinguish feedback from compliments.22
The response rate to the questionnaire was around 30%, and the sample of opinion from focus group and semi-structured interviews was small, but there is no reason to believe the data were unrepresentative. The New Third Year completed the questionnaire two thirds the way through the year, whereas the Old Third Year completed it four months after year end. The Old Third Year responses might be expected to have the advantage of mature reflection but miss important immediate issues. Comparable opposite considerations might have applied to the New Third Year.
The results of this and previous studies allow some conclusions about case reports within clinical clerkships.
This study highlighted the keen awareness of the need to optimise learning in the clinical environment amongst staff and students at Queen’s University Belfast. The change in the New Third Year, placing greater emphasis on formative assessment and feedback, was well received. Although previous research suggests that better feedback should enhance student learning, it is not known if the clinical skills of students in Belfast improved following the changes.
Mr Mike Stevenson, Medical Statistician, advised on questionnaire design and statistical analysis. Mrs Jane Fox, Centre of Medical Education, kindly uploaded the questionnaire onto Survey Monkey, and distributed it to the Third and Fourth Year medical students. Mrs Michelle Whiteside, Medical Secretary, transcribed with great care and precision the interview and focus group tapes. Mr Cieran Ennis, Research Technician, assisted with computer and formatting queries and preparing figures. The data within this paper were presented at the Irish Network of Medical Educators in Limerick in February 2015.