The results of this study show that the modified assessment form and procedure resulted in frequent reporting of feedback, less frequent reporting of reflection and only rare reporting of action plans. The results also show, however, that, generally, the reflections, feedback and action plans that were provided were specific. Based on the importance of specific comments as indicator of quality [1
], we can conclude that, if
comments were made, the modified assessment form elicited useful qualitative comments. It would be interesting, however, to further investigate the different frequencies of the different types of comments. Perhaps the modification, consisting only of encouragement and facilitation of written narrative reflections, feedback and an action plan, was not sufficiently powerful to induce trainers and trainees to make full use of the form. The way the assessment form was introduced and the availability of the online manual appear to have been inadequate to achieve reflective behaviour for all trainees and formulation of an action plan on a significant proportion of forms. Although the layout of the form directs which type of comment should be provided in which space, users retain the possibility to use it otherwise. It is also possible that feedback from the trainer and – to a lesser degree – reflection by the trainee are more firmly embedded in the assessment routine than linking these comments to the broader learning context by formulating an action plan. Perhaps, among trainers and trainees there already was a culture of giving feedback and, to a lesser extent, of reflecting on performance, but not (yet) of making plans for action to follow-up on feedback. This conclusion appears to be supported also by the large differences we found between trainee-trainer pairs. It seems that some pairs do have
a culture of feedback, reflection and action plans, while for others such a culture remains to be developed. Apparently, some trainers and trainees do apply the information from assessment training and the online manual. These findings suggest that in order to enhance the effectiveness of assessment training, there should be a special focus on reflection and action plans. Additionally, trainers and trainees who use all the feedback modalities might be asked to share their experiences.
An important quality of our study is the response rate. A large number of assessment forms was analysed and almost 80% of GP trainees in the sample handed in their forms. However, the number of completed forms per trainee differed widely, with some trainees handing in only one form, even though the minimum required for the study period was six. The overrepresentation of forms from first year trainees may be attributable to the introduction of the new assessment form. Since the version of the form that was the subject of this study was first introduced in March 2008, first year trainees had used the form from their first day of training, while for the third year trainees (who had started their training before 2008) it meant a change. However, since the percentages of written comments in response to the seven questions did not differ between these groups, the overrepresentation of first year residents apparently did not impact on the results.
It should be noted, that although the reported action plans were specific, this finding is based on limited data, because the majority of forms did not contain an action plan. Only a few trainee-trainer pairs provided comments relating to an action plan. Next, we only studied written narrative comments entered in the assessment form. This is a limitation because we do not know what actually happened during the discussion between trainee and trainer when the text was formulated, and therefore a comparison with the results of Holmboe et al. cannot be made [16
]. We chose our method because written narrative texts are one of the positive qualities of these formative assessment forms. Forms can be stored by trainees in their portfolio to help them reflect on a series of mini-CEX results in order to formulate learning goals, and they can help trainees and trainers to gain an overall impression of development of performance. Another limitation is the focus on the qualitative part of the assessment form. Further research should examine relationships between narrative feedback and the quantitative part of the assessment form.
In this study we looked at the written results of workplace-based observation and feedback. In a qualitative study [18
] we examined ideas, barriers and motives experienced by trainers and trainees in relation to observation, reflection and feedback. Based on the results of the present study we would recommend a different approach to training to stimulate reflection in trainees and more attention to the formulation of an action plan, elements that are important for the effectiveness of feedback [12
]. Further research is needed to explore how feedback and reflections that are specific as well as goal-oriented, as evidenced by the formulation of an action plan, impact on performance improvement, the ultimate aim of assessment of observed performance. The implications of the substantial differences between trainee-trainer pairs in relation to the percentages of specific comments require further investigation as well.