The expectations of the leader of PGME in the clinical departments are high and they vary across stakeholders according to their position in the department [3]. Although the various stakeholders have only limited knowledge of the role of leaders in PGME, they suggest formal leadership education to meet the expectations [3].

Although it has recently been questioned it is still the general opinion that education of leaders results in improvement of leaders' performance [4-7], especially if the content of the leadership courses relates to the organisation the leader works in [5,6,8]. Many authors have suggested the use of a combination of methods to develop leadership skills [4-9]. McKimm used a combination of a leadership course and mentoring in a program for leaders in health and social care education [4]. Mintzberg advocated for a combination of personal development, leadership courses and experience by linking the leadership courses to practice in real or simulated situations [8]. In other organisations than health care it has been suggested to combine leadership courses with multi-source feedback (MSF) or other initiatives supposed to initiate personal development in coherence with development of the organisation [7,9].

We have previously demonstrated that MSF used by consultants responsible for PGME at clinical hospital departments resulted in clear and concrete plans for improvement of management skills, while plans for development of leadership skills and hence personal development were scarce and less concrete. These difficulties in formulating goals and plans for personal development were interpreted by stakeholders as a need for further initiatives to support leadership development [10].

Table ​Table33 shows the MSF mean scores with SD of CREs and respondents from I-group and C-group respectively. Scores are shown for baseline and one year after the intervention. At baseline there were no differences in MSF scores between the I-group and the C-group in the scores from the CREs and respondents respectively. There was no difference in MSF scores for completers and dropouts in the C-group at baseline. The covariance analysis showed no statistical difference between I-group and C-group considering measurements at baseline and one year after (p = 0.149) when respondent type, level of baseline score and change in C-group from baseline to one year after was taken into account.

The use of a convenience sample of course participants may be a limitation in this study. The leadership course was offered to all CREs in the Northern Educational Region. Thus although all CREs in the region had the opportunity to participate in the course we cannot exclude a selection-bias in our sample, who represented those CREs who voluntarily signed up for the course. However, no difference in MSF scores was found between the I-group and C-group at baseline indicating that the groups were comparable.

The high drop-out rate especially in the C-group reflects the difficulties in recruiting busy clinicians to studies of this kind which included a time consuming MSF procedure. Another explanation to the high drop-out rate in the C-group might be that participants in this group were not supported in the same way as participants in the I-group who all passed through a course where they got help to solve various problems in their daily leadership of PGME in the departments. Therefore the motivation to complete the MSF-II procedure might have been much higher in the I-group than in the C-group who only got a phone call asking them to complete the MSF once more. Hence the C-group who completed the second MSF procedure might be the most enthusiastic CREs; they might be the CREs who actively seeks personal development. A selection-bias of this kind could have reduced the differences between the I-group and the C-group. However, at baseline, the MSF scores of participants from the C-group who completed both MSF procedures and scores of dropouts did not differ indicating that the influence of the high dropout might be small.

MSF in itself is an intervention intended to improve leaders' performance and hence the use of MSF as a measuring instrument might have been problematic. However, the results of the C-group do not indicate a major influence of the MSF procedure in itself. This is in accordance with our previous study where the development plans for CREs were not representative for areas needing improvement [10]. At baseline, both the scores of CREs and respondents are quite high and a ceiling effect cannot be ruled out.

Pooling the scores from respondents (the head of the department, the other consultants and the trainees in the department) into one score might have blurred the results, since it is well known that there are different perspectives on leaders' performance according to the position you hold in the organization (chief, peer, subordinate), and since the perception of the concept "good leader" varies among stakeholders [6,11]. According to a previous study, stakeholders' knowledge of the job as CRE is scarce and the expectations to the CRE as leader of the medical education in the department differs according to stakeholders' position in the department [3]. However, it has been shown that these different expectations did not result in differences in the various stakeholders' scoring in a MSF process [10]. We therefore feel confident in pooling the results from various stakeholders into one score of respondents for each CRE.

The second MSF was performed one year after the CREs participated in the leadership course. It is generally agreed that repeated MSF processes would eventually improve leaders' performance [11-14]. However, some describe that a time-span of one year is too short to detect any improvement in MSF scores [12,13]. It might therefore have been interesting to see if improvement in leadership skills of the CRE could be detected if the MSF procedure was repeated after 2-3 years, since developing as a leader is a time-consuming process and initiating changes in a highly bureaucratic organization is a slow process [6]. However, expanding the study period to more than one year might also have been problematic and might have resulted in even higher drop-out rates than was actually found.

The participants in our study came from both university and non-university hospitals and represented a variety of specialties. This supports the generalisability of the results within the domain of medical education. However, as both the instrument and the leadership course were tailored to CREs it is questionable whether results can be translated into health care leaders in general or to other domains of organisations.

Despite the limitations of the study we feel confident in concluding that the impact of leadership courses on performance in actual practice must be disputed. Previous studies have mainly described a positive effect of courses on leaders' knowledge and performance based on self-reported data [15,16]. The well-known gap between knowing and doing might be reflected in our study where the stakeholders' ratings do not indicate any improvement in leader performance, while at the same time participants in the study reported to have learned from the course. Some of the factors influencing this knowing-doing gap include physician barriers (peer influence and inertia), organizational barriers, and support/resource barriers [17,18]. In a previous study it was described how the CRE is considered to be in a weak position regarding influence and power, and how difficult it is for the CRE to take on a leadership role due to various environmental factors like peer influence, his place in the hierarchy and the general inertia existing for developmental initiatives in hospitals. [3]. When evaluating course effects factors such as support and follow-up from superiors, openness to change and new knowledge in the organization, stability and resources in the organization, and the possibility to practice what has been learned should be considered [15,18,19]. In addition when studying the effect of MSF procedures environmental factors like feedback orientation, organisational cynicism, how MSF fits into other developmental initiatives and stability in the organisation must be considered [11,13].

LM: MD, PhD and associate professor in postgraduate medical education is consultant at the Department of Internal Medicine and program director at the Regional Hospital, Viborg, Denmark.

TB is consultant in human resources at Department for Human Resource, The North Denmark Region, Aalborg, Denmark.

AS: MD, PhD is professor of medical education and scientific director of the Institute for Education, Faculty Health, Medicine and Life Sciences, Maastricht University, Netherlands.