In this study the effect of a leadership course for consultants responsible for education in clinical departments following a MSF procedure was compared to MSF alone especially regarding development of leadership skills over time. Surprisingly, the study did not show the expected effect of a combination of a leadership course and MSF compared to MSF alone. In the following we first discuss various validity threats to our results including selection biases, problems of study design and instrumentation. Next we discuss various explanations of the un-expected results.
The use of a convenience sample of course participants may be a limitation in this study. The leadership course was offered to all CREs in the Northern Educational Region. Thus although all CREs in the region had the opportunity to participate in the course we cannot exclude a selection-bias in our sample, who represented those CREs who voluntarily signed up for the course. However, no difference in MSF scores was found between the I-group and C-group at baseline indicating that the groups were comparable.
The high drop-out rate especially in the C-group reflects the difficulties in recruiting busy clinicians to studies of this kind which included a time consuming MSF procedure. Another explanation to the high drop-out rate in the C-group might be that participants in this group were not supported in the same way as participants in the I-group who all passed through a course where they got help to solve various problems in their daily leadership of PGME in the departments. Therefore the motivation to complete the MSF-II procedure might have been much higher in the I-group than in the C-group who only got a phone call asking them to complete the MSF once more. Hence the C-group who completed the second MSF procedure might be the most enthusiastic CREs; they might be the CREs who actively seeks personal development. A selection-bias of this kind could have reduced the differences between the I-group and the C-group. However, at baseline, the MSF scores of participants from the C-group who completed both MSF procedures and scores of dropouts did not differ indicating that the influence of the high dropout might be small.
MSF in itself is an intervention intended to improve leaders' performance and hence the use of MSF as a measuring instrument might have been problematic. However, the results of the C-group do not indicate a major influence of the MSF procedure in itself. This is in accordance with our previous study where the development plans for CREs were not representative for areas needing improvement [10
]. At baseline, both the scores of CREs and respondents are quite high and a ceiling effect cannot be ruled out.
Pooling the scores from respondents (the head of the department, the other consultants and the trainees in the department) into one score might have blurred the results, since it is well known that there are different perspectives on leaders' performance according to the position you hold in the organization (chief, peer, subordinate), and since the perception of the concept "good leader" varies among stakeholders [6
]. According to a previous study, stakeholders' knowledge of the job as CRE is scarce and the expectations to the CRE as leader of the medical education in the department differs according to stakeholders' position in the department [3
]. However, it has been shown that these different expectations did not result in differences in the various stakeholders' scoring in a MSF process [10
]. We therefore feel confident in pooling the results from various stakeholders into one score of respondents for each CRE.
The second MSF was performed one year after the CREs participated in the leadership course. It is generally agreed that repeated MSF processes would eventually improve leaders' performance [11
]. However, some describe that a time-span of one year is too short to detect any improvement in MSF scores [12
]. It might therefore have been interesting to see if improvement in leadership skills of the CRE could be detected if the MSF procedure was repeated after 2-3 years, since developing as a leader is a time-consuming process and initiating changes in a highly bureaucratic organization is a slow process [6
]. However, expanding the study period to more than one year might also have been problematic and might have resulted in even higher drop-out rates than was actually found.
The participants in our study came from both university and non-university hospitals and represented a variety of specialties. This supports the generalisability of the results within the domain of medical education. However, as both the instrument and the leadership course were tailored to CREs it is questionable whether results can be translated into health care leaders in general or to other domains of organisations.
Despite the limitations of the study we feel confident in concluding that the impact of leadership courses on performance in actual practice must be disputed. Previous studies have mainly described a positive effect of courses on leaders' knowledge and performance based on self-reported data [15
]. The well-known gap between knowing and doing might be reflected in our study where the stakeholders' ratings do not indicate any improvement in leader performance, while at the same time participants in the study reported to have learned from the course. Some of the factors influencing this knowing-doing gap include physician barriers (peer influence and inertia), organizational barriers, and support/resource barriers [17
]. In a previous study it was described how the CRE is considered to be in a weak position regarding influence and power, and how difficult it is for the CRE to take on a leadership role due to various environmental factors like peer influence, his place in the hierarchy and the general inertia existing for developmental initiatives in hospitals. [3
]. When evaluating course effects factors such as support and follow-up from superiors, openness to change and new knowledge in the organization, stability and resources in the organization, and the possibility to practice what has been learned should be considered [15
]. In addition when studying the effect of MSF procedures environmental factors like feedback orientation, organisational cynicism, how MSF fits into other developmental initiatives and stability in the organisation must be considered [11
Further investigations are needed to explore the degree to which the clinical department supports the leaders of PGME and to investigate the relations between the culture in the department and the opportunity for CREs to display leadership skills.