Exchanging assessment experiences and exam tasks created at other faculties form an important basis for improving medical education. The use of a web-based platform to support assessment significantly eases such cooperation. Both the MAA and the IMS assessment platform are in line with the Consensus Statement and Recommendations from the 2010 Ottawa conference
which states that “the medical education community still needs to develop a deeper understanding of how technology can underpin and extend assessment practices” [18
years it has been possible to establish a successful and stable inter-faculty cooperation among 28 faculties. Despite differences in the human and technical resources of the partner faculties, MAA has a high acceptance which, amongst other things, is clearly demonstrated by the speed with which it has grown to include 28 participating faculties and the intensive use of the system. The accession of the medical faculty of Geneva as a French-speaking university and of the faculties in the Netherlands as Dutch-speaking universities is a further signal that the IMS not only offers extra value due to its extensive collection of questions (which for the most part available in German only) but is a complete system with a high degree of functionality and the workflow assistance. This differentiates the IMS from other known assessment associations such as the IDEAL Consortium, which focuses mainly on building a database of exam questions [4
] and UMAP, which involves a commitment to the quality assurance of tasks. A faculty conducts quality assurance reviews as a contribution and in return receives access to a certain number of reviewed tasks. However, a department can only view the descriptors of each question prior to using it, not the question itself [5
]. Other international networks are either not focused on assessment (FAIMER [19
] or do not support the entire assessment process using IT [21
One of the advantages of the IMS is that authors can decide whether they want to share items with others. Some people choose not to publish their items because they only want them to be used locally. The full benefit of the system will be realized as authors begin to see the quality and efficiency inherent in cross faculty collaboration.
The analysis of user experience shows that the further development of the IMS promotes quality development processes at participating faculties. For example, the review activity of IMS participants increased significantly during the past year following changes to the process which included e.g. more flexible ways to perform a review of an item or a better presentation of the review results. To further expand review activity, cooperation in specialised assessment associations (e.g. paediatrics, occupational health) is consistently promoted and supported by the MAA (e.g., advice, presentations at professional conferences, and help with networking).
A technical challenge is the development of powerful search algorithms. At present, more than 17.000 questions are managed in the shared pool and the number is rising steadily. Using additional filters which take statistical test parameters, review results, usage frequencies of the tasks, and MeSH terminology into account [22
], there are plans to increase the efficiency of the search functions in future to further improve access.
Currently, in spite of the variety of question formats (long menu, free text, MCQs, key feature cases, OSCE, etc.) available in the IMS, the main type used are MCQs (96%). This mirrors results which showed that MCQs are still the predominant format at German medical faculties [2
]. In this regard, the IMS offers the possibility of supporting further implementation of urgently needed competency-based assessment formats and content due to its support features for practical and oral assessments.
The development of competency-based assessment formats and the creation of the relevant content are increasingly seen as a major challenge internationally. Qualification frameworks like the competencies developed by the Accreditation Council for Graduate Medical Education (ACGME) [23
] or the National Competency-based Learning Targets Catalogue in Medicine (NKLM) for Germany [13
] which are in line with the CanMEDS roles [24
] will contribute to giving further impetus to competency-based assessment. As these competency-based Catalogues are applied nationwide, the individual learning targets can be deposited in the IMS and linked directly with the exam papers. Additionally, the Alliance regularly submits joint applications for joint research projects, for example focussing on the question of how to integrate other test formats which are now on the horizon. To catalyse these changes, the German Federal Ministry of Education and Research has funded a five-year project starting 2012 to develop competency-based examination formats and generate content.
In the past, assessments at individual faculties have focused on summative examinations. It is crucial for steering the students’ learning process to receive feedback on their current knowledge through formative tests. For example, the progress test is a format, growing in popularity, demonstrates growth in knowledge over time [25
]. As the creation and evaluation of progress tests is very resource intensive for a single faculty, the MAA has already done several steps to realize this project as a joint endeavour together with the faculty of Maastricht (Netherlands).
The concept of having many partners who contribute to improving quality by providing their special knowledge is one of the main reasons for the success of MAA. Through the consensus meetings and the direct exchange between doctors, psychologists, and IT developers, the system enhances the quality development processes at the faculties (see Table 1). Additionally the participation in the MAA provides economic benefits for each participating faculty because this way of cooperating directly results in savings for every partner.
Over a three year period, nearly 450.000€ were needed to develop the first version of the IMS. Following this, the development and maintenance of the IMS is financed by the fees of the participating faculties. The costs for one faculty in the pilot phase is 10 € per student. Thereafter, the cost per year is 30 € per student to a maximum of 25.000 € per year per faculty. This includes organizational and technical support as well as further development of the IMS. Participating faculties benefit economically because they do not have to develop a similar system but only pay for the running costs of the IMS.