Search tips
Search criteria

Results 1-25 (1265316)

Clipboard (0)

Related Articles

1.  A Comprehensive, Multi-modal Evaluation of the Assessment System of an Undergraduate Research Methodology Course: Translating Theory into Practice 
Objective: To evaluate assessment system of the 'Research Methodology Course' using utility criteria (i.e. validity, reliability, acceptability, educational impact, and cost-effectiveness). This study demonstrates comprehensive evaluation of assessment system and suggests a framework for similar courses.
Methods: Qualitative and quantitative methods used for evaluation of the course assessment components (50 MCQ, 3 Short Answer Questions (SAQ) and research project) using the utility criteria. Results of multiple evaluation methods for all the assessment components were collected and interpreted together to arrive at holistic judgments, rather than judgments based on individual methods or individual assessment.
Results: Face validity, evaluated using a self-administered questionnaire (response rate-88.7%) disclosed that the students perceived that there was an imbalance in the contents covered by the assessment. This was confirmed by the assessment blueprint. Construct validity was affected by the low correlation between MCQ and SAQ scores (r=0.326). There was a higher correlation between the project and MCQ (r=0.466)/SAQ (r=0.463) scores. Construct validity was also affected by the presence of recall type of MCQs (70%; 35/50), item construction flaws and non-functioning distractors. High discriminating indices (>0.35) were found in MCQs with moderate difficulty indices (0.3-0.7). Reliability of the MCQs was 0.75 which could be improved up to 0.8 by increasing the number of MCQs to at least 70. A positive educational impact was found in the form of the research project assessment driving students to present/publish their work in conferences/peer reviewed journals. Cost per student to complete the course was US$164.50.
Conclusions: The multi-modal evaluation of an assessment system is feasible and provides thorough and diagnostic information. Utility of the assessment system could be further improved by modifying the psychometrically inappropriate assessment items.
PMCID: PMC3998984  PMID: 24772117
Assessment; Evaluation; Utility Criteria; Research Course
2.  Evaluation of Multiple Choice and Short Essay Question items in Basic Medical Sciences  
Objectives: To evaluate Multiple Choice and Short Essay Question items in Basic Medical Sciences by determining item writing flaws (IWFs) of MCQs along with cognitive level of each item in both methods.
Methods: This analytical study evaluated the quality of the assessment tools used for the first batch in a newly established medical college in Karachi, Pakistan. First and sixth module assessment tools in Biochemistry during 2009-2010 were analyzed. Cognitive level of MCQs and SEQs, were noted and MCQ item writing flaws were also evaluated.
Results: A total of 36 SEQs and 150 MCQs of four items were analyzed. The cognitive level of 83.33% of SEQs was at recall level while remaining 16.67% were assessing interpretation of data. Seventy six percent of the MCQs were at recall level while remaining 24% were at the interpretation. Regarding IWFs, 69 IWFs were found in 150 MCQs. The commonest among them were implausible distracters (30.43%), unfocused stem (27.54%) and unnecessary information in the stem (24.64%).
Conclusion: There is a need to review the quality including the content of assessment tools. A structured faculty development program is recommended for developing improved assessment tools that align with learning outcomes and measure competency of medical students.
PMCID: PMC3955531  PMID: 24639820
Assessment; MCQ; SEQ; Item analysis
3.  Evaluation of Modified Essay Questions (MEQ) and Multiple Choice Questions (MCQ) as a tool for Assessing the Cognitive Skills of Undergraduate Medical Students 
Developing and testing the cognitive skills and abstract thinking of undergraduate medical students are the main objectives of problem based learning. Modified Essay Questions (MEQ) and Multiple Choice Questions (MCQ) may both be designed to test these skills. The objectives of this study were to assess the effectiveness of both forms of questions in testing the different levels of the cognitive skills of undergraduate medical students and to detect any item writing flaws in the questions.
A total of 50 MEQs and 50 MCQs were evaluated. These questions were chosen randomly from various examinations given to different batches of undergraduate medical students taking course MED 411–412 at the Department of Medicine, Qassim University from the years 2005 to 2009. The effectiveness of the questions was determined by two assessors and was defined by the question’s ability to measure higher cognitive skills, as determined by modified Bloom’s taxonomy, and its quality as determined by the presence of item writing flaws. ‘SPSS15’ and ‘Medcalc’ programs were used to tabulate and analyze the data.
The percentage of questions testing the level III (problem solving) cognitive skills of the students was 40% for MEQs and 60% for the MCQs; the remaining questions merely assessed the recall and comprehension. No significant difference was found between MEQ and MCQ in relation to the type of questions (recall; comprehension or problem solving x2 = 5.3, p = 0.07).The agreement between the two assessors was quite high in case of MCQ (kappa=0.609; SE 0.093; 95%CI 0.426 – 0.792) but lower in case of MEQ (kappa=0.195; SE 0.073; 95%CI 0.052 – 0.338). 16% of the MEQs and 12% of the MCQs had item writing flaws.
A well constructed MCQ is superior to MEQ in testing the higher cognitive skills of undergraduate medical students in a problem based learning setup. Constructing an MEQ for assessing the cognitive skills of a student is not a simple task and is more frequently associated with item writing flaws.
PMCID: PMC3312767  PMID: 22489228
Modified essay question; Multiple-choice question; Bloom’s Taxonomy; cognition
4.  The Advanced Trauma Operative Management course in a Canadian residency program 
Canadian Journal of Surgery  2008;51(3):185-189.
The Advanced Trauma Operative Management (ATOM) course was first introduced into Canada in 2003 at the University of Toronto, with senior general surgery residents being the primary focus. We present an assessment of the course in this Canadian general surgery residency program.
We compared trainees' pre-and postcourse self-efficacy scores and multiple choice question (MCQ) examination results, using paired t tests and resident (n = 24) and faculty (n = 7) course ratings made according to a 10-item, 5-point Likert scale. Faculty were previously trained as ATOM instructors.
Mean pre-and postcourse self-efficacy scores were 68.9 (standard deviation [SD] 24.0) and 101.4 (SD 14.8), respectively (p < 0.001). Mean pre-and post-MCQ scores were 16.4 (SD 3.2) and 18.8 (SD 2.7), respectively (p = 0.006). On the Likert scale (1 = strongly disagree, 5 = strongly agree), all faculty and residents rated the following items as 4–5: objectives were met; knowledge, skills, clinical training, judgment and confidence improved; the live animal is a useful representation of clinical trauma; and the course should be continued but would be more appropriate for the fourth rather than the fifth year of residency. Residents rated as 1–2 the item that the human cadaver would be preferable for learning the surgical skills. Of 24 residents, 20 rated as 3 or less the item stating that the course prepares them for trauma management more adequately than their regular training program.
Self-efficacy, trauma knowledge and skills improved significantly with ATOM training. Preference was expressed for the live animal versus cadaver model, for ATOM training in the fourth rather than fifth year of residency and for the view that it complements general surgery trauma training. The data suggest that including ATOM training in Canadian general surgical residency should be considered.
PMCID: PMC2496585  PMID: 18682791
5.  Item and Test Analysis to Identify Quality Multiple Choice Questions (MCQs) from an Assessment of Medical Students of Ahmedabad, Gujarat 
Multiple choice questions (MCQs) are frequently used to assess students in different educational streams for their objectivity and wide reach of coverage in less time. However, the MCQs to be used must be of quality which depends upon its difficulty index (DIF I), discrimination index (DI) and distracter efficiency (DE).
To evaluate MCQs or items and develop a pool of valid items by assessing with DIF I, DI and DE and also to revise/ store or discard items based on obtained results.
Study was conducted in a medical school of Ahmedabad.
Materials and Methods:
An internal examination in Community Medicine was conducted after 40 hours teaching during 1st MBBS which was attended by 148 out of 150 students. Total 50 MCQs or items and 150 distractors were analyzed.
Statistical Analysis:
Data was entered and analyzed in MS Excel 2007 and simple proportions, mean, standard deviations, coefficient of variation were calculated and unpaired t test was applied.
Out of 50 items, 24 had “good to excellent” DIF I (31 - 60%) and 15 had “good to excellent” DI (> 0.25). Mean DE was 88.6% considered as ideal/ acceptable and non functional distractors (NFD) were only 11.4%. Mean DI was 0.14. Poor DI (< 0.15) with negative DI in 10 items indicates poor preparedness of students and some issues with framing of at least some of the MCQs. Increased proportion of NFDs (incorrect alternatives selected by < 5% students) in an item decrease DE and makes it easier. There were 15 items with 17 NFDs, while rest items did not have any NFD with mean DE of 100%.
Study emphasizes the selection of quality MCQs which truly assess the knowledge and are able to differentiate the students of different abilities in correct manner.
PMCID: PMC3968575  PMID: 24696535
Difficulty index; discrimination index; distractor efficiency; multiple choice question or item; nonfunctional distractor (NFD); teaching evaluation
6.  Identification of technical item flaws leads to improvement of the quality of single best Multiple Choice Questions 
Objective: The purpose of the study was to identify technical item flaws in the multiple choice questions submitted for the final exams for the years 2009, 2010 and 2011.
Methods: This descriptive analytical study was carried out in Islamic International Medical College (IIMC). The Data was collected from the MCQ’s submitted by the faculty for the final exams for the year 2009, 2010 and 2011. The data was compiled and evaluated by a three member assessment committee. The data was analyzed for frequency and percentages the categorical data was analyzed by chi-square test.
Results: Overall percentage of flawed item was 67% for the year 2009 of which 21% were for testwiseness and 40% were for irrelevant difficulty. In year 2010 the total item flaws were 36% and 11% testwiseness and 22% were for irrelevant difficulty. The year 2011 data showed decreased overall flaws of 21%. The flaws of testwisness were 7%, irrelevant difficulty were 11%.
Conclusion: Technical item flaws are frequently encountered during MCQ construction, and the identification of flaws leads to improved quality of the single best MCQ’s.
PMCID: PMC3809311  PMID: 24353614
Frequency; Item writing flaws; Testwiseness
Assessment has a powerful influence on curriculum delivery. Medical instructors must use tools which conform to educational principles, and audit them as part of curriculum review.
To generate information to support recommendations for improving curriculum delivery.
Pre-clinical and clinical departments in a College of Medicine, Saudi Arabia.
A self-administered questionnaire was used in a cross-sectional survey to see if assessment tools being used met basic standards of validity, reliability and currency, and if feedback to students was adequate. Excluded were cost, feasibility and tool combinations.
Thirty-one (out of 34) courses were evaluated. All 31 respondents used MCQs, especially one-best (28/31) and true/false (13/31). Groups of teachers selected test questions mostly. Pre-clinical departments sourced equally from “new” (10/14) and “used” (10/14) MCQs; clinical departments relied on ‘banked’ MCQs (16/17). Departments decided pass marks (28/31) and chose the College-set 60%; the timing was pre-examination in 13/17 clinical but post-examination in 5/14 pre-clinical departments. Of six essay users, five used model answers but only one did double marking. OSCE was used by 7/17 clinical departments; five provided checklist. Only 3/31 used optical reader. Post-marking review was done by 13/14 pre-clinical but 10/17 clinical departments. Difficulty and discriminating indices were determined by only 4/31 departments. Feedback was provided by 12/14 pre-clinical and 7/17 clinical departments. Only 10/31 course coordinators had copies of examination regulations.
MCQ with single-best answer, if properly constructed and adequately critiqued, is the preferred tool for assessing theory domain. However, there should be fresh questions, item analyses, comparisons with pervious results, optical reader systems and double marking. Departments should use OSCE or OSPE more often. Long essays, true/false, fill-in-the-blank-spaces and more-than-one-correct-answer can be safely abolished. Departments or teams should set test papers and collectively take decisions. Feedback rates should be improved. A Center of Medical Education, including an Examination Center is required. Fruitful future studies can be repeat audit, use of “negative questions” and the number of MCQs per test paper. Comparative audit involving other regional medical schools may be of general interest.
PMCID: PMC3410120  PMID: 23012084
Assessment Technique; Curriculum review; MCQ
8.  Palliative Care teaching in Germany – concepts and future developments  
Background: Following recent modifications of the Medical Licensure Act (ÄApprO) in the year 2009, palliative care was introduced as a compulsory 13th cross-disciplinary subject (Q13) in the undergraduate curriculum. Its implementation must have taken place before the beginning of the final year (´practical year´) in August 2013 and has to be substantiated for the medical exams taking place in October 2014.
Very diverse structures pertaining to palliative care teaching were described in previous surveys at various medical faculties in Germany. As a result, the current and future plans and concepts related to content and exams of a mandatory Q13 course at the respective faculty sites should be ascertained.
Methods: Since 2006, the German Medical Students' Association (bvmd) has been carrying out a bi-annual survey at all medical faculties in Germany regarding the current situation of teaching in the field of palliative care.
After designing and piloting an online survey in May 2010, a one-month online survey took place. The data was assessed using a descriptive approach.
Results: 31 of 36 medical faculties took part in the survey. At the time of questioning, 15 faculties already taught courses according to the requirements of the new ÄApprO; at three sites the Q13 is yet to be introduced commencing in 2012. A teaching curriculum for Q13 already existed at 15 faculty sites, partly based on the curricular requirements of the German Association for Palliative Medicine (DGP). Six sites described an implementation process as yet without an independent curriculum. Most of the faculties aim for 21-40 course hours, which will for the most part be provided as lectures, seminars or less often in more assisted and intense formats.
The majority of the participating faculties intend an examination containing multiple choice questions. At 8 universities there is an independent Chair for palliative medicine (5 more are planned); this was linked with a higher degree of mandatory teaching in alignment with the requirements of the ÄApprO. A broad spectrum of educationally-involved occupational groups, specialist disciplines and external co-operating partners, were mentioned.
Conclusion: The infrastructural prerequisites of the present curricular concepts and the degree of implementation of the Q13 according to the requirements of the new ÄApprO diverge significantly among the various medical faculties. The efforts made to produce a qualitatively high standard of teaching with regard to the multifaceted questions concerning the support for severely and terminally ill patients is as much reflected in the survey, as the special implications of an independent Chair for palliative medicine for the implementation of the requirements by law. The participation of various occupational groups in this survey as well as the broad spectrum of those involved highlights the interdisciplinary and multi-professional dimension of teaching in palliative care.
PMCID: PMC3374143  PMID: 22737202
Palliative care; cross-disciplinary subject; implementation process; curriculum; examinational didactics
9.  Integrating personalized medical test contents with XML and XSL-FO 
In 2004 the adoption of a modular curriculum at the medical faculty in Muenster led to the introduction of centralized examinations based on multiple-choice questions (MCQs). We report on how organizational challenges of realizing faculty-wide personalized tests were addressed by implementation of a specialized software module to automatically generate test sheets from individual test registrations and MCQ contents.
Key steps of the presented method for preparing personalized test sheets are (1) the compilation of relevant item contents and graphical media from a relational database with database queries, (2) the creation of Extensible Markup Language (XML) intermediates, and (3) the transformation into paginated documents.
The software module by use of an open source print formatter consistently produced high-quality test sheets, while the blending of vectorized textual contents and pixel graphics resulted in efficient output file sizes. Concomitantly the module permitted an individual randomization of item sequences to prevent illicit collusion.
The automatic generation of personalized MCQ test sheets is feasible using freely available open source software libraries, and can be efficiently deployed on a faculty-wide scale.
PMCID: PMC3056837  PMID: 21362187
10.  E-Learning as New Method of Medical Education 
Acta Informatica Medica  2008;16(2):102-117.
Distance learning refers to use of technologies based on health care delivered on distance and covers areas such as electronic health, tele-health (e-health), telematics, telemedicine, tele-education, etc. For the need of e-health, telemedicine, tele-education and distance learning there are various technologies and communication systems from standard telephone lines to the system of transmission digitalized signals with modem, optical fiber, satellite links, wireless technologies, etc. Tele-education represents health education on distance, using Information Communication Technologies (ICT), as well as continuous education of a health system beneficiaries and use of electronic libraries, data bases or electronic data with data bases of knowledge. Distance learning (E-learning) as a part of tele-education has gained popularity in the past decade; however, its use is highly variable among medical schools and appears to be more common in basic medical science courses than in clinical education. Distance learning does not preclude traditional learning processes; frequently it is used in conjunction with in-person classroom or professional training procedures and practices. Tele-education has mostly been used in biomedical education as a blended learning method, which combines tele-education technology with traditional instructor-led training, where, for example, a lecture or demonstration is supplemented by an online tutorial. Distance learning is used for self-education, tests, services and for examinations in medicine i.e. in terms of self-education and individual examination services. The possibility of working in the exercise mode with image files and questions is an attractive way of self education. Automated tracking and reporting of learners’ activities lessen faculty administrative burden. Moreover, e-learning can be designed to include outcomes assessment to determine whether learning has occurred. This review article evaluates the current status and level of tele-education development in Bosnia and Herzegovina outlining its components, faculty development needs for implementation and the possibility of its integration as official learning standard in biomedical curricula in Bosnia and Herzegovina. Tele-education refers to the use of information and communication technologies (ICT) to enhance knowledge and performance. Tele-education in biomedical education is widely accepted in the medical education community where it is mostly integrated into biomedical curricula forming part of a blended learning strategy. There are many biomedical digital repositories of e-learning materials worldwide, some peer reviewed, where instructors or developers can submit materials for widespread use. First pilot project with the aim to introduce tele-education in biomedical curricula in Bosnia and Herzegovina was initiated by Department for Medical Informatics at Medical Faculty in Sarajevo in 2002 and has been developing since. Faculty member’s skills in creating tele-education differ from those needed for traditional teaching and faculty rewards must recognize this difference and reward the effort. Tele-education and use of computers will have an impact of future medical practice in a life long learning. Bologna process, which started last years in European countries, provide us to promote and introduce modern educational methods of education at biomedical faculties in Bosnia and Herzegovina. Cathedra of Medical informatics and Cathedra of Family medicine at Medical Faculty of University of Sarajevo started to use Web based education as common way of teaching of medical students. Satisfaction with this method of education within the students is good, but not yet suitable for most of medical disciplines at biomedical faculties in Bosnia and Herzegovina.
PMCID: PMC3789161  PMID: 24109154
Medical education; Distance learning; Bosnia and Herzegovina
11.  An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis 
Four- or five-option multiple choice questions (MCQs) are the standard in health-science disciplines, both on certification-level examinations and on in-house developed tests. Previous research has shown, however, that few MCQs have three or four functioning distractors. The purpose of this study was to investigate non-functioning distractors in teacher-developed tests in one nursing program in an English-language university in Hong Kong.
Using item-analysis data, we assessed the proportion of non-functioning distractors on a sample of seven test papers administered to undergraduate nursing students. A total of 514 items were reviewed, including 2056 options (1542 distractors and 514 correct responses). Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.
The proportion of items containing 0, 1, 2, and 3 functioning distractors was 12.3%, 34.8%, 39.1%, and 13.8% respectively. Overall, items contained an average of 1.54 (SD = 0.88) functioning distractors. Only 52.2% (n = 805) of all distractors were functioning effectively and 10.2% (n = 158) had a choice frequency of 0. Items with more functioning distractors were more difficult and more discriminating.
The low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.
PMCID: PMC2713226  PMID: 19580681
12.  The validity and reliability of the sixth-year internal medical examination administered at the King Abdulaziz University Medical College 
BMC Medical Education  2015;15:10.
Exams are essential components of medical students’ knowledge and skill assessment during their clinical years of study. The paper provides a retrospective analysis of validity evidence for the internal medicine component of the written and clinical exams administered in 2012 and 2013 at King Abdulaziz University’s Faculty of Medicine.
Students’ scores for the clinical and written exams were obtained. Four faculty members (two senior members and two junior members) were asked to rate the exam questions, including MCQs and OSCEs, for evidence of content validity using a rating scale of 1–5 for each item.
Cronbach’s alpha was used to measure the internal consistency reliability. Correlations were used to examine the associations between different forms of assessment and groups of students.
A total of 824 students completed the internal medicine course and took the exam. The numbers of rated questions were 320 and 46 for the MCQ and OSCE, respectively. Significant correlations were found between the MCQ section, the OSCE section, and the continuous assessment marks, which include 20 long-case presentations during the course; participation in daily rounds, clinical sessions and tutorials; the performance of simple procedures, such as IV cannulation and ABG extraction; and the student log book.
Although the OSCE exam was reliable for the two groups that had taken the final clinical OSCE, the clinical long- and short-case exams were not reliable across the two groups that had taken the oral clinical exams. The correlation analysis showed a significant linear association between the raters with respect to evidence of content validity for both the MCQ and OSCE, r = .219 P < .001 and r = .678 P < .001, respectively, and r = .241 P < .001 and r = .368 P = .023 for the internal structure validity, respectively. Reliability measured using Cronbach’s alpha was greater for assessments administered in 2013.
The pattern of relationships between the MCQ and OSCE scores provides evidence of the validity of these measures for use in the evaluation of knowledge and clinical skills in internal medicine. The OSCE exam is more reliable than the short- and long-case clinical exams and requires less effort on the part of examiners and patients.
PMCID: PMC4322795  PMID: 25638149
Validity; Assessment; Undergraduate medical education
13.  Negatively-Marked MCQ Assessments That Reward Partial Knowledge Do Not Introduce Gender Bias Yet Increase Student Performance and Satisfaction and Reduce Anxiety 
PLoS ONE  2013;8(2):e55956.
Multiple-choice question (MCQ) examinations are increasingly used as the assessment method of theoretical knowledge in large class-size modules in many life science degrees. MCQ-tests can be used to objectively measure factual knowledge, ability and high-level learning outcomes, but may also introduce gender bias in performance dependent on topic, instruction, scoring and difficulty. The ‘Single Answer’ (SA) test is often used in which students choose one correct answer, in which they are unable to demonstrate partial knowledge. Negatively marking eliminates the chance element of guessing but may be considered unfair. Elimination testing (ET) is an alternative form of MCQ, which discriminates between all levels of knowledge, while rewarding demonstration of partial knowledge. Comparisons of performance and gender bias in negatively marked SA and ET tests have not yet been performed in the life sciences. Our results show that life science students were significantly advantaged by answering the MCQ test in elimination format compared to single answer format under negative marking conditions by rewarding partial knowledge of topics. Importantly, we found no significant difference in performance between genders in either cohort for either MCQ test under negative marking conditions. Surveys showed that students generally preferred ET-style MCQ testing over SA-style testing. Students reported feeling more relaxed taking ET MCQ and more stressed when sitting SA tests, while disagreeing with being distracted by thinking about best tactics for scoring high. Students agreed ET testing improved their critical thinking skills. We conclude that appropriately-designed MCQ tests do not systematically discriminate between genders. We recommend careful consideration in choosing the type of MCQ test, and propose to apply negative scoring conditions to each test type to avoid the introduction of gender bias. The student experience could be improved through the incorporation of the elimination answering methods in MCQ tests via rewarding partial and full knowledge.
PMCID: PMC3577794  PMID: 23437081
14.  Association of Medical Students' Reports of Interactions with the Pharmaceutical and Medical Device Industries and Medical School Policies and Characteristics: A Cross-Sectional Study 
PLoS Medicine  2014;11(10):e1001743.
Aaron Kesselheim and colleagues compared US medical students' survey responses regarding pharmaceutical company interactions with the schools' AMSA PharmFree scorecard and Institute on Medicine as a Profession's (IMAP) scores.
Please see later in the article for the Editors' Summary
Professional societies use metrics to evaluate medical schools' policies regarding interactions of students and faculty with the pharmaceutical and medical device industries. We compared these metrics and determined which US medical schools' industry interaction policies were associated with student behaviors.
Methods and Findings
Using survey responses from a national sample of 1,610 US medical students, we compared their reported industry interactions with their schools' American Medical Student Association (AMSA) PharmFree Scorecard and average Institute on Medicine as a Profession (IMAP) Conflicts of Interest Policy Database score. We used hierarchical logistic regression models to determine the association between policies and students' gift acceptance, interactions with marketing representatives, and perceived adequacy of faculty–industry separation. We adjusted for year in training, medical school size, and level of US National Institutes of Health (NIH) funding. We used LASSO regression models to identify specific policies associated with the outcomes. We found that IMAP and AMSA scores had similar median values (1.75 [interquartile range 1.50–2.00] versus 1.77 [1.50–2.18], adjusted to compare scores on the same scale). Scores on AMSA and IMAP shared policy dimensions were not closely correlated (gift policies, r = 0.28, 95% CI 0.11–0.44; marketing representative access policies, r = 0.51, 95% CI 0.36–0.63). Students from schools with the most stringent industry interaction policies were less likely to report receiving gifts (AMSA score, odds ratio [OR]: 0.37, 95% CI 0.19–0.72; IMAP score, OR 0.45, 95% CI 0.19–1.04) and less likely to interact with marketing representatives (AMSA score, OR 0.33, 95% CI 0.15–0.69; IMAP score, OR 0.37, 95% CI 0.14–0.95) than students from schools with the lowest ranked policy scores. The association became nonsignificant when fully adjusted for NIH funding level, whereas adjusting for year of education, size of school, and publicly versus privately funded school did not alter the association. Policies limiting gifts, meals, and speaking bureaus were associated with students reporting having not received gifts and having not interacted with marketing representatives. Policy dimensions reflecting the regulation of industry involvement in educational activities (e.g., continuing medical education, travel compensation, and scholarships) were associated with perceived separation between faculty and industry. The study is limited by potential for recall bias and the cross-sectional nature of the survey, as school curricula and industry interaction policies may have changed since the time of the survey administration and study analysis.
As medical schools review policies regulating medical students' industry interactions, limitations on receipt of gifts and meals and participation of faculty in speaking bureaus should be emphasized, and policy makers should pay greater attention to less research-intensive institutions.
Please see later in the article for the Editors' Summary
Editors' Summary
Making and selling prescription drugs and medical devices is big business. To promote their products, pharmaceutical and medical device companies build relationships with physicians by providing information on new drugs, by organizing educational meetings and sponsored events, and by giving gifts. Financial relationships begin early in physicians' careers, with companies providing textbooks and other gifts to first-year medical students. In medical school settings, manufacturers may help to inform trainees and physicians about developments in health care, but they also create the potential for harm to patients and health care systems. These interactions may, for example, reduce trainees' and trained physicians' skepticism about potentially misleading promotional claims and may encourage physicians to prescribe new medications, which are often more expensive than similar unbranded (generic) drugs and more likely to be recalled for safety reasons than older drugs. To address these and other concerns about the potential career-long effects of interactions between medical trainees and industry, many teaching hospitals and medical schools have introduced policies to limit such interactions. The development of these policies has been supported by expert professional groups and medical societies, some of which have created scales to evaluate the strength of the implemented industry interaction policies.
Why Was This Study Done?
The impact of policies designed to limit interactions between students and industry on student behavior is unclear, and it is not known which aspects of the policies are most predictive of student behavior. This information is needed to ensure that the policies are working and to identify ways to improve them. Here, the researchers investigate which medical school characteristics and which aspects of industry interaction policies are most predictive of students' reported behaviors and beliefs by comparing information collected in a national survey of US medical students with the strength of their schools' industry interaction policies measured on two scales—the American Medical Student Association (AMSA) PharmFree Scorecard and the Institute on Medicine as a Profession (IMAP) Conflicts of Interest Policy Database.
What Did the Researchers Do and Find?
The researchers compared information about reported gift acceptance, interactions with marketing representatives, and the perceived adequacy of faculty–industry separation collected from 1,610 medical students at 121 US medical schools with AMSA and IMAP scores for the schools evaluated a year earlier. Students at schools with the highest ranked interaction policies based on the AMSA score were 63% less likely to accept gifts as students at the lowest ranked schools. Students at the highest ranked schools based on the IMAP score were about half as likely to accept gifts as students at the lowest ranked schools, although this finding was not statistically significant (it could be a chance finding). Similarly, students at the highest ranked schools were 70% less likely to interact with sales representatives as students at the lowest ranked schools. These associations became statistically nonsignificant after controlling for the amount of research funding each school received from the US National Institutes of Health (NIH). Policies limiting gifts, meals, and being a part of speaking bureaus (where companies pay speakers to present information about the drugs for dinners and other events) were associated with students' reports of receiving no gifts and of non-interaction with sales representatives. Finally, policies regulating industry involvement in educational activities were associated with the perceived separation between faculty and industry, which was regarded as adequate by most of the students at schools with such policies.
What Do These Findings Mean?
These findings suggest that policies designed to limit industry interactions with medical students need to address multiple aspects of these interactions to achieve changes in the behavior and attitudes of trainees, but that policies limiting gifts, meals, and speaking bureaus may be particularly important. These findings also suggest that the level of NIH funding plays an important role in students' self-reported behaviors and their perceptions of industry, possibly because institutions with greater NIH funding have the resources needed to implement effective policies. The accuracy of these findings may be limited by recall bias (students may have reported their experiences inaccurately), and by the possibility that industry interaction policies may have changed in the year that elapsed between policy grading and the student survey. Nevertheless, these findings suggest that limitations on gifts should be emphasized when academic medical centers refine their policies on interactions between medical students and industry and that particular attention should be paid to the design and implementation of policies that regulate industry interactions in institutions with lower levels of NIH funding.
Additional Information
Please access these websites via the online version of this summary at
The UK General Medical Council provides guidance on financial and commercial arrangements and conflicts of interest as part of its good medical practice document, which describes what is required of all registered doctors in the UK
Information about the American Medical Student Association (AMSA) Just Medicine campaign (formerly the PharmFree campaign) and about the AMSA Scorecard is available
Information about the Institute on Medicine as a Profession (IMAP) and about its Conflicts of Interest Policy Database is also available
“Understanding and Responding to Pharmaceutical Promotion: A Practical Guide” is a manual prepared by Health Action International and the World Health Organization that medical schools can use to train students how to recognize and respond to pharmaceutical promotion
The US Institute of Medicine's report “Conflict of Interest in Medical Research, Education, and Practice” recommends steps to identify, limit, and manage conflicts of interest
The ALOSA Foundation provides evidence-based, non-industry-funded education about treating common conditions and using prescription drugs
PMCID: PMC4196737  PMID: 25314155
15.  Contextual adaptation of the Personnel Evaluation Standards for assessing faculty evaluation systems in developing countries: the case of Iran 
Faculty evaluations can identify needs to be addressed in effective development programs. Generic evaluation models exist, but these require adaptation to a particular context of interest. We report on one approach to such adaptation in the context of medical education in Iran, which is integrated into the delivery and management of healthcare services nationwide.
Using a triangulation design, interviews with senior faculty leaders were conducted to identify relevant areas for faculty evaluation. We then adapted the published checklist of the Personnel Evaluation Standards to fit the Iranian medical universities' context by considering faculty members' diverse roles. Then the adapted instrument was administered to faculty at twelve medical schools in Iran.
The interviews revealed poor linkages between existing forms of development and evaluation, imbalance between the faculty work components and evaluated areas, inappropriate feedback and use of information in decision making. The principles of Personnel Evaluation Standards addressed almost all of these concerns and were used to assess the existing faculty evaluation system and also adapted to evaluate the core faculty roles. The survey response rate was 74%. Responses showed that the four principles in all faculty members' roles were met occasionally to frequently. Evaluation of teaching and research had the highest mean scores, while clinical and healthcare services, institutional administration, and self-development had the lowest mean scores. There were statistically significant differences between small medium and large medical schools (p < 0.000).
The adapted Personnel Evaluation Standards appears to be valid and applicable for monitoring and continuous improvement of a faculty evaluation system in the context of medical universities in Iran. The approach developed here provides a more balanced assessment of multiple faculty roles, including educational, clinical and healthcare services. In order to address identified deficiencies, the evaluation system should recognize, document, and uniformly reward those activities that are vital to the academic mission. Inclusion of personal developmental concerns in the evaluation discussion is essential for evaluation systems.
PMCID: PMC2680845  PMID: 19400932
16.  Comparison of Collegial Individual and Group Reviews of General Practice Multiple Choice Questions 
Aims: In most German medical faculties, credits in general practice can be earned via exams using multiple-choice questions (MCQ). Measures such as peer-reviews may help assure the quality of these exams. In order to use time and personnel intensive peer reviews effectively and efficiently, the procedures used are key. Therefore, we wanted to find out whether there are differences between group and individual reviews regarding defined parameters.
Methods: We conducted a controlled cross-over study with three GP reviewers from four different German universities. Each reviewed 80 MCQs, 40 individually and 40 within a group, including external assessments by a panel of experts. Furthermore all reviewers were asked to evaluate the review process and the time spent carrying out these reviews.
Outcomes: We found no significant differences between the reliability and the validity of individual reviews versus group reviews. On average slightly more time was spent on group reviews compared with the individual reviews. The subjective assessments of the study participants regarding their satisfaction with the process and the efficiency and effectiveness of the reviews suggest a preference for group reviews.
Conclusions: Based on this study, there are no definite recommendations for or against either approach. When choosing between the two, the specific work structures and organisation at the local faculty should be taken into account.
PMCID: PMC3420119  PMID: 22916083
Medical Educatio; assessment; Multiple-Choice-Questions; Review
17.  Home Telehealth for Patients With Chronic Obstructive Pulmonary Disease (COPD) 
Executive Summary
In July 2010, the Medical Advisory Secretariat (MAS) began work on a Chronic Obstructive Pulmonary Disease (COPD) evidentiary framework, an evidence-based review of the literature surrounding treatment strategies for patients with COPD. This project emerged from a request by the Health System Strategy Division of the Ministry of Health and Long-Term Care that MAS provide them with an evidentiary platform on the effectiveness and cost-effectiveness of COPD interventions.
After an initial review of health technology assessments and systematic reviews of COPD literature, and consultation with experts, MAS identified the following topics for analysis: vaccinations (influenza and pneumococcal), smoking cessation, multidisciplinary care, pulmonary rehabilitation, long-term oxygen therapy, noninvasive positive pressure ventilation for acute and chronic respiratory failure, hospital-at-home for acute exacerbations of COPD, and telehealth (including telemonitoring and telephone support). Evidence-based analyses were prepared for each of these topics. For each technology, an economic analysis was also completed where appropriate. In addition, a review of the qualitative literature on patient, caregiver, and provider perspectives on living and dying with COPD was conducted, as were reviews of the qualitative literature on each of the technologies included in these analyses.
The Chronic Obstructive Pulmonary Disease Mega-Analysis series is made up of the following reports, which can be publicly accessed at the MAS website at:
Chronic Obstructive Pulmonary Disease (COPD) Evidentiary Framework
Influenza and Pneumococcal Vaccinations for Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Smoking Cessation for Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Community-Based Multidisciplinary Care for Patients With Stable Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Pulmonary Rehabilitation for Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Long-term Oxygen Therapy for Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Noninvasive Positive Pressure Ventilation for Acute Respiratory Failure Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Noninvasive Positive Pressure Ventilation for Chronic Respiratory Failure Patients With Stable Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Hospital-at-Home Programs for Patients With Acute Exacerbations of Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Home Telehealth for Patients With Chronic Obstructive Pulmonary Disease (COPD): An Evidence-Based Analysis
Cost-Effectiveness of Interventions for Chronic Obstructive Pulmonary Disease Using an Ontario Policy Model
Experiences of Living and Dying With COPD: A Systematic Review and Synthesis of the Qualitative Empirical Literature
For more information on the qualitative review, please contact Mita Giacomini at:
For more information on the economic analysis, please visit the PATH website:
The Toronto Health Economics and Technology Assessment (THETA) collaborative has produced an associated report on patient preference for mechanical ventilation. For more information, please visit the THETA website:
The objective of this analysis was to conduct an evidence-based assessment of home telehealth technologies for patients with chronic obstructive pulmonary disease (COPD) in order to inform recommendations regarding the access and provision of these services in Ontario. This analysis was one of several analyses undertaken to evaluate interventions for COPD. The perspective of this assessment was that of the Ontario Ministry of Health and Long-Term Care, a provincial payer of medically necessary health care services.
Clinical Need: Condition and Target Population
Canada is facing an increase in chronic respiratory diseases due in part to its aging demographic. The projected increase in COPD will put a strain on health care payers and providers. There is therefore an increasing demand for telehealth services that improve access to health care services while maintaining or improving quality and equality of care. Many telehealth technologies however are in the early stages of development or diffusion and thus require study to define their application and potential harms or benefits. The Medical Advisory Secretariat (MAS) therefore sought to evaluate telehealth technologies for COPD.
Telemedicine (or telehealth) refers to using advanced information and communication technologies and electronic medical devices to support the delivery of clinical care, professional education, and health-related administrative services.
Generally there are 4 broad functions of home telehealth interventions for COPD:
to monitor vital signs or biological health data (e.g., oxygen saturation),
to monitor symptoms, medication, or other non-biologic endpoints (e.g., exercise adherence),
to provide information (education) and/or other support services (such as reminders to exercise or positive reinforcement), and
to establish a communication link between patient and provider.
These functions often require distinct technologies, although some devices can perform a number of these diverse functions. For the purposes of this review, MAS focused on home telemonitoring and telephone only support technologies.
Telemonitoring (or remote monitoring) refers to the use of medical devices to remotely collect a patient’s vital signs and/or other biologic health data and the transmission of those data to a monitoring station for interpretation by a health care provider.
Telephone only support refers to disease/disorder management support provided by a health care provider to a patient who is at home via telephone or videoconferencing technology in the absence of transmission of patient biologic data.
Research Questions
What is the effectiveness, cost-effectiveness, and safety of home telemonitoring compared with usual care for patients with COPD?
What is the effectiveness, cost-effectiveness, and safety of telephone only support programs compared with usual care for patients with COPD?
Research Methods
Literature Search
Search Strategy
A literature search was performed on November 3, 2010 using OVID MEDLINE, MEDLINE In-Process and Other Non-Indexed Citations, EMBASE, the Cumulative Index to Nursing & Allied Health Literature (CINAHL), the Cochrane Library, and the International Agency for Health Technology Assessment (INAHTA) for studies published from January 1, 2000 until November 3, 2010. Abstracts were reviewed by a single reviewer and, for those studies meeting the eligibility criteria, full-text articles were obtained. Reference lists were also examined for any additional relevant studies not identified through the search. Articles with unknown eligibility were reviewed with a second clinical epidemiologist, and then a group of epidemiologists until consensus was established. The quality of evidence was assessed as high, moderate, low, or very low according to GRADE methodology.
Inclusion Criteria – Question #1
frequent transmission of a patient’s physiological data collected at home and without a health care professional physically present to health care professionals for routine monitoring through the use of a communication technology;
monitoring combined with a coordinated management and feedback system based on transmitted data;
telemonitoring as a key component of the intervention (subjective determination);
usual care as provided by the usual care provider for the control group;
randomized controlled trials (RCTs), controlled clinical trials (CCTs), systematic reviews, and/or meta-analyses;
published between January 1, 2000 and November 3, 2010.
Inclusion Criteria – Question #2
scheduled or frequent contact between patient and a health care professional via telephone or videoconferencing technology in the absence of transmission of patient physiological data;
monitoring combined with a coordinated management and feedback system based on transmitted data;
telephone support as a key component of the intervention (subjective determination);
usual care as provided by the usual care provider for the control group;
RCTs, CCTs, systematic reviews, and/or meta-analyses;
published between January 1, 2000 and November 3, 2010.
Exclusion Criteria
published in a language other than English;
intervention group (and not control) receiving some form of home visits by a medical professional, typically a nurse (i.e., telenursing) beyond initial technology set-up and education, to collect physiological data, or to somehow manage or treat the patient;
not recording patient or health system outcomes (e.g., technical reports testing accuracy, reliability or other development-related outcomes of a device, acceptability/feasibility studies, etc.);
not using an independent control group that received usual care (e.g., studies employing historical or periodic controls).
Outcomes of Interest
hospitalizations (primary outcome)
emergency department visits
length of stay
quality of life
other […]
Subgroup Analyses (a priori)
length of intervention (primary)
severity of COPD (primary)
Quality of Evidence
The quality of evidence assigned to individual studies was determined using a modified CONSORT Statement Checklist for Randomized Controlled Trials. (1) The CONSORT Statement was adapted to include 3 additional quality measures: the adequacy of control group description, significant differential loss to follow-up between groups, and greater than or equal to 30% study attrition. Individual study quality was defined based on total scores according to the CONSORT Statement checklist: very low (0 to < 40%), low (≥ 40 to < 60%), moderate (≥ 60 to < 80%), and high (≥ 80 to 100%).
The quality of the body of evidence was assessed as high, moderate, low, or very low according to the GRADE Working Group criteria. The following definitions of quality were used in grading the quality of the evidence:
Summary of Findings
Six publications, representing 5 independent trials, met the eligibility criteria for Research Question #1. Three trials were RCTs reported across 4 publications, whereby patients were randomized to home telemonitoring or usual care, and 2 trials were CCTs, whereby patients or health care centers were nonrandomly assigned to intervention or usual care.
A total of 310 participants were studied across the 5 included trials. The mean age of study participants in the included trials ranged from 61.2 to 74.5 years for the intervention group and 61.1 to 74.5 years for the usual care group. The percentage of men ranged from 40% to 64% in the intervention group and 46% to 72% in the control group.
All 5 trials were performed in a moderate to severe COPD patient population. Three trials initiated the intervention following discharge from hospital. One trial initiated the intervention following a pulmonary rehabilitation program. The final trial initiated the intervention during management of patients at an outpatient clinic.
Four of the 5 trials included oxygen saturation (i.e., pulse oximetry) as one of the biological patient parameters being monitored. Additional parameters monitored included forced expiratory volume in one second, peak expiratory flow, and temperature.
There was considerable clinical heterogeneity between trials in study design, methods, and intervention/control. In relation to the telemonitoring intervention, 3 of the 5 included studies used an electronic health hub that performed multiple functions beyond the monitoring of biological parameters. One study used only a pulse oximeter device alone with modem capabilities. Finally, in 1 study, patients measured and then forwarded biological data to a nurse during a televideo consultation. Usual care varied considerably between studies.
Only one trial met the eligibility criteria for Research Question #2. The included trial was an RCT that randomized 60 patients to nurse telephone follow-up or usual care (no telephone follow-up). Participants were recruited from the medical department of an acute-care hospital in Hong Kong and began receiving follow-up after discharge from the hospital with a diagnosis of COPD (no severity restriction). The intervention itself consisted of only two 10-to 20-minute telephone calls, once between days 3 to 7 and once between days 14 to 20, involving a structured, individualized educational and supportive programme led by a nurse that focused on 3 components: assessment, management options, and evaluation.
Regarding Research Question #1:
Low to very low quality evidence (according to GRADE) finds non-significant effects or conflicting effects (of significant or non-significant benefit) for all outcomes examined when comparing home telemonitoring to usual care.
There is a trend towards significant increase in time free of hospitalization and use of other health care services with home telemonitoring, but these findings need to be confirmed further in randomized trials of high quality.
There is severe clinical heterogeneity between studies that limits summary conclusions.
The economic impact of home telemonitoring is uncertain and requires further study.
Home telemonitoring is largely dependent on local information technologies, infrastructure, and personnel, and thus the generalizability of external findings may be low. Jurisdictions wishing to replicate home telemonitoring interventions should likely test those interventions within their jurisdictional framework before adoption, or should focus on home-grown interventions that are subjected to appropriate evaluation and proven effective.
Regarding Research Question #2:
Low quality evidence finds significant benefit in favour of telephone-only support for self-efficacy and emergency department visits when compared to usual care, but non-significant results for hospitalizations and hospital length of stay.
There are very serious issues with the generalizability of the evidence and thus additional research is required.
PMCID: PMC3384362  PMID: 23074421
18.  Multiple choice questions are superior to extended matching questions to identify medicine and biomedical sciences students who perform poorly 
Perspectives on Medical Education  2013;2(5-6):252-263.
In recent years, medical faculties at Dutch universities have implemented a legally binding study advice to students of medicine and biomedical sciences during their propaedeutic phase. Appropriate examination is essential to discriminate between poor (grade <6), moderate (grade 6–8) and excellent (grade ≥8) students. Therefore, we compared the discriminatory properties of extended matching questions (EMQs) versus multiple-choice questions (MCQs) and identified the role of sex, age and examination preference on this score. Data were collected for 452 first-year medical and biomedical science students during three distinct course examinations: one examination with EMQ only, one with MCQ only and one mixed examination (including EMQ and MCQ). Logistic regression analysis revealed that MCQ examination was 3 times better in identifying poor students compared with EMQ (RR 3.0, CI 2.0–4.5), whereas EMQ better detected excellent students (average grade ≥8) (RR 1.93, CI 1.47–2.53). Mixed examination had comparable characteristics to MCQ. Sex and examination preference did not impact the score of the student. Students ≥20 years had a 4-fold higher risk ratio of obtaining a poor grade (<6) compared with students ≤18 years old (RR 4.1, CI 2.1–8.0). Given the strong discriminative capacity of MCQ examinations to identify poor students, we recommend the use of this type of examination during the propaedeutic phase of medicine and biomedical science study programmes, in the light of the binding study advice.
PMCID: PMC3824749  PMID: 24203858
Binding study advice; Medical education; Medicine; Biomedical sciences; Propaedeutic phase; Age
19.  Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper 
Reliable and valid written tests of higher cognitive function are difficult to produce, particularly for the assessment of clinical problem solving. Modified Essay Questions (MEQs) are often used to assess these higher order abilities in preference to other forms of assessment, including multiple-choice questions (MCQs). MEQs often form a vital component of end-of-course assessments in higher education. It is not clear how effectively these questions assess higher order cognitive skills. This study was designed to assess the effectiveness of the MEQ to measure higher-order cognitive skills in an undergraduate institution.
An analysis of multiple-choice questions and modified essay questions (MEQs) used for summative assessment in a clinical undergraduate curriculum was undertaken. A total of 50 MCQs and 139 stages of MEQs were examined, which came from three exams run over two years. The effectiveness of the questions was determined by two assessors and was defined by the questions ability to measure higher cognitive skills, as determined by a modification of Bloom's taxonomy, and its quality as determined by the presence of item writing flaws.
Over 50% of all of the MEQs tested factual recall. This was similar to the percentage of MCQs testing factual recall. The modified essay question failed in its role of consistently assessing higher cognitive skills whereas the MCQ frequently tested more than mere recall of knowledge.
Construction of MEQs, which will assess higher order cognitive skills cannot be assumed to be a simple task. Well-constructed MCQs should be considered a satisfactory replacement for MEQs if the MEQs cannot be designed to adequately test higher order skills. Such MCQs are capable of withstanding the intellectual and statistical scrutiny imposed by a high stakes exit examination.
PMCID: PMC2148038  PMID: 18045500
20.  Medical Informatics Education at Medical Faculty of Sarajevo University - 15 Years Experience 
Acta Informatica Medica  2008;16(1):4-9.
In Bosnia and Herzegovina, Medical informatics has been a separate subject for the last 15 years with regard to Medical curriculum at the biomedical faculties in the country (1,2). Education in the field of Medical informatics is based on the concept which is used in developed countries, according to the recommendations of the working groups EDU – Education of Medical Informatics, of the European Federation for Medical Informatics (EFMI) and International Medical Informatics Association (IMIA). Theoretical and practical teaching and training performance as a whole is performed by use of the computer equipment, and the final knowledge check of the students is also performed using the Data Base Management System MS Access specifically designed to cover full teaching and training material by using question sets in the data base which encircled nearly 1500 question combinations. The distance learning is logical step that can further improve this method of education. In this paper, authors present 15 years of experience of Medical informatics education at biomedical faculties in Bosnia and Herzegovina. Medical Informatics, as an obligatory subject, was introduced to the biomedical faculties in Sarajevo (medical, dental and pharmaceutical as well as the High medical school) in 1992 and 1993. Students have practical computer exercises for a period of 7 weeks. Students had training in Excel, Word etc. During the semester, the students perform specific operation such as creation of data carrier for manipulation with medical information. The information was analyzed by statistical program such as Excel. From 2002 years Medical Informatics is divided in two parts in order to facilitate data processing and other procedure that are necessary to perform at time when student’s knowledge of medicine is sufficient for practicing specific tasks that include management the data about patient, anamnesis and similar parameters cause we noticed that students without such knowledge cannot figure out the whole picture without difficulties. The Theoretical part of examination is done using the multiple choice answer form provided by special software with randomly selected questions for each student. Such way of practical and theoretical path of final exam make possible to perform such procedures such as electronic registration for exam and distance testing. Possibilities of introduction of distance learning in medical curriculum are the title of project which has been realizing at Cathedra for medical Informatics, Medical faculty since year 2002. Our undergraduate and postgraduate students are satisfied with contents and organization of the teaching process.
PMCID: PMC3789160  PMID: 24109152
Medical informatics; education; distance learning
21.  A self-evaluation tool for integrated care services: the Development Model for Integrated Care applied in practice 
The purpose of the workshop is to show the applications of the Development Model for Integrated Care (DMIC) in practice. This relatively new and validated model, can be used by integrated care practices to evaluate their integrated care, to assess their phase of development and reveal improvement areas. In the workshop the results of the use of the model in three types of integrated care settings in the Netherlands will be presented. Participants are offered practical instruments based on the validated DMIC to use in their own setting and will be introduced to the webbased tool.
To integrate care from multiple providers into a coherent and streamlined client-focused service, a large number of activities and agreements have to be implemented like streamlining information flows and adequate transfers of clients. In the large range of possible activities it is often not clear what essential activities are and where to start or continue. Also, knowledge about how to further develop integrated care services is needed. The Development Model for Integrated Care (DMIC), based on PhD research of Mirella Minkman, describes nine clusters containing in total 89 elements that contribute to the integration of care. The clusters are named: ‘client-centeredness’, ‘delivery system’, ‘performance management’, ‘quality of care’, ‘result-focused learning’, ‘interprofessional teamwork’, ‘roles and tasks’, ‘commitment’, and ‘transparant entrepreneurship’ [1–3]. In 2011 a new digital webbased self-evolution tool which contains the 89 elements grouped in nine clusters was developed. The DMIC also describes four phases of development [4]. The model is empirically validated in practice by assessing the relevance and implementation of the elements and development phases in 84 integrated care services in The Netherlands: in stroke, acute myocardial infarct (AMI), and dementia services. The validation studies are recently published [5, 6]. In 2011 also other integrated care services started using the model [7]. Vilans developed a digital web-based self-evaluation tool for integrated care services based on the DMIC. A palliative care network, four diabetes services, a youth care service and a network for autism used the self-evaluation tool to evaluate the development of their integrated care service. Because of its generic character, the model and tool are believed to be also interesting internationally.
Data sources
In the workshop we will present the results of three studies in integrated diabetes, youth and palliative care. The three projects consist of multiple steps, see below. Workshop participants could also work with the DMIC following these steps.
One: Preparation of the digital self-evolution tool for integrated care services
Although they are very different, the three integrated care services all wanted to gain insight in their development and improvement opportunities. We tailored the digital self-evaluation tool for each specific integrated care services, but for all the basis was the DMIC. Personal accounts for the digital DMIC self-evalution survey were sent to multiple partners working in each integrated care service (4–16 partners).
Two: Use of the online self-evaluation tool each partner of the local integrated care setting evaluated the integrated care by filling in the web-based questionnaire. The tool consists of three parts (A-C) named: general information about the integrated care practice (A); the clusters and elements of the DMIC (B); and the four phases of development (C). The respondents rated the relevance and presence of each element in their integrated care practice. Respondents were asked to estimate in which phase of development their thought their service was.
Three: Analysing the results
Advisers from Vilans, the Centre of excellence for long-term care in the Netherlands, analysed the self-evolution results in cooperation with the integrated care coordinators. The results show the total amount of implemented integrated care elements per cluster in spider graphs and the development phase as calculated by the DMIC model. Suggestions for further development of the integrated care services were analysed and reported.
Four: Discussing the implications for further development
In a workshop with the local integrated care partners the results of the self-evaluation were presented and discussed. We noticed remarkable results and highlight elements for further development. In addition, we gave advice for further development appropriate to the development phase of the integrated care service. Furthermore, the professionals prioritized the elements and decided which elements to start working on. This resulted in a (quality improvement) plan for the further development of the integrated care service.
Five: Reporting results
In a report all the results of the survey (including consensus scores) and the workshops came together. The integrated care coordinators stated that the reports really helped them to assess their improvement strategy. Also, there was insight in the development phase of their service which gave tools for further development.
Case description
The three cases presented are a palliative network, an integrated diabetes services and an integrated care network for youth in the Netherlands. The palliative care network wanted to reflect on their current development, to build a guiding framework for further development of the network. About sixteen professionals within the network worked with the digital self-evaluation tool and the DMIC: home care organisations, welfare organizations, hospice centres, health care organisations, community organizations.
For diabetes care, a Dutch health care insurance company wished to gain insight in the development of the contracted integrated care services to stimulate further development of the services. Professionals of three diabetes integrated care services were invited to fill in the digital self-evaluation tool. Of each integrated care service professionals like a general practitioner, a diabetes nurse, a medical specialist, a dietician and a podiatrist were invited. In youth care, a local health organisation wondered whether the DMIC could be helpful to visualize the results of youth integrated care services at process- and organisational level. The goal of the project was to define indicators at a process- and organisational level for youth care services based on the DMIC. In the future, these indicators might be used to evaluate youth care integrated care services and improve the quality of youth care within the Netherlands.
Conclusions and discussion
It is important for the quality of integrated care services that the involved coordinators, managers and professionals are aware of the development process of the integrated care service and that they focus on elements which can further develop and improve their integrated care. However, we noticed that integrated care services in the Netherlands experience difficulties in developing their integrated care service. It is often not clear what essential activities are to work on and how to further develop the integrated care service. A guiding framework for the development of integrated care was missing. The DMIC model has been developed for that reason and offers a useful tool for assessment, self-evaluation or improvement of integrated care services in practice. The model has been validated for AMI, dementia and stroke services. The latest new studies in diabetes, palliative care and youth care gave further insight in the generic character of the DMIC. Based on these studies it can be assumed that the DMIC can be used for multiple types of integrated care services. The model is assumed to be interesting for an international audience. Improving integrated care is a complex topic in a large number of countries; the DMIC is also based on the international literature. Dutch integrated care coordinators stated that the DMIC helped them to assess their integrated care development in practice and supported them in obtaining ideas for expanding and improving their integrated care activities.
The web-based self-evaluation tool focuses on a process- and organisational level of integrated care. Also, the self assessed development phase can be compared to the development phase as calculated by the DMIC tool. The cases showed this is fruitful input for discussions. When using the tool, the results can also be used in quality policy reports and improvement plans. The web-based tool is being tested at this moment in practice, but in San Marino we can present the latest webversion and demonstrate with a short video how to use the tool and model. During practical exercises in the workshop the participants will experience how the application of the DMIC can work for them in practice or in research. For integrated care researchers and policy makers, the DMIC questionnaire and tool is a promising method for further research and policy plans in integrated care.
PMCID: PMC3617779
development model for integrated care; development of integrated care services; implementation and improvement of integrated care; self evaluation
22.  Impact of HMO market structure on physician-hospital strategic alliances. 
Health Services Research  2000;35(1 Pt 1):101-132.
OBJECTIVE: To assess the impact of HMO market structure on the formation of physician-hospital strategic alliances from 1993 through 1995. The two trends, managed care and physician-hospital integration have been prominent in reshaping insurance and provider markets over the past decade. STUDY DESIGN: Pooled cross-sectional data from the InterStudy HMO Census and the Annual Survey conducted by the American Hospital Association (AHA) between 1993 and the end of 1995 to examine the effects of HMO penetration and HMO numbers in a market on the formation of hospital-sponsored alliances with physicians. Because prior research has found nonlinear effects of HMOs on a variety of dependent variables, we operationalized HMO market structure two ways: using a Taylor series expansion and cross-classifying quartile distributions of HMO penetration and numbers into 16 dummy indicators. Alliance formation was operationalized using the presence of any alliance model (IPA, PHO, MSO, and foundation) and the sum of the four models present in the hospital. Because managed care and physician-hospital integration are endogenous (e.g., some hospitals also sponsor HMOs), we used an instrumental variables approach to model the determinants of HMO penetration and HMO numbers. These instruments were then used with other predictors of alliance formation: physician supply characteristics, the extent of hospital competition, hospital-level descriptors, population size and demographic characteristics, and indicators for each year. All equations were estimated at the MSA level using mixed linear models and first-difference models. PRINCIPAL FINDINGS: Contrary to conventional wisdom, alliance formation is shaped by the number of HMOs in the market rather than by HMO penetration. This confirms a growing perception that hospital-sponsored alliances with physicians are contracting vehicles for managed care: the greater the number of HMOs to contract with, the greater the development of alliances. The models also show that alliance formation is low in markets where a small number of HMOs have deeply penetrated the market. First-difference models further show that alliance formation is linked to HMO consolidation (drop in the number of HMOs in a market) and hospital downsizing. Alliance formation is not linked to changes in hospital costs, profitability, or market competition with other hospitals. CONCLUSIONS: Hospitals appear to form alliances with physicians for several reasons. Alliances serve to contract with the growing number of HMOs, to pose a countervailing bargaining force of providers in the face of HMO consolidation, and to accompany hospital downsizing and restructuring efforts. IMPLICATIONS FOR POLICY, DELIVERY, OR PRACTICE: Physician-hospital integration is often mentioned as a provider response to increasing cost-containment pressures due to rising managed care penetration. Our findings do not support this view. Alliances appear to serve the hospital's interest in bargaining with managed care plans on a more even basis.
PMCID: PMC1089117  PMID: 10778826
23.  Developing a Practical and Sustainable Faculty Development Program With a Focus on Teaching Quality Improvement and Patient Safety: An Alliance for Independent Academic Medical Centers National Initiative III Project 
The Ochsner Journal  2012;12(4):338-343.
Teaching the next generation of physicians requires more than traditional teaching models. The Accreditation Council for Graduate Medical Education's Next Accreditation System places considerable emphasis on developing a learning environment that fosters resident education in quality improvement and patient safety. The goal of this project was to develop a comprehensive and sustainable faculty development program with a focus on teaching quality improvement and patient safety.
A multidisciplinary team representing all stakeholders in graduate medical education developed a validated survey to assess faculty and house officer baseline perceptions of their experience with faculty development opportunities, quality improvement tools and training, and resident participation in quality improvement and patient safety programs at our institution. We then developed a curriculum to address these 3 areas.
Our pilot survey revealed a need for a comprehensive program to teach faculty and residents the art of teaching. Two other areas of need are (1) regular resident participation in quality improvement and patient safety efforts and (2) effective tools for developing skills and habits to analyze practices using quality improvement methods.
Resident and faculty pairs in 17 Ochsner training programs developed and began quality improvement projects while completing the first learning module. Resident and faculty teams also have been working on the patient safety modules and incorporating aspects of patient safety into their individual work environments.
Our team's goal is to develop a sustainable and manageable faculty development program that includes modules addressing quality improvement and patient safety in accordance with Accreditation Council for Graduate Medical Education accreditation requirements.
PMCID: PMC3527861  PMID: 23267260
Faculty development; graduate medical education; performance improvement; quality improvement
24.  Linking a Motivational Interviewing Curriculum to the Chronic Care Model 
Journal of General Internal Medicine  2010;25(Suppl 4):620-626.
Unhealthy lifestyle choices frequently cause or worsen chronic diseases. Many internal medicine residents are inadequately trained to provide effective health behavior counseling, in part, due to prioritization of acute care in the traditional model of medical education and to other systemic barriers to teaching psychosocial aspects of patient care.
To address this gap in training, we developed and piloted a curriculum for a Primary Care Internal Medicine residency program that links a practical form of motivational interviewing (MI) training to the self-management support (SMS) component of the chronic care model.
All 30 primary care residents at Alameda County Medical Center were trained in the curriculum since it was initiated in 2007 during the California Academic Chronic Care Collaborative.
Residents participated in three modules during which the chronic care model was introduced and motivational interviewing skills were linked to the model’s self-management support component. This training was then reinforced in the clinical setting. Case-based interactive instruction, teaching videotapes, group role-plays, faculty demonstration, and observation of resident-patient interactions in the clinical setting were used to teach the curriculum.
A preliminary, qualitative assessment of this curriculum was done from a program standpoint and from the perspective of the learners. Residents reported increased sense of confidence when approaching patients about health behavior change. Faculty directly observed residents during clinical encounters using MI and SMS skills to work more collaboratively with patients and to improve patient readiness for self-management goal setting.
A curriculum that links motivational interviewing skills to the chronic care model’s self-management support component and is reinforced in the clinical setting is feasible to develop and implement. This curriculum may improve residents’ confidence with health behavior counseling and with preparing patients to become active participants in management of their chronic conditions.
PMCID: PMC2940440  PMID: 20737238
chronic care model; self-management support; motivational interviewing; residency training; primary care
25.  Undergraduate medical education in emergency medical care: A nationwide survey at German medical schools 
Since June 2002, revised regulations in Germany have required "Emergency Medical Care" as an interdisciplinary subject, and state that emergency treatment should be of increasing importance within the curriculum. A survey of the current status of undergraduate medical education in emergency medical care establishes the basis for further committee work.
Using a standardized questionnaire, all medical faculties in Germany were asked to answer questions concerning the structure of their curriculum, representation of disciplines, instructors' qualifications, teaching and assessment methods, as well as evaluation procedures.
Data from 35 of the 38 medical schools in Germany were analysed. In 32 of 35 medical faculties, the local Department of Anaesthesiology is responsible for the teaching of emergency medical care; in two faculties, emergency medicine is taught mainly by the Department of Surgery and in another by Internal Medicine. Lectures, seminars and practical training units are scheduled in varying composition at 97% of the locations. Simulation technology is integrated at 60% (n = 21); problem-based learning at 29% (n = 10), e-learning at 3% (n = 1), and internship in ambulance service is mandatory at 11% (n = 4). In terms of assessment methods, multiple-choice exams (15 to 70 questions) are favoured (89%, n = 31), partially supplemented by open questions (31%, n = 11). Some faculties also perform single practical tests (43%, n = 15), objective structured clinical examination (OSCE; 29%, n = 10) or oral examinations (17%, n = 6).
Emergency Medical Care in undergraduate medical education in Germany has a practical orientation, but is very inconsistently structured. The innovative options of simulation technology or state-of-the-art assessment methods are not consistently utilized. Therefore, an exchange of experiences and concepts between faculties and disciplines should be promoted to guarantee a standard level of education in emergency medical care.
PMCID: PMC2689168  PMID: 19435518

Results 1-25 (1265316)