We evaluated a newly designed electronic portfolio (e-Portfolio) that provided quantitative evaluation of surgical skills. Medical students at the University of Seville used the e-Portfolio on a voluntary basis for evaluation of their performance in undergraduate surgical subjects.
Our new web-based e-Portfolio was designed to evaluate surgical practical knowledge and skills targets. Students recorded each activity on a form, attached evidence, and added their reflections. Students self-assessed their practical knowledge using qualitative criteria (yes/no), and graded their skills according to complexity (basic/advanced) and participation (observer/assistant/independent). A numerical value was assigned to each activity, and the values of all activities were summated to obtain the total score. The application automatically displayed quantitative feedback. We performed qualitative evaluation of the perceived usefulness of the e-Portfolio and quantitative evaluation of the targets achieved.
Thirty-seven of 112 students (33%) used the e-Portfolio, of which 87% reported that they understood the methodology of the portfolio. All students reported an improved understanding of their learning objectives resulting from the numerical visualization of progress, all students reported that the quantitative feedback encouraged their learning, and 79% of students felt that their teachers were more available because they were using the e-Portfolio. Only 51.3% of students reported that the reflective aspects of learning were useful. Individual students achieved a maximum of 65% of the total targets and 87% of the skills targets. The mean total score was 345 ± 38 points. For basic skills, 92% of students achieved the maximum score for participation as an independent operator, and all achieved the maximum scores for participation as an observer and assistant. For complex skills, 62% of students achieved the maximum score for participation as an independent operator, and 98% achieved the maximum scores for participation as an observer or assistant.
Medical students reported that use of an electronic portfolio that provided quantitative feedback on their progress was useful when the number and complexity of targets were appropriate, but not when the portfolio offered only formative evaluations based on reflection. Students felt that use of the e-Portfolio guided their learning process by indicating knowledge gaps to themselves and teachers.
Electronic portfolio; Surgical subjects; Self-guided learning; Self-assessment; Evaluative portfolio
Portfolios, widely used in undergraduate and postgraduate medicine, have variable purposes, formats and success. A recent systematic review summarised factors necessary for successful portfolio introduction but there are no studies investigating the views of students inexperienced in portfolio use towards portfolio learning. This study's aim was to survey student views about a prospective Professional and Personal Development (PPD) portfolio.
This was a qualitative, focus group study. All focus groups were taped and transcribed verbatim, and anonymised. The transcripts were analysed inductively, using framework analysis.
Four focus groups were carried out with 32 undergraduate medical students naïve in portfolio use. Three themes relevant to portfolio introduction emerged. The first theme was the need for clear information and support for portfolio introduction, and anxieties about how this could be supported effectively. The second was that students had negative views about reflective learning and whether this could be taught and assessed, believing formal assessment could foster socially acceptable content. The third was that participants revealed little understanding of reflective learning and its potential benefits. Rather portfolios were seen as useful for concrete purposes (e.g., job applications) not intrinsic benefits.
Undergraduate medical students without experience of portfolios are anxious about portfolio introduction. They require support in developing reflective learning skills. Care must be taken to ensure students do not see portfolios as merely yet another assessment hurdle.
Portfolio learning enables students to collect evidence of their learning. Component tasks making up a portfolio can be devised that relate directly to intended learning outcomes. Reflective tasks can stimulate students to recognise their own learning needs.
Assessment of portfolios using a rating scale relating to intended learning outcomes offers high content validity.
This study evaluated a reflective portfolio used during a final-year attachment in general practice (family medicine). Students were asked to evaluate the portfolio (which used significant event analysis as a basis for reflection) as a learning tool. The validity and reliability of the portfolio as an assessment tool were also measured.
81 final-year medical students completed reflective significant event analyses as part of a portfolio created during a three-week attachment (clerkship) in general practice (family medicine). As well as two reflective significant event analyses each portfolio contained an audit and a health needs assessment.
Portfolios were marked three times; by the student's GP teacher, the course organiser and by another teacher in the university department of general practice. Inter-rater reliability between pairs of markers was calculated. A questionnaire enabled the students' experience of portfolio learning to be determined.
Benefits to learning from reflective learning were limited. Students said that they thought more about the patients they wrote up in significant event analyses but information as to the nature and effect of this was not forthcoming.
Moderate inter-rater reliability (Spearman's Rho .65) was found between pairs of departmental raters dealing with larger numbers (20 – 60) of portfolios. Inter-rater reliability of marking involving GP tutors who only marked 1 – 3 portfolios was very low.
Students rated highly their mentoring relationship with their GP teacher but found the portfolio tasks time-consuming.
The inter-rater reliability observed in this study should be viewed alongside the high validity afforded by the authenticity of the learning tasks (compared with a sample of a student's learning taken by an exam question). Validity is enhanced by the rating scale which directly connects the grade given with intended learning outcomes.
The moderate inter-rater reliability may be increased if a portfolio is completed over a longer period of time and contains more component pieces of work.
The questionnaire used in this study only accessed limited information about the effect of reflection on students' learning. Qualitative methods of evaluation would determine the students experience in greater depth. It would be useful to evaluate the effects of reflective learning after students have had more time to get used to this unfamiliar method of learning and to overcome any problems in understanding the task.
Objective. To evaluate pharmacy students' self-assessment skills with an electronic portfolio program using mentor evaluators.
Design. First-year (P1) and second-year (P2) pharmacy students used online portfolios that required self-assessments of specific graded class assignments. Using a rubric, faculty and alumni mentors evaluated students' self-assessments and provided feedback.
Assessment. Eighty-four P1 students, 74 P2 students, and 59 mentors participated in the portfolio program during 2010-2011. Both student groups performed well overall, with only a small number of resubmissions required. P1 students showed significant improvements across semesters for 2 of the self-assessment questions; P2 students' scores did not differ significantly. The P1 scores were significantly higher than P2 scores for 3 questions during spring 2011. Mentors and students had similar levels of agreement with the extent to which students put forth their best effort on the self-assessments.
Conclusion. An electronic portfolio using mentors based inside and outside the school provided students with many opportunities to practice their self-assessment skills. This system represents a useful method of incorporating self-assessments into the curriculum that allows for feedback to be provided to the students.
portfolio; assessment; self-assessment; professional development; mentor
To implement and assess a Web-based patient care portfolio system for development of pharmaceutical care plans by students completing advanced pharmacy practice experiences (APPEs) throughout a statewide preceptor network.
Using a Web database, students in APPEs documented 6 patient cases within 5 disease state categories. Through discussion of the disease states and inclusion of patient information such as problems, desired outcomes, and interventions, a complete pharmaceutical care plan was developed for each patient.
Student interventions were compared by geographical regions to assess continuity of patient care activities by students. Additionally, students completed an evaluation of the portfolio course to provide feedback on the portfolio process. Students documented an average of 1.8 therapeutic interventions per patient case and documented interventions in all geographical regions. The majority of students indicated that the portfolio process improved their ability to develop a pharmaceutical care plan.
The Web-based patient care portfolio process assisted with documentation of compliance with Accreditation Council of Pharmacy Education (ACPE) standards and College of Pharmacy Competency Statements. Students indicated the portfolio process was beneficial in developing skills needed for creating pharmaceutical care plans.
portfolio; pharmaceutical care; advanced pharmacy practice experience; Web; experiential education
Objective. To develop and implement a flexible-credit elective course to empower student pharmacists to develop lifelong leadership skills and provide teaching practice opportunities for graduate students.
Design. An elective course focusing on leadership development for second- and third-year doctor of pharmacy (PharmD) students was designed and taught by 4 graduate students under the mentorship of 2 faculty members. Student pharmacists could enroll in a 1-, 2-, or 3-credit-hour version of the course.
Assessment. Attainment of course objectives was measured using student pharmacist reflection papers and continuing professional development portfolios. Additionally, self-assessments of graduate students and faculty members delivering the course were conducted. In their responses on course evaluations, student pharmacists indicated they found the course a valuable learning experience. Graduate students found course development to be challenging but useful in developing faculty skills.
Conclusion. This flexible-credit elective course taught by graduate students was an innovative way to offer formal leadership instruction using limited college resources.
leadership; graduate students; faculty development; pharmacy education; elective course
Portfolios are increasingly used in undergraduate and postgraduate medical education. Four medical schools have collaborated with an established NHS electronic portfolio provider to develop and implement an authentic professional electronic portfolio for undergraduate students. We hypothesized that using an authentic portfolio would have significant advantages for students, particularly in familiarizing them with the tool many will continue to use for years after graduation. This paper describes the early evaluation of this undergraduate portfolio at two participating medical schools.
To gather data, a questionnaire survey with extensive free text comments was used at School 1, and three focus groups were held at School 2. This paper reports thematic analysis of students’ opinions expressed in the free text comments and focus groups.
Five main themes, common across both schools were identified. These concerned the purpose, use and acceptability of the portfolio, advantages of and barriers to the use of the portfolio, and the impacts on both learning and professional identity.
An authentic portfolio mitigated some of the negative aspects of using a portfolio, and had a positive effect on students’ perception of themselves as becoming past of the profession. However, significant barriers to portfolio use remained, including a lack of understanding of the purpose of a portfolio and a perceived damaging effect on feedback.
Electronic supplementary material
The online version of this article (doi:10.1186/s12909-014-0265-2) contains supplementary material, which is available to authorized users.
Portfolio; Assessment; Feedback; Transition; Enculturation
Objectives. To identify the prevalence of portfolio use in US pharmacy programs, common components of portfolios, and advantages of and limitations to using portfolios.
Methods. A cross-sectional electronic survey instrument was sent to experiential coordinators at US colleges and schools of pharmacy to collect data on portfolio content, methods, training and resource requirements, and benefits and challenges of portfolio use.
Results. Most colleges and schools of pharmacy (61.8%) use portfolios in experiential courses and the majority (67.1%) formally assess them, but there is wide variation regarding content and assessment. The majority of respondents used student portfolios as a formative evaluation primarily in the experiential curriculum.
Conclusions. Although most colleges and schools of pharmacy have a portfolio system in place, few are using them to fulfill accreditation requirements. Colleges and schools need to carefully examine the intended purpose of their portfolio system and follow-through with implementation and maintenance of a system that meets their goals.
portfolio; assessment; evaluation; competency achievement; pharmacy practice experiences; pharmacy education
Objectives. To design, integrate, and assess the effectiveness of an introductory pharmacy practice experience intended to redefine first-year student pharmacists’ views on aging and medication use through their work with a healthy, community-based older-adult population.
Design. All students (N = 273) completed live skills training in an 8-hour boot camp provided during orientation week. Teams were assigned an independently living senior partner, completed 10 visits and reflections, and documented health-related information using an electronic portfolio (e-portfolio).
Assessment. As determined by pre- and post-experience survey instruments, students gained significant confidence in 7 skill areas related to communication, medication interviews, involving the partner in health care, and applying patient-care skills. Student reflections, in-class presentations, and e-portfolios documented that personal attitudes toward seniors changed over time. Senior partners enjoyed mentoring and interacting with students and many experienced health improvements as a result of the interaction.
Conclusions. The model for partnering first-year student pharmacists with community-based older adults improved students’ skills and fostered their connections to pharmacist roles and growth as person-centered providers.
geriatrics; senior partner; senior mentor; introductory pharmacy practice experience
To determine whether the integration of an automated electronic clinical portfolio into clinical clerkships can improve the quality of feedback given to students on their patient write-ups and the quality of students’ write-ups.
The authors conducted a single-blinded, randomized controlled study of an electronic clinical portfolio that automatically collects all students’ clinical notes and notifies their teachers (attending and resident physicians) via e-mail. Third-year medical students were randomized to use the electronic portfolio or traditional paper means. Teachers in the portfolio group provided feedback directly on the student’s write-up using a web-based application. Teachers in the control group provided feedback directly on the student’s write-up by writing in the margins of the paper. Outcomes were teacher and student assessment of the frequency and quality of feedback on write-ups, expert assessment of the quality of student write-ups at the end of the clerkship, and participant assessment of the value of the electronic portfolio system.
Teachers reported giving more frequent and detailed feedback using the portfolio system (p = 0.01). Seventy percent of students who used the portfolio system, versus 39% of students in the control group (p = 0.001), reported receiving feedback on more than half of their write-ups. Write-ups of portfolio students were rated of similar quality to write-ups of control students. Teachers and students agreed that the system was a valuable teaching tool and easy to use.
An electronic clinical portfolio that automatically collects students’ clinical notes is associated with improved teacher feedback on write-ups and similar quality of write-ups.
portfolio; feedback; medical education
Little is known about the technical adequacy of portfolios in reporting multiple complex academic and performance-based assessments. We explored, first, the influencing factors on the precision of scoring within a programmatic assessment of student learning outcomes within an integrated clinical placement. Second, the degree to which validity evidence supported interpretation of student scores.
Within generalisability theory, we estimated the contribution that each wanted factor (i.e. student capability) and unwanted factors (e.g. the impact of assessors) made to the variation in portfolio task scores. Relative and absolute standard errors of measurement provided a confidence interval around a pre-determined pass/fail standard for all six tasks. Validity evidence was sought through demonstrating the internal consistency of the portfolio and exploring the relationship of student scores with clinical experience.
The mean portfolio mark for 257 students, across 372 raters, based on six tasks, was 75.56 (SD, 6.68). For a single student on one assessment task, 11% of the variance in scores was due to true differences in student capability. The most significant interaction was context specificity (49%), the tendency for one student to engage with one task and not engage with another task. Rater subjectivity was 29%. An absolute standard error of measurement of 4.74%, gave a 95% CI of +/- 9.30%, and a 68% CI of +/- 4.74% around a pass/fail score of 57%. Construct validity was supported by demonstration of an assessment framework, the internal consistency of the portfolio tasks, and higher scores for students who did the clinical placement later in the academic year.
A portfolio designed as a programmatic assessment of an integrated clinical placement has sufficient evidence of validity to support a specific interpretation of student scores around passing a clinical placement. It has modest precision in assessing students’ achievement of a competency standard. There were identifiable areas for reducing measurement error and providing more certainty around decision-making. Reducing the measurement error would require engaging with the student body on the value of the tasks, more focussed academic and clinical supervisor training, and revisiting the rubric of the assessment in the light of feedback.
Portfolio; Programmatic assessment; Competency-based assessment; Clinical placement; Longitudinal integrated clerkship; Generalisability theory; Reliability; Validity
To evaluate the knowledge of basic life support (BLS) among students and health providers in Medicine, Pharmacy, Dentistry, and Allied Health Science Colleges at Qassim University.
A cross sectional study was performed using an online BLS survey that was completed by 139 individuals.
Ninety-three responders were medical students, 7 were medical interns, 6 were dental students, 7 were pharmacy students, 11 were medical science students and 15 were clinical practitioners. No responder scored 100% on the BLS survey. Only two out of the 139 responders (1.4%) scored 90–99%. Both of these individuals were fifth year medical students. Six responders (4.3%) scored 80–89%. Of these, 5 were fifth year medical students, and one was fourth-year medical student. Eleven responders (7.9%) scored 70–79%. Of these, eight were fifth year medical students, two were medical interns and one was a pharmacist. Twenty-three responders (16.5%) scored 60–69%. Of these, 11 were fifth year medical students, 1 was a fourth-year medical student, 3 were medical interns, 2 were medical science students, 1 was a dentistry student, and 5 were pharmacists. Twenty-eight responders (20.1%) scored 50–59%. Of these, 11 were fifth year medical students, 3 were fourth-year medical students, 1 was a third-year medical student, 1 was a second-year medical student, 2 were first-year medical students, 1 was a pharmacy student, 3 were dental students, 1 was a allied health science student, 2 were doctors, and 3 were pharmacists. The remaining 69 responders (49.6%) scored less than 50%.
Knowledge of BLS among medicine, pharmacy, dentistry, and allied health science students and health providers at Qassim University is poor and needs to be improved. We suggest that inclusion of a BLS course in the undergraduate curriculum with regular reassessment would increase awareness and application of this valuable life-saving skill set.
To examine the impact of implementation of the Accreditation Council for Pharmacy Education's (ACPE's) Standards 2007 on pharmacy students’ preparation for their first advanced pharmacy practice experience (APPE).
The doctor of pharmacy (PharmD) curriculum was altered to include introductory pharmacy practice experiences (IPPE), second-year therapeutics, classroom integration of practice experiences, more biomedical sciences, an electronic portfolio system, life-long learning exercises, and additional content based on Appendix B of Standards 2007. Curricular outcomes and the assessment plan also were revised based on Standards 2007.
To evaluate the impact of these changes to the curriculum, faculty members rated 9 behaviors of students observed during the third week of their first APPE and compared their scores with those of students who were evaluated in 2004 before the curriculum had been revised. Students completing the revised curriculum performed all 9 behaviors more often and had a better average score than students evaluated in 2004.
Curricular revisions implemented to address ACPE Standards 2007 were associated with positive clinical behaviors in students beginning their experiential education.
Standards 2007; accreditation; assessment; curriculum; Accreditation Council for Pharmacy Education; advanced pharmacy practice experience
Assessment is such an integral part of the educational system that we rarely reflect on its value and impact. Portfolios have gained in popularity, but much attention has emphasized the end-user and portfolio assessment. Here we focus on the portfolio creator (the student) and examine whether their educational needs are met with such an assessment method. This study aims to investigate how assessment practices influence classroom performance and the learning experience of the student in a graduate education setting. Studied were 33 medical students at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, a program utilizing a portfolio-based system. The students may elect to simultaneously enroll in a Masters program; however, these programs employ traditional letter grades. Thus creating a unique opportunity to assess 25 portfolio only (P) students and 8 portfolio and grade (PG) students concurrently taking a course that counts for both programs. Classroom performance was measured via a comprehensive evaluation where the PG students scored modestly better (median total scores, 72% P vs. 76% PG). Additionally, a survey was conducted to gain insight into student’s perspective on how assessment method impacts the learning experience. The students in the PG group (those receiving a grade) reported increased stress but greater affirmation and self-assurance regarding their knowledge and skill mastery. Incorporation of such affirmation remains a challenge for portfolio-based systems and an area for investigation and improvement.
assessment; portfolio; grading; student performance; student perspective
Objective. To determine whether a 2-year continuing professional development (CPD) training program improved first-year (P1) and second-year (P2) pharmacy students’ ability to write SMART (specific, measurable, achievable, relevant, and timed) learning objectives.
Design. First-year students completed live or online CPD training, including creating portfolios and writing SMART objectives prior to their summer introductory pharmacy practice experience (IPPE). In year 2, P1 and P2 students were included. SMART learning objectives were graded and analyzed.
Assessment. On several objectives, the 2011 P1 students (n = 130) scored higher than did the P2 cohort (n = 105). In 2011, P2 students outscored their own performance in 2010. In 2011, P1 students who had been trained in online modules performed the same as did live-session trainees with respect to SMART objectives.
Conclusion. With focused online or live training, students are capable of incorporating principles of CPD by writing SMART learning objectives.
continuing professional development; learning objectives; introductory pharmacy practice experience curriculum
Interactive pharmacy case studies are an essential component of the pharmacy curriculum. We recently developed an elective course at the Rangel College of Pharmacy in pharmacy case studies for second- and third-year Doctor of Pharmacy students using Second Life® (SL), an interactive three-dimensional virtual environment that simulates the real world. This course explored the use of SL for education and training in pharmacy, emphasizing a case-based approach. Virtual worlds such as SL promote inquiry-based learning and conceptual understanding, and can potentially develop problem-solving skills in pharmacy students. Students were presented ten case scenarios that primarily focused on drug safety and effective communication with patients. Avatars, representing instructors and students, reviewed case scenarios during sessions in a virtual classroom. Individually and in teams, students participated in active-learning activities modeling both the pharmacist’s and patient’s roles. Student performance and learning were assessed based on SL class participation, activities, assignments, and two formal, essay-type online exams in Blackboard 9. Student course-evaluation results indicated favorable perceptions of content and delivery. Student comments included an enhanced appreciation of practical issues in pharmacy practice, flexibility of attendance, and an increased ability to focus on course content. Excellent student participation and performance in weekly active-learning activities translated into positive performance on subsequent formal assessments. Students were actively engaged and exposed to topics pertinent to pharmacy practice that were not covered in the required pharmacy curriculum. The multiple active-learning assignments were successful in increasing students’ knowledge, and provided additional practice in building the communication skills beneficial for students preparing for experiential clinical rotations.
Second Life; virtual worlds; pharmacy case studies; computer simulation; health education; pharmacy education
Ultrasonography (US) at the medical student level is developing. As clinical skills and simulation centers expand, US equipment miniaturizes, and more students are exposed to ultrasound; a digital portfolio comprised of US images and videos may be useful in demonstrating experience and possibly competency.
Medical students participated in US curricula consisting of didactics and hands-on training. From 1 July 2006 to 30 June 2008, student images and videos were saved. Total images and videos were evaluated and catalogued.
A total of 10,074 images and 1,227 videos were saved during the 2-year period. For the academic year 2006 to 2007, 159 medical students obtained 3,641 of the images (84.9%) and 270 of the videos (86.0%). First year students obtained 778 images and 20 videos; second year students, 1,174 images and 64 videos; third year students, 211 images and 20 videos; and fourth year students, 1,478 images and 166 videos.
For the academic year 2007 to 2008, 222 medical students obtained 4,340 images (75%) and 619 videos (67.8%). First year students obtained 624 images and 109 videos; second year students, 555 images and 81 videos; third year students, 132 images and 14 videos; and fourth year students, 3,029 images and 415 videos.
The ultrasound digital portfolio allows medical students to collate and document their ultrasound experience. Currently, there is no requirement for ultrasound training, documentation of competency, or minimum numbers of US exams for medical education. The ultrasound digital portfolio may be a useful tool in documenting ultrasound proficiency.
Medical education; Digital portfolio; Ultrasound images
Objective. To assess the health-related quality of life (HRQoL) of student pharmacists and explore factors related to HRQoL outcomes of student pharmacists in a doctor of pharmacy (PharmD) program at a public university.
Methods. A survey instrument was administered to all student pharmacists in a PharmD program at a public university to evaluate differences and factors related to the HRQoL outcomes of first-year (P1), second-year (P2), third-year (P3), and fourth-year (P4) student pharmacists in the college. The survey instrument included attitudes and academic-related self-perception, a 12-item short form health survey, and personal information components.
Results. There were 304 students (68.6%) who completed the survey instrument. The average health state classification measure and mental health component scale (MCS-12) scores were significantly higher for P4 students when compared with the P1through P3 students. There was no difference observed in the physical component scale (PCS-12) scores among each of the 4 class years. Significant negative impact on HRQoL outcomes was observed in students with higher levels of confusion about how they should study (scale lack of regulation) and concern about not being negatively perceived by others (self-defeating ego orientation), while school satisfaction increased HRQoL outcomes (SF-6D, p<0.001; MCS-12, p=0.013). A greater desire to be judged capable (self-enhancing ego-orientation) and career satisfaction were positively associated with the PCS-12 scores (p<0.05).
Conclusion. Factors associated with the HRQoL of student pharmacists were confusion regarding how to study, ego orientation, satisfaction with the chosen college of pharmacy, and career satisfaction. First-year through third-year student pharmacists had lower HRQoL as compared with P4 students and the US general population. Support programs may be helpful for students to maintain or improve their mental and overall health.
health-related quality of life; student pharmacists; perceived self-efficacy; ego-orientation
Since 2007 a portfolio of learning has become a requirement for assessment of postgraduate family medicine training by the Colleges of Medicine of South Africa. A uniform portfolio of learning has been developed and content validity established among the eight postgraduate programmes. The aim of this study was to investigate the portfolio’s acceptability, educational impact, and perceived usefulness for assessment of competence.
Two structured questionnaires of 35 closed and open-ended questions were delivered to 53 family physician supervisors and 48 registrars who had used the portfolio. Categorical and nominal/ordinal data were analysed using simple descriptive statistics. The open-ended questions were analysed with ATLAS.ti software.
Half of registrars did not find the portfolio clear, practical or feasible. Workshops on portfolio use, learning, and supervision were supported, and brief dedicated time daily for reflection and writing. Most supervisors felt the portfolio reflected an accurate picture of learning, but just over half of registrars agreed. While the portfolio helped with reflection on learning, participants were less convinced about how it helped them plan further learning. Supervisors graded most rotations, suggesting understanding the summative aspect, while only 61% of registrars reflected on rotations, suggesting the formative aspects are not yet optimally utilised. Poor feedback, the need for protected academic time, and pressure of service delivery impacting negatively on learning.
This first introduction of a national portfolio for postgraduate training in family medicine in South Africa faces challenges similar to those in other countries. Acceptability of the portfolio relates to a clear purpose and guide, flexible format with tools available in the workplace, and appreciating the changing educational environment from university-based to national assessments. The role of the supervisor in direct observations of the registrar and dedicated educational meetings, giving feedback and support, cannot be overemphasized.
Objective. To incrementally create and embed biannual integrated knowledge and skills examinations into final examinations of the pharmacy practice courses offered in the first 3 years of the pharmacy curriculum that would account for 10% of each course’s final course grade.
Design. An ad hoc integrated examination committee was formed and tasked with addressing 4 key questions. Integrated examination committees for the first, second, and third years of the curriculum were established and tasked with identifying the most pertinent skills and knowledge-based content from each required course in the curriculum, developing measurable objectives addressing the pertinent content, and creating or revising multiple-choice and performance-based questions derived from integrated examination objectives. An Integrated Examination Review Committee evaluated all test questions, objectives, and student performance on each question, and revised the objectives and questions as needed for the following year’s iteration. Eight performance objectives for the examinations were measured.
Assessment. All 8 performance objectives were achieved. Sixty-four percent of the college’s faculty members participated in the integrated examination process, improving the quality of the examination. The incremental development and implementation of the examinations over a 3-year period minimized the burden on faculty time while engaging them in the process. Student understanding of expectations for knowledge and skill retention in the curriculum also improved.
Conclusions. Development of biannual integrated examinations in the first 3 years of the classroom curriculum enhanced the college’s culture of assessment and addressed accreditation guidelines for formative and summative assessment of students’ knowledge and skills. The course will continue to be refined each year.
assessment; evaluation; integrated; progress exams; milestone exams
The Accreditation Council for Graduate Medical Education (ACGME) recommends resident portfolios as 1 method for assessing competence in practice-based learning and improvement. In July 2005, when anesthesiology residents in our department were required to start a portfolio, the residents and their faculty advisors did not readily accept this new requirement. Intensive education efforts addressing the goals and importance of portfolios were undertaken. We hypothesized that these educational efforts improved acceptance of the portfolio and retrospectively audited the portfolio evaluation forms completed by faculty advisors.
Intensive education about the goals and importance of portfolios began in January 2006, including presentations at departmental conferences and one-on-one education sessions. Faculty advisors were instructed to evaluate each resident's portfolio and complete a review form. We retrospectively collected data to determine the percentage of review forms completed by faculty. The portfolio reviews also assessed the percentage of 10 required portfolio components residents had completed.
Portfolio review forms were completed by faculty advisors for 13% (5/38) of residents during the first advisor-advisee meeting in December 2005. Initiation of intensive education efforts significantly improved compliance, with review forms completed for 68% (26/38) of residents in May 2006 (P < .0001) and 95% (36/38) in December 2006 (P < .0001). Residents also significantly improved the completeness of portfolios between May and December of 2006.
Portfolios are considered a best methods technique by the ACGME for evaluation of practice-based learning and improvment. We have found that intensive education about the goals and importance of portfolios can enhance acceptance of this evaluation tool, resulting in improved compliance in completion and evaluation of portfolios.
To implement and assess the effectiveness of a program to teach pharmacy students the importance of taking personal responsibility for their health.
The My First Patient Program was created and lectures were incorporated into an existing first-year course to introduce the concepts of health beliefs, behavior modification, stress management, substance abuse, and nutrition. Each student received a comprehensive health screening and health risk assessment which they used to develop a personal health portfolio and identify strategies to attain and/or maintain their personal health goals.
Student learning was assessed through written assignments and student reflections, follow-up surveys, and course evaluations. Students' attainment of health goals and their ability to identify their personal health status illustrated the positive impact of the program.
This program serves as a model for colleges and schools of pharmacy and for other health professions in the instruction of health promotion, disease prevention, and behavior modification.
health promotion; disease prevention; behavior change
Objectives. To develop, pilot test, and evaluate a continuous professional development (CPD) process for first-year pharmacy (P1) students.
Design. Students and faculty members were introduced to the important elements of the CPD process via a live training program. Students completed the year-long 4-step CPD cycle by identifying a learning objective, creating a plan, completing the learning activity, evaluating their learning outcome, documenting each step, and meeting with their faculty advisor for feedback and advice.
Assessment. Seventy-five first-year students (100%) successfully completed the CPD process during the 2009-2010 academic year. The students spent an average of 7 hours (range 2 to 20 hours) on the CPD process. The majority of faculty members (83%) completing the survey instrument found the process valuable for the students and would like to see the program continued.
Conclusion. Integrating a CPD requirement for students in a college or school of pharmacy is feasible and valuable to students’ developing life-long learning skills. Effective and frequent training of faculty members and students is a key element in the CPD process.
continuous professional development; continuing pharmacy education; pharmacy students
To implement a required capstone experience in research for pharmacy students, assess course outcomes, and solicit mentors' and students' opinions regarding the structure and efficacy of the course.
Fourth-year pharmacy students chose a research project, selected a mentor, and completed a 5-week capstone advanced pharmacy practice experience (APPE), during which they wrote a research paper and presented their research at a poster session.
Eighty students completed the capstone experience in 2008-2009 and 56 faculty and non-faculty pharmacists served as mentors. Based on their responses on a course evaluation, the students' experience with their mentor and course instructor were positive. Thirty-one mentors completed a survey on which they indicated their overall support of the capstone project, but wanted their role to be better defined and felt the students needed to have additional training in statistics, survey question design, and the IRB process before completing the APPE.
The capstone APPE was perceived by students and mentors as a positive learning experience that allowed the student to take information from the curriculum and apply it to a real-world situation. Additional research is needed to determine whether pharmacy students will use the research skills acquired in their future careers.
advanced pharmacy practice experience; research; curriculum
Objective. To determine the amount and type of feedback needed to improve pharmacy students’ problem-solving skills using team-based learning (TBL) and a problem-solving rubric.
Methods. A problem-solving rubric was developed to assess the ability of pharmacy students’ to prioritize, organize, and defend the best and alternative options on TBL cases The study involved 3 groups of pharmacy students: second-year students in a cardiology class who received no feedback (control group), third-year students in an endocrinology class who received written feedback only, and third-year students in an endocrinology class who received written and verbal feedback. Students worked in groups on all TBL cases except the first and last one (beginning and end of course), which students completed independently as it served as a pretest and posttest.
Results. Significant improvements were seen in the ability of the third-year students who received verbal and written feedback to prioritize the information presented in the case and in their total score on the problem-solving rubric.
Conclusion. Providing pharmacy students with written and verbal explanations may help them improve their problem-solving skills overall. During verbal feedback, faculty members can provide more examples of how to improve and can field questions if needed.
team-based learning; problem-solving; rubrics; assessment; feedback