To document teaching evaluation practices in colleges and schools of pharmacy.
A 51-item questionnaire was developed based on the instrument used in a previous study with modifications made to address changes in pharmacy education. An online survey service was used to distribute the electronic questionnaire to the deans of 98 colleges and schools of pharmacy in the United States.
Completed surveys were received from 89 colleges and schools of pharmacy. All colleges/schools administered student evaluations of classroom and experiential teaching. Faculty peer evaluation of classroom teaching was used by 66% of colleges/schools. Use of other evaluation methods had increased over the previous decade, including use of formalized self-appraisal of teaching, review of teaching portfolios, interviews with samples of students, and review by teaching experts. While the majority (55%) of colleges/schools administered classroom teaching evaluations at or near the conclusion of a course, 38% administered them at the midpoint and/or conclusion of a faculty member's teaching within a team-taught course. Completion of an online evaluation form was the most common method used for evaluation of classroom (54%) and experiential teaching (72%).
Teaching evaluation methods used in colleges and schools of pharmacy expanded from 1996 to 2007 to include more evaluation of experiential teaching, review by peers, formalized self-appraisal of teaching, review of teaching portfolios, interviews with samples of students, review by teaching experts, and evaluation by alumni. Procedures for conducting student evaluations of teaching have adapted to address changes in curriculum delivery and technology.
teaching; evaluation; assessment; survey
Since 2007 a portfolio of learning has become a requirement for assessment of postgraduate family medicine training by the Colleges of Medicine of South Africa. A uniform portfolio of learning has been developed and content validity established among the eight postgraduate programmes. The aim of this study was to investigate the portfolio’s acceptability, educational impact, and perceived usefulness for assessment of competence.
Two structured questionnaires of 35 closed and open-ended questions were delivered to 53 family physician supervisors and 48 registrars who had used the portfolio. Categorical and nominal/ordinal data were analysed using simple descriptive statistics. The open-ended questions were analysed with ATLAS.ti software.
Half of registrars did not find the portfolio clear, practical or feasible. Workshops on portfolio use, learning, and supervision were supported, and brief dedicated time daily for reflection and writing. Most supervisors felt the portfolio reflected an accurate picture of learning, but just over half of registrars agreed. While the portfolio helped with reflection on learning, participants were less convinced about how it helped them plan further learning. Supervisors graded most rotations, suggesting understanding the summative aspect, while only 61% of registrars reflected on rotations, suggesting the formative aspects are not yet optimally utilised. Poor feedback, the need for protected academic time, and pressure of service delivery impacting negatively on learning.
This first introduction of a national portfolio for postgraduate training in family medicine in South Africa faces challenges similar to those in other countries. Acceptability of the portfolio relates to a clear purpose and guide, flexible format with tools available in the workplace, and appreciating the changing educational environment from university-based to national assessments. The role of the supervisor in direct observations of the registrar and dedicated educational meetings, giving feedback and support, cannot be overemphasized.
To implement and assess a Web-based patient care portfolio system for development of pharmaceutical care plans by students completing advanced pharmacy practice experiences (APPEs) throughout a statewide preceptor network.
Using a Web database, students in APPEs documented 6 patient cases within 5 disease state categories. Through discussion of the disease states and inclusion of patient information such as problems, desired outcomes, and interventions, a complete pharmaceutical care plan was developed for each patient.
Student interventions were compared by geographical regions to assess continuity of patient care activities by students. Additionally, students completed an evaluation of the portfolio course to provide feedback on the portfolio process. Students documented an average of 1.8 therapeutic interventions per patient case and documented interventions in all geographical regions. The majority of students indicated that the portfolio process improved their ability to develop a pharmaceutical care plan.
The Web-based patient care portfolio process assisted with documentation of compliance with Accreditation Council of Pharmacy Education (ACPE) standards and College of Pharmacy Competency Statements. Students indicated the portfolio process was beneficial in developing skills needed for creating pharmaceutical care plans.
portfolio; pharmaceutical care; advanced pharmacy practice experience; Web; experiential education
Portfolios, widely used in undergraduate and postgraduate medicine, have variable purposes, formats and success. A recent systematic review summarised factors necessary for successful portfolio introduction but there are no studies investigating the views of students inexperienced in portfolio use towards portfolio learning. This study's aim was to survey student views about a prospective Professional and Personal Development (PPD) portfolio.
This was a qualitative, focus group study. All focus groups were taped and transcribed verbatim, and anonymised. The transcripts were analysed inductively, using framework analysis.
Four focus groups were carried out with 32 undergraduate medical students naïve in portfolio use. Three themes relevant to portfolio introduction emerged. The first theme was the need for clear information and support for portfolio introduction, and anxieties about how this could be supported effectively. The second was that students had negative views about reflective learning and whether this could be taught and assessed, believing formal assessment could foster socially acceptable content. The third was that participants revealed little understanding of reflective learning and its potential benefits. Rather portfolios were seen as useful for concrete purposes (e.g., job applications) not intrinsic benefits.
Undergraduate medical students without experience of portfolios are anxious about portfolio introduction. They require support in developing reflective learning skills. Care must be taken to ensure students do not see portfolios as merely yet another assessment hurdle.
Objective. To evaluate pharmacy students' self-assessment skills with an electronic portfolio program using mentor evaluators.
Design. First-year (P1) and second-year (P2) pharmacy students used online portfolios that required self-assessments of specific graded class assignments. Using a rubric, faculty and alumni mentors evaluated students' self-assessments and provided feedback.
Assessment. Eighty-four P1 students, 74 P2 students, and 59 mentors participated in the portfolio program during 2010-2011. Both student groups performed well overall, with only a small number of resubmissions required. P1 students showed significant improvements across semesters for 2 of the self-assessment questions; P2 students' scores did not differ significantly. The P1 scores were significantly higher than P2 scores for 3 questions during spring 2011. Mentors and students had similar levels of agreement with the extent to which students put forth their best effort on the self-assessments.
Conclusion. An electronic portfolio using mentors based inside and outside the school provided students with many opportunities to practice their self-assessment skills. This system represents a useful method of incorporating self-assessments into the curriculum that allows for feedback to be provided to the students.
portfolio; assessment; self-assessment; professional development; mentor
Within the 52 health districts in South Africa, the family physician is seen as the clinical leader within a multi-professional district health team. Family physicians must be competent to meet 90% of the health needs of the communities in their districts. The eight university departments of Family Medicine have identified five unit standards, broken down into 85 training outcomes, for postgraduate training. The family medicine registrar must prove at the end of training that all the required training outcomes have been attained. District health managers must be assured that the family physician is competent to deliver the expected service. The Colleges of Medicine of South Africa (CMSA) require a portfolio to be submitted as part of the uniform assessment of all registrars applying to write the national fellowship examinations. This study aimed to achieve a consensus on the contents and principles of the first national portfolio for use in family medicine training in South Africa.
A workshop held at the WONCA Africa Regional Conference in 2009 explored the purpose and broad contents of the portfolio. The 85 training outcomes, ideas from the WONCA workshop, the literature, and existing portfolios in the various universities were used to develop a questionnaire that was tested for content validity by a panel of 31 experts in family medicine in South Africa, via the Delphi technique in four rounds. Eighty five content items (national learning outcomes) and 27 principles were tested. Consensus was defined as 70% agreement. For those items that the panel thought should be included, they were also asked how to provide evidence for the specific item in the portfolio, and how to assess that evidence.
Consensus was reached on 61 of the 85 national learning outcomes. The panel recommended that 50 be assessed by the portfolio and 11 should not be. No consensus could be reached on the remaining 24 outcomes and these were also omitted from the portfolio. The panel recommended that various types of evidence be included in the portfolio. The panel supported 26 of the 27 principles, but could not reach consensus on whether the portfolio should reflect on the relationship between the supervisor and registrar.
A portfolio was developed and distributed to the eight departments of Family Medicine in South Africa, and the CMSA, to be further tested in implementation.
Current hospital and health-system participation in and the future capacity for experiential education for pharmacy students was investigated.
An online survey of ASHP members identified as U.S. pharmacy directors was conducted to assess their current and future involvement in partnering with colleges and schools to meet the experiential education requirements for doctor of pharmacy students and the current status of the student learning experiences. Questionnaire items examined the factors on which expanded involvement in experiential education would depend, the nature of support provided by colleges and schools, the types of experiences available for students, respondents' perceptions of factors influencing the quality of experiential education, the value of experiential education to the sites, respondents' challenges and concerns about experiential education, and respondents' current capacity and projections for introductory and advanced experiences through 2012.
Data from 549 respondents were analyzed. Most respondents indicated that they had conducted advanced experiences for their 2007 graduates and anticipated that they would continue to do so. Among the top challenges identified regarding advanced experiences were concerns about time to serve and be trained as preceptors and a lack of standardization and coordination among colleges and schools. Hospitals forecasting their future capacity to accommodate students indicated that their projections were highly dependent on the number of pharmacists at their hospitals. Many respondents noted that their capacity projections were tied to their ability to expand clinical services at their hospitals.
A survey of pharmacy directors suggested an ability of U.S. hospitals to conduct advanced experiential education opportunities for pharmacy students through 2012 and to expand introductory experiences.
curriculum; data collection; education; pharmaceutical; pharmacy; institutional; hospital; pharmacy
The Feik School of Pharmacy collaborated with a commercial software development company to create a Web-based e-portfolio system to document student achievement of curricular outcomes and performance in pharmacy practice experiences. The multi-functional system also could be used for experiential site selection and assignment and continuing pharmacy education. The pharmacy school trained students, faculty members, and pharmacist preceptors to use the e-portfolio system. All pharmacy students uploaded the required number of documents and assessments to the program as evidence of achievement of each of the school's curricular outcomes and completion of pharmacy practice experiences.
portfolio; assessment; documentation; curricular outcomes
Portfolio learning enables students to collect evidence of their learning. Component tasks making up a portfolio can be devised that relate directly to intended learning outcomes. Reflective tasks can stimulate students to recognise their own learning needs.
Assessment of portfolios using a rating scale relating to intended learning outcomes offers high content validity.
This study evaluated a reflective portfolio used during a final-year attachment in general practice (family medicine). Students were asked to evaluate the portfolio (which used significant event analysis as a basis for reflection) as a learning tool. The validity and reliability of the portfolio as an assessment tool were also measured.
81 final-year medical students completed reflective significant event analyses as part of a portfolio created during a three-week attachment (clerkship) in general practice (family medicine). As well as two reflective significant event analyses each portfolio contained an audit and a health needs assessment.
Portfolios were marked three times; by the student's GP teacher, the course organiser and by another teacher in the university department of general practice. Inter-rater reliability between pairs of markers was calculated. A questionnaire enabled the students' experience of portfolio learning to be determined.
Benefits to learning from reflective learning were limited. Students said that they thought more about the patients they wrote up in significant event analyses but information as to the nature and effect of this was not forthcoming.
Moderate inter-rater reliability (Spearman's Rho .65) was found between pairs of departmental raters dealing with larger numbers (20 – 60) of portfolios. Inter-rater reliability of marking involving GP tutors who only marked 1 – 3 portfolios was very low.
Students rated highly their mentoring relationship with their GP teacher but found the portfolio tasks time-consuming.
The inter-rater reliability observed in this study should be viewed alongside the high validity afforded by the authenticity of the learning tasks (compared with a sample of a student's learning taken by an exam question). Validity is enhanced by the rating scale which directly connects the grade given with intended learning outcomes.
The moderate inter-rater reliability may be increased if a portfolio is completed over a longer period of time and contains more component pieces of work.
The questionnaire used in this study only accessed limited information about the effect of reflection on students' learning. Qualitative methods of evaluation would determine the students experience in greater depth. It would be useful to evaluate the effects of reflective learning after students have had more time to get used to this unfamiliar method of learning and to overcome any problems in understanding the task.
The Accreditation Council for Graduate Medical Education (ACGME) recommends resident portfolios as 1 method for assessing competence in practice-based learning and improvement. In July 2005, when anesthesiology residents in our department were required to start a portfolio, the residents and their faculty advisors did not readily accept this new requirement. Intensive education efforts addressing the goals and importance of portfolios were undertaken. We hypothesized that these educational efforts improved acceptance of the portfolio and retrospectively audited the portfolio evaluation forms completed by faculty advisors.
Intensive education about the goals and importance of portfolios began in January 2006, including presentations at departmental conferences and one-on-one education sessions. Faculty advisors were instructed to evaluate each resident's portfolio and complete a review form. We retrospectively collected data to determine the percentage of review forms completed by faculty. The portfolio reviews also assessed the percentage of 10 required portfolio components residents had completed.
Portfolio review forms were completed by faculty advisors for 13% (5/38) of residents during the first advisor-advisee meeting in December 2005. Initiation of intensive education efforts significantly improved compliance, with review forms completed for 68% (26/38) of residents in May 2006 (P < .0001) and 95% (36/38) in December 2006 (P < .0001). Residents also significantly improved the completeness of portfolios between May and December of 2006.
Portfolios are considered a best methods technique by the ACGME for evaluation of practice-based learning and improvment. We have found that intensive education about the goals and importance of portfolios can enhance acceptance of this evaluation tool, resulting in improved compliance in completion and evaluation of portfolios.
To examine the impact of implementation of the Accreditation Council for Pharmacy Education's (ACPE's) Standards 2007 on pharmacy students’ preparation for their first advanced pharmacy practice experience (APPE).
The doctor of pharmacy (PharmD) curriculum was altered to include introductory pharmacy practice experiences (IPPE), second-year therapeutics, classroom integration of practice experiences, more biomedical sciences, an electronic portfolio system, life-long learning exercises, and additional content based on Appendix B of Standards 2007. Curricular outcomes and the assessment plan also were revised based on Standards 2007.
To evaluate the impact of these changes to the curriculum, faculty members rated 9 behaviors of students observed during the third week of their first APPE and compared their scores with those of students who were evaluated in 2004 before the curriculum had been revised. Students completing the revised curriculum performed all 9 behaviors more often and had a better average score than students evaluated in 2004.
Curricular revisions implemented to address ACPE Standards 2007 were associated with positive clinical behaviors in students beginning their experiential education.
Standards 2007; accreditation; assessment; curriculum; Accreditation Council for Pharmacy Education; advanced pharmacy practice experience
Objective. To evaluate pharmacy students’ perceived benefits of the portfolio process and to gather suggestions for improving the process.
Methods. A questionnaire was designed and administered to 250 first-, second-, and third-year pharmacy students at the University of Arizona College of Pharmacy.
Results. Although the objectives of the portfolio process were for students to understand the expected outcomes, understand the impact of extracurricular activities on attaining competencies, identify what should be learned, identify their strengths and weaknesses, and modify their approach to learning, overall students perceived the portfolio process as having less than moderate benefit. First-year students wanted more examples of portfolios while second- and third-year students suggested that more time with their advisor would be beneficial.
Conclusions. The portfolio process will continue to be refined and efforts made to improve students’ perceptions of the process as it is intended to develop the self-assessments skills they will need to improve their knowledge and professional skills throughout their pharmacy careers.
survey; portfolio; questionnaire; expected outcomes; assessment
To implement a long-term continuing education course for pharmacy practitioners to acquire competency in and accreditation for conducting collaborative comprehensive medication reviews (CMRs).
A 1½- year curriculum for practicing pharmacists that combined distance learning (using e-learning tools) and face-to-face learning was created. The training consisted of 5 modules: (1) Multidisciplinary Collaboration; (2) Clinical Pharmacy and Pharmacotherapy; (3) Rational Pharmacotherapy; (4) CMR Tools; and (5) Optional Studies.
The curriculum and participants' learning were evaluated using essays and learning diaries. At the end of the course, students submitted portfolios and completed an Internet-based survey instrument. Almost all respondents (92%) indicated their educational needs had been met by the course and 68% indicated they would conduct CMRs in their practice. The most important factors facilitating learning were working with peers and in small groups. Factors preventing learning were mostly related to time constraints.
Comprehensive medication review competencies were established by a 1½- year continuing education curriculum that combined different teaching methods and experiential learning. Peer support was greatly appreciated as a facilitator of learning by course participants.
medication review; continuing education; experiential learning; distance education
To describe and evaluate a new student orientation program designed to lay the foundations for a community of learners.
A weeklong orientation program structured as the first week of an 18-week fall semester was held for the first-professional year class. Each of the activities supported program objectives and developed elements of a community of learners.
Students' reflective portfolios, daily evaluations and final program evaluations provided evidence of development of a community of learners. Positive student observations included the use of technology, a discussion of the curriculum and experiential education, the use of reflective portfolios, and presentations from pharmacy practitioners. Students also appreciated becoming acquainted with the faculty, staff, and their peers in a non-threatening atmosphere. Some of the aspects rated as least helpful were the learning styles exercise, library tour, history of pharmacy session, and the overall length of the orientation.
A model for a new student orientation program that builds the foundations for the development of a community of learning, which is vital to preparing students to provide pharmaceutical care in interdisciplinary teams and become critical thinkers, was successfully established. This model could be implemented at other schools of pharmacy.
Objective. To assess financial, personnel, and curricular characteristics of US pharmacy practice experiential education programs and follow-up on results of a similar survey conducted in 2001.
Methods. Experiential education directors at 118 accredited US pharmacy colleges and schools were invited to participate in a blinded, Web-based survey in 2011. Aggregate responses were analyzed using descriptive statistics and combined with data obtained from the American Association of Colleges of Pharmacy to assess program demographics, faculty and administrative organizational structure, and financial support.
Results. The number of advanced pharmacy practice experience (APPE) sites had increased by 24% for medium, 50% for large, and 55% for very large colleges and schools. Introductory pharmacy practice experience (IPPE) sites outnumbered APPEs twofold. The average experiential education team included an assistant/associate dean (0.4 full-time equivalent [FTE]), a director (1.0 FTE), assistant/associate director (0.5 FTE), coordinator (0.9 FTE), and multiple administrative assistants (1.3 FTE). Most faculty members (63%-75%) were nontenure track and most coordinators (66%) were staff members. Estimated costs to operate an experiential education program represented a small percentage of the overall expense budget of pharmacy colleges and schools.
Conclusion. To match enrollment growth, pharmacy practice experiential education administrators have expanded their teams, reorganized responsibilities, and found methods to improve cost efficiency. These benchmarks will assist experiential education administrators to plan strategically for future changes.
experiential learning; experiential education; advanced pharmacy practice experiences; introductory pharmacy practice experiences; pharmacy faculty; faculty development; administration; organizational structure; financial support; salary; budget
Assessment is such an integral part of the educational system that we rarely reflect on its value and impact. Portfolios have gained in popularity, but much attention has emphasized the end-user and portfolio assessment. Here we focus on the portfolio creator (the student) and examine whether their educational needs are met with such an assessment method. This study aims to investigate how assessment practices influence classroom performance and the learning experience of the student in a graduate education setting. Studied were 33 medical students at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, a program utilizing a portfolio-based system. The students may elect to simultaneously enroll in a Masters program; however, these programs employ traditional letter grades. Thus creating a unique opportunity to assess 25 portfolio only (P) students and 8 portfolio and grade (PG) students concurrently taking a course that counts for both programs. Classroom performance was measured via a comprehensive evaluation where the PG students scored modestly better (median total scores, 72% P vs. 76% PG). Additionally, a survey was conducted to gain insight into student’s perspective on how assessment method impacts the learning experience. The students in the PG group (those receiving a grade) reported increased stress but greater affirmation and self-assurance regarding their knowledge and skill mastery. Incorporation of such affirmation remains a challenge for portfolio-based systems and an area for investigation and improvement.
assessment; portfolio; grading; student performance; student perspective
Objectives. To identify and assess changes made to the Indiana Pharmacy Resident Teaching Certificate program over 10 years to adapt to the growing number and changing needs of pharmacy educators in the next generation.
Design. In 2011, all resident program participants and directors were sent an electronic survey instrument designed to assess the perceived value of each program component.
Assessment. Since 2003, the number of program participants has tripled, and the program has expanded to include additional core requirements and continuing education. Participants generally agreed that the speakers, seminar topics, seminar video recordings, and seminar offerings during the fall semester were program strengths. The program redesign included availability of online registration; a 2-day conference format; retention of those seminars perceived to be most important, according to survey results; implementation of a registration fee; electronic teaching portfolio submission; and establishment of teaching mentors.
Conclusion. With the growing number of residents and residency programs, pharmacy teaching certificate programs must accommodate more participants while continuing to provide quality instruction, faculty mentorship, and opportunities for classroom presentations and student precepting. The Indiana Pharmacy Resident Teaching Certificate program has successfully evolved over the last 10 years to meet these challenges by implementing successful programmatic changes in response to residency program director and past program participant feedback.
teaching certificate program; residency training
Portfolios are increasingly used in undergraduate and postgraduate medical education. Four medical schools have collaborated with an established NHS electronic portfolio provider to develop and implement an authentic professional electronic portfolio for undergraduate students. We hypothesized that using an authentic portfolio would have significant advantages for students, particularly in familiarizing them with the tool many will continue to use for years after graduation. This paper describes the early evaluation of this undergraduate portfolio at two participating medical schools.
To gather data, a questionnaire survey with extensive free text comments was used at School 1, and three focus groups were held at School 2. This paper reports thematic analysis of students’ opinions expressed in the free text comments and focus groups.
Five main themes, common across both schools were identified. These concerned the purpose, use and acceptability of the portfolio, advantages of and barriers to the use of the portfolio, and the impacts on both learning and professional identity.
An authentic portfolio mitigated some of the negative aspects of using a portfolio, and had a positive effect on students’ perception of themselves as becoming past of the profession. However, significant barriers to portfolio use remained, including a lack of understanding of the purpose of a portfolio and a perceived damaging effect on feedback.
Electronic supplementary material
The online version of this article (doi:10.1186/s12909-014-0265-2) contains supplementary material, which is available to authorized users.
Portfolio; Assessment; Feedback; Transition; Enculturation
Formal quality-improvement (QI) projects require that participants are educated in QI methods to provide them with the capability to carry out successful, meaningful work. However, orchestrating a portfolio of projects that addresses the strategic mission of the institution requires an extension of basic QI training to provide the division or business unit with the capacity to successfully develop and manage the portfolio. Advanced Improvement Leadership Systems is a program to help units create a meaningful portfolio. This program, used by the Division of Pediatric General and Thoracic Surgery at Cincinnati Children’s Hospital Medical Center, helped establish a portfolio of targeted QI projects designed to achieve outstanding outcomes at competitive costs in multiple clinical areas aligned with the institution’s strategic goals (improve disease-based outcomes, patient safety, flow, and patient and family experience). These objectives are addressed in an institutional strategic plan built around 5 core areas: Safety, Productivity, Care Coordination and Outcomes, Patient and Family Experience, and Value. By combining the portfolio of QI projects with improvements in the divisional infrastructure, effective improvement efforts were realized throughout the division. In the 9 months following the program, divisional capability resulted in a 16.5% increase (5.7% to 22.2%) of formally trained staff working on 10 QI teams. Concurrently, a leadership team, designed to coordinate projects, remove barriers, and provide technical support, provided the capacity to pursue this ongoing effort. The Advanced Improvement Leadership Systems program increased the Division’s efficiency and effectiveness in pursing the QI mission that is integral at our hospital.
To demonstrate achievement of ability-based outcomes through a structured review of electronic student portfolios in an advanced pharmacy practice experience (APPE) program.
One hundred thirty-eight students produced electronic portfolios containing select work products from APPEs, including a self-assessment reflective essay that demonstrated achievement of course manual-specified ability-based outcomes.
Through portfolio submissions, all students demonstrated the achievement of ability-based outcomes for providing pharmaceutical care, evaluating the literature, and managing the medication use system with patient case reports most frequently submitted. The rubric review of self-reflective essays addressed student learning through APPEs and continuing professional development plans.
The electronic portfolio with reflective essay proved to be a useful vehicle to demonstrate achievement of ability-based outcomes.
electronic reflective portfolio; APPE evaluation; assessment rubric; ability-based outcomes; standards 2007
Objective. To conduct a follow-up survey of curriculum committee chairs in US colleges and schools of pharmacy to describe current committee structures and functions and determine whether changes have occurred over time.
Methods. A descriptive cross-sectional study design using a 30-item survey instrument regarding the structure, function, and charges of curriculum committees was sent to 100 curriculum committee chairs. Several new variables were added to the questionnaire to explore the use of systematic reviews, oversight of experiential education, and the impact of accreditation standards on work focus.
Results. Eighty-five chairs responded. Curriculum committees are on average 1 person larger, less likely to have a student vote, more likely to have formal charges, and more likely to be involved in implementing an outcomes-based curriculum compared with 1994. Committees have shifted their work focus from review of curricular content to curricular revision.
Conclusions. Curriculum committees continue to evolve as they respond to changes in pharmacy education and accreditation standards.
curriculum committee; curriculum; pharmacy education; survey
Interactive pharmacy case studies are an essential component of the pharmacy curriculum. We recently developed an elective course at the Rangel College of Pharmacy in pharmacy case studies for second- and third-year Doctor of Pharmacy students using Second Life® (SL), an interactive three-dimensional virtual environment that simulates the real world. This course explored the use of SL for education and training in pharmacy, emphasizing a case-based approach. Virtual worlds such as SL promote inquiry-based learning and conceptual understanding, and can potentially develop problem-solving skills in pharmacy students. Students were presented ten case scenarios that primarily focused on drug safety and effective communication with patients. Avatars, representing instructors and students, reviewed case scenarios during sessions in a virtual classroom. Individually and in teams, students participated in active-learning activities modeling both the pharmacist’s and patient’s roles. Student performance and learning were assessed based on SL class participation, activities, assignments, and two formal, essay-type online exams in Blackboard 9. Student course-evaluation results indicated favorable perceptions of content and delivery. Student comments included an enhanced appreciation of practical issues in pharmacy practice, flexibility of attendance, and an increased ability to focus on course content. Excellent student participation and performance in weekly active-learning activities translated into positive performance on subsequent formal assessments. Students were actively engaged and exposed to topics pertinent to pharmacy practice that were not covered in the required pharmacy curriculum. The multiple active-learning assignments were successful in increasing students’ knowledge, and provided additional practice in building the communication skills beneficial for students preparing for experiential clinical rotations.
Second Life; virtual worlds; pharmacy case studies; computer simulation; health education; pharmacy education
The Accreditation Council for Graduate Medical Education (ACGME) Learning Portfolio is recommended as a tool to develop and document reflective, practice-based learning and improvement. There is no consensus regarding the appropriate content of a learning portfolio in medical education. Studying lessons selected for inclusion in their learning portfolios by surgical trainees could help identify useful subject matter for this purpose.
Each month, all residents in our surgery residency program submit entries into their individual Surgical Learning and Instructional Portfolio (SLIP). The SLIP entries from July 2008 to 2009 (n = 420) were deidentified and randomized using a random number generator. We conducted a thematic content analysis of 50 random portfolio entries to identify lessons learned. Two independent raters analyzed the “3 lessons learned” portion of the portfolio entries and identified themes and subthemes using the constant comparative method used in grounded theory.
The collaborative coding process resulted in theme saturation after the identification of 7 themes and their subthemes. Themes in decreasing order of frequency included complications, disease epidemiology, disease presentation, surgical management of disease, medical management of disease, operative techniques, and pathophysiology. Junior residents chose to focus on a broad array of foundational topics including disease presentation, epidemiology, and overall management of diseases, whereas postgraduate year-4 (PGY-4) and PGY-5 residents most frequently chose to focus on complications as learning points.
Lessons learned reflect perceived needs of the trainees based on training year. When given a template to follow, junior and senior residents choose to reflect on different subject matter to meet their learning goals.
Little is known about the technical adequacy of portfolios in reporting multiple complex academic and performance-based assessments. We explored, first, the influencing factors on the precision of scoring within a programmatic assessment of student learning outcomes within an integrated clinical placement. Second, the degree to which validity evidence supported interpretation of student scores.
Within generalisability theory, we estimated the contribution that each wanted factor (i.e. student capability) and unwanted factors (e.g. the impact of assessors) made to the variation in portfolio task scores. Relative and absolute standard errors of measurement provided a confidence interval around a pre-determined pass/fail standard for all six tasks. Validity evidence was sought through demonstrating the internal consistency of the portfolio and exploring the relationship of student scores with clinical experience.
The mean portfolio mark for 257 students, across 372 raters, based on six tasks, was 75.56 (SD, 6.68). For a single student on one assessment task, 11% of the variance in scores was due to true differences in student capability. The most significant interaction was context specificity (49%), the tendency for one student to engage with one task and not engage with another task. Rater subjectivity was 29%. An absolute standard error of measurement of 4.74%, gave a 95% CI of +/- 9.30%, and a 68% CI of +/- 4.74% around a pass/fail score of 57%. Construct validity was supported by demonstration of an assessment framework, the internal consistency of the portfolio tasks, and higher scores for students who did the clinical placement later in the academic year.
A portfolio designed as a programmatic assessment of an integrated clinical placement has sufficient evidence of validity to support a specific interpretation of student scores around passing a clinical placement. It has modest precision in assessing students’ achievement of a competency standard. There were identifiable areas for reducing measurement error and providing more certainty around decision-making. Reducing the measurement error would require engaging with the student body on the value of the tasks, more focussed academic and clinical supervisor training, and revisiting the rubric of the assessment in the light of feedback.
Portfolio; Programmatic assessment; Competency-based assessment; Clinical placement; Longitudinal integrated clerkship; Generalisability theory; Reliability; Validity
Objective. To determine how colleges and schools of pharmacy interpreted the Accreditation Council for Pharmacy Education’s (ACPE’s) Standards 2007 definitions for core advanced pharmacy practice experiences (APPEs), and how they differentiated community and institutional practice activities for introductory pharmacy practice experiences (IPPEs) and APPEs.
Methods. A cross-sectional, qualitative, thematic analysis was done of survey data obtained from experiential education directors in US colleges and schools of pharmacy. Open-ended responses to invited descriptions of the 4 core APPEs were analyzed using grounded theory to determine common themes. Type of college or school of pharmacy (private vs public) and size of program were compared.
Results. Seventy-one schools (72%) with active APPE programs at the time of the survey responded. Lack of strong frequent themes describing specific activities for the acute care/general medicine core APPE indicated that most respondents agreed on the setting (hospital or inpatient) but the student experience remained highly variable. Themes were relatively consistent between public and private institutions, but there were differences across programs of varying size.
Conclusion. Inconsistencies existed in how colleges and schools of pharmacy defined the core APPEs as required by ACPE. More specific descriptions of core APPEs would help to standardize the core practice experiences across institutions and provide an opportunity for quality benchmarking.
experiential education; advanced pharmacy practice experiences; acute care; general medicine; ambulatory care; community pharmacy; health systems; qualitative research