PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of cureusLink to Publisher's site
 
Cureus. 2017 January; 9(1): e992.
PMCID: PMC5323028

Resident Development via Progress Testing and Test-Marking: An Innovation and Program Evaluation

Monitoring Editor: Alexander Muacevic and John R Adler

Abstract

Introduction

Since 2008, the McMaster University Royal College Emergency Medicine residency training program has run practice Short Answer Question (SAQ) examinations to help residents test their knowledge and gain practice in answering exam-style questions. However, marking this type of SAQ exam is time-consuming.

Methods

To help address this problem, we require that senior residents help mark at least one exam per year alongside faculty members. Examinees’ identities are kept anonymous by assigning a random number to each resident, which is only decoded after marking. Aggregation of marks is done by faculty only. The senior residents and faculty members all share sequential marking of each question. Each question is reviewed, and exemplar “best practice” answers are discussed. As novel/unusual answers appear, instantaneous fact-checking (via textbooks, or the internet) and discussions occur allowing for real-time modification to the answer keys as needed.

Results

A total of 22 out of 37 residents (post graduate year 1 to post graduate year 5 (PGY1 to PGY5)) participated in a recent program evaluation focus group. This evaluation showed that residents feel quite positive about this process. With the anonymization process, residents do not object to their colleagues seeing and marking their answers. Senior residents have found this process informative and have felt that this process helps them gain insight into better “examsmanship.”

Conclusions

Involving residents in marking short-answer exams is acceptable and perceived as useful experience for improving exam-taking skills. More studies of similar innovations would be required to determine to what extent this may be the case.

Keywords: progress testing, residency education, program evaluation

Introduction

Background

Examinations are still markers of final success for medical learners. Even as we progress to a competency-based framework for postgraduate medical education, it is unlikely that we will entirely abandon this age-old tradition. Additionally, the role of examinations as a part of an assessment portfolio for determining and fostering competence can be better integrated. Testing in medical education has been well-described previously [1-3], with the most recent body of literature focusing on test-enhanced learning (TEL). The theory behind TEL is that well-spaced and frequent testing is beneficial for learning, since each test is a chance for further retrieval practice and theoretically strengthens the connections between concepts [2-3].

Progress testing is a form of formative testing that harnesses the properties of test-enhanced learning and manifests them into a specific practice. As a result, progress testing has been utilized in medical education, both as a method of encouraging retrieval practice and also as a way of allowing learners to gain external insight into their performance compared to their peers, possibly spurring further engagement and interest in studies via social motivation. At McMaster University, we have had a long tradition of frequent progress testing for undergraduate students [4-5], but in postgraduate learning, progress testing is less often used. In most Royal College of Physicians and Surgeons of Canada (RCPSC) Emergency Medicine (EM) residency programs, trainees usually only engage in annual in-training examinations (e.g. the American Board of Emergency Medicine in-training exam, ABEM, or the new Canadian in-training exam, CITE). Few programs engage in additional progress testing, largely because of two factors: 1) Exams are difficult to construct, 2) Exams are also difficult to mark. At McMaster University’s RCPSC EM program, we attempted to create a system that might overcome these barriers and allow our trainees to harness the educational benefits of progress testing and retrieval practice.

Purpose

Since 2008, McMaster Emergency Medicine has run quarterly, locally-developed progress tests that are aligned with our curricular blocks (dubbed “Block Exams”). The purpose of these progress tests is to encourage continuous studying and review of the curriculum topics and to provide opportunity for retrieval practice. Moreover, these exams consist of short answer questions (SAQs), mirroring the style of both the national in-training exam (CITE) and the terminal examination given by the RCPSC.

Each block, half of our week’s academic time is allocated for a test (75 minutes), usually consisting of seven to 10 SAQs. The answer key is generated by the exam question author and peer reviewed by our associate program director (KS) before use. Questions are aligned to the main topics covered in the preceding three months’ academic time, although there is usually one question per block exam that is meant to force more remote information retrieval.

When this program began in 2009, our program had only 20 residents; however, in the subsequent years our program nearly doubled in size (37 residents at present). As such, in five short years the human resources required to mark and provide good quality feedback to the learners on their practice tests significantly increased.

Materials and methods

Description of the innovation

To help address this problem, our program decided to pilot and then implement a program that involved senior residents in exam marking. Initially this program was volunteer-based; however, as residents participated and saw merit, our program shifted to make this a mandatory experience for all senior residents each year. Currently, we require senior residents to help mark at least one exam per year alongside faculty members. Figure Figure11 depicts the steps taken to ensure the anonymity of residents’ identities and create a safe procedure for the peer-marking.

Figure 1
Steps taken to protect the residents’ identities

The marking experience

The senior residents and faculty members all share sequential marking of each question. Each question is reviewed and exemplar “best practice” answers are discussed. If novel or unusual answers occur, instantaneous fact-checking (via textbooks, or the internet) and discussions occur allowing for real-time modification to the answer keys as needed.

Evaluation methods

We conducted a focus group-based program evaluation about this program. We received an exemption from the Hamilton Integrated Research Ethics Board for this program development initiative, and we did receive informed verbal consent from our participants. After receiving an exemption from our institutional review board, we conducted a series of focus groups with our residents using a non-MD (SAL) interviewer. These sessions were recorded and subsequently transcribed and then thematically analyzed with two members of our team (an MD (TC) and a non-MD (SAL)). We used an interpretive descriptive technique [6] for our analysis of these transcripts couched within a realist evaluation framework [7], as we were interested in the pragmatic interaction of our residents and the current educational program. All names were redacted to preserve anonymity.

Results

Twenty-two out of 37 (60%) residents from our local program participated in our focus groups. Residents were interviewed in their postgraduate year cohorts. Participants ranged from postgraduate year (PGY) 1 to 5. Generally, the progress testing was viewed positively. Our residents noted that the progress testing was beneficial in providing a source of motivation to study and a measure of performance that could be trended overtime and compared to their PGY cohort. Details of the themes, subthemes, codes and quotes that emerged in the focus groups are displayed in Tables Tables11--22.

Table 1
Resident perceptions of progress testing
Table 2
Resident perceptions of the peer-marking experience

Residents discussed the usefulness and barriers for progress testing. The grades from the block exams provided a trend for professional progression in the residency program, which afforded the resident the opportunity for personal reflection on their performance to date. Residents also used grades as a barometer to compare their progress with other peers in the same cohort. The block exams also helped engage residents to identify learning gaps and validate the knowledge and skills acquired during residency. Residents saw block exams as a motivator, specifically for encouraging the review of recently learned content, ensuring performance conformity among their peers, and managing their own identity as a physician in training. Even though residents viewed block exams as a source of stress, they perceived the block exams as a very helpful motivator to review the learned material. Our learners noted variable motivation levels in the various years of residency, with increasing motivation as residents approached their final examinations. Junior residents noted that they were more motivated to learn by clinical encounters and assessments.

We also inquired about their perceptions about the peer marking experience. With the anonymization process, residents did not object to their colleagues seeing and marking their answers. Factors that facilitated acceptability were the use of a standardized marking key and that most residents felt that their identities were protected by the anonymization system (e.g. numbered identifiers). Only one senior resident thought he could identify a peer’s handwriting. One junior resident especially felt that senior residents provided better feedback, since they perceived that these near-peers could better explain how to improve. Furthermore, junior residents perceived senior residents as more reliable and having the ability to provide a more thorough review. Senior residents found that participating as a marker of these exams was informative, many of them remarking that this process helped them gain insight into better exam-taking skills (“examsmanship”). For instance, senior residents felt that having exposure to their peers’ exams during marking would widen their perspective in terms of how certain questions are answered. Overall, the residents agreed that there is educational value in marking exams because it creates an opportunity to discuss and elaborate on the answers with other peers and faculty members.  

 

Discussion

In the age of competency-based medical education, it is likely that single, final, high-stakes exams may become de-emphasized as we integrate more continuous monitoring of performance. That said, it is likely that there may still be some component of testing in most jurisdictions, as it is simply a very efficient way to ensure knowledge acquisition or retention. The merits of test-enhanced learning have also shown that simply integrating more progress testing may be of benefit for our mid-stream learners who are still developing knowledge and skills.

As such, strategies such as this innovation may allow educators to both more efficiently mentor junior residents through exam-taking skills (i.e. expanding the pool of possible teachers by including senior residents), and also to help prepare senior residents to assume more educational/administrative roles. Not only this, but our program evaluation suggests that substantial adjunctive learning may occur in the senior residents themselves when they are forced to reflect on the answers of others. This may allow senior residents to better calibrate and combat misalignment between self-perceptions of competence and actual efficacy (e.g. the Dunning-Kruger effect) [8]. Indeed, this is aligned with previous work, which has shown that peer assessment of written [9] exams has been useful in various settings. Our findings align with theories of peer learning [10], which may be why junior residents so highly value their senior residents’ feedback.

Limitations

One of the main limitations of our present study is that the evaluation of our innovation is restricted to a single-centre and a single specialty. We used a constructivist epistemology; however, we are not purporting to attempt to generalize our findings. That said, we feel that the attitudes towards senior-resident development via marking peer resident exams is akin to other literature around near peer mentoring, and hence, we feel that when considering other literature our findings are in alignment with previous work. As such, we feel that there is likely some level of transferability that can be inferred from our findings. Other limitations would be our sample size and convenience sampling. It is possible that we did not reach full data saturation; we did, however, feel our data reached a level of sufficiency [11].

Future directions

Annually in Canada, the RCPSC EM residents have a short answer question based in-training examination for our specialty. Instead of a centralized marking centre, each residency program is charged with marking their own examinations and then reporting the data. That said, it is often a difficult task to find enough core faculty members who are able to donate their time. Our work suggests there may be opportunity to allow senior residents to mark peer residents' exams during this process, allowing us to expand our evaluation of a peer-based test-marking experience.

Conclusions

Including residents in the marking of local progress tests can be educationally beneficial for all parties involved. Peer-marking has been deemed to be acceptable or desired by the residents who receive the peer-feedback and therefore should not pose a barrier to implementation. It may be a useful adjunctive educational experience for centers lacking the core faculty to sustain a regular, in-house progress testing system.

Acknowledgments

Our team would like to thank the residents and attending physicians who have participated in helping to make our progress test program so successful. Thank you in particular to Ms. Melissa Hymers and Ms. Teresa Vallera for their tireless work on this project and all the rest of our residency-related programming. We would also like to thank Dr. Kelly Van Diepen who has taken over the leadership of this worthwhile endeavour.

Notes

The content published in Cureus is the result of clinical experience and/or research by independent individuals or organizations. Cureus is not responsible for the scientific accuracy or reliability of data or conclusions published herein. All content published within Cureus is intended only for educational, research and reference purposes. Additionally, articles published within Cureus should not be deemed a suitable substitute for the advice of a qualified health care professional. Do not disregard or avoid professional medical advice due to content published within Cureus.

The authors have declared that no competing interests exist.

Human Ethics

Hamilton Integrated Research Ethics Board issued approval N/A. This project was granted an exemption from Hamilton Integrated Research Ethics Board as it was deemed a program development initiative.

Animal Ethics

Animal subjects: This study did not involve animal subjects or tissue.

References

1. The effects of assessments and examinations on the learning of medical students. Newble DI, Jaeger K. Med Educ. 1983;17:165–171. [PubMed]
2. Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Karpicke JD, Roediger HL. J Exp Psychol. 2007;33:704–719. [PubMed]
3. Test-enhanced learning in medical education. Larsen DP, Butler AC, Roediger HL. Med Educ. 2008;42:959–966. [PubMed]
4. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Norman G, Neville A, Blake JM, Mueller B. Med Teach. 2010;32:496–499. [PubMed]
5. Introducing progress testing in McMaster University’s problem-based medical curriculum: psychometric properties and effect on learning. Blake JM, Norman GR, Keane DR, Mueller CB, Cunnington J, Didyk N. http://www.ncbi.nlm.nih.gov/pubmed/?term=9125989. Acad Med. 1996;71:1002–1007. [PubMed]
6. The analytic challenge in interpretive description. Thorne S, Reimer Kirkham S, O’Flynn-Magee K. http://sites.ualberta.ca/~iiqm/backissues/3_1/pdf/thorneetal.pdf Int J Qual Methods. 2004;3.1:1–11.
7. Pawson R, Tilley N. London: Sage; 1997. Realistic Evaluation.
8. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Kruger J, Dunning D. J Pers Soc Psychol. 1999;77:1121–1134. [PubMed]
9. The effectiveness and reliability of peer-marking in first-year medical students. English R, Brookes ST, Avery K, Blazeby JM, Ben-Shlomo Y. Med Educ. 2006;40:965–972. [PubMed]
10. Trends in peer learning. Topping KJ. Educ Psychol. 2005;25:631–645.
11. Verification strategies for establishing reliability and validity in qualitative research. Morse JM, Barrett M, Mayan M, Olson K, Spiers J. Int J Qual Methods. 2002;1:13–22.

Articles from Cureus are provided here courtesy of Cureus Inc.