Does mentoring new peer reviewers improve review quality? A randomized trial

doi:10.1186/1472-6920-12-83

BMC Med Educ. 2012; 12: 83.

Published online 2012 August 28. doi: 10.1186/1472-6920-12-83

PMCID: PMC3494517

Does mentoring new peer reviewers improve review quality? A randomized trial

Debra Houry,¹ Steven Green,² and Michael Callaham³

¹Department of Emergency Medicine, Emory University School of Medicine, Atlanta, GA, USA

²Department of Emergency Medicine, Loma Linda School of Medicine, Los Angeles, CA, USA

³Department of Emergency Medicine, University of California, San Francisco, CA, USA

Corresponding author.

Debra Houry: Dhoury/at/aol.com; Steven Green: steve/at/viridissimo.com; Michael Callaham: Michael.Callaham/at/ucsf.edu

Received April 7, 2012; Accepted August 16, 2012.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

This article has been cited by other articles in PMC.

Abstract

Background

Prior efforts to train medical journal peer reviewers have not improved subsequent review quality, although such interventions were general and brief. We hypothesized that a manuscript-specific and more extended intervention pairing new reviewers with high-quality senior reviewers as mentors would improve subsequent review quality.

Methods

Over a four-year period we randomly assigned all new reviewers for Annals of Emergency Medicine to receive our standard written informational materials alone, or these materials plus a new mentoring intervention. For this program we paired new reviewers with a high-quality senior reviewer for each of their first three manuscript reviews, and asked mentees to discuss their review with their mentor by email or phone. We then compared the quality of subsequent reviews between the control and intervention groups, using linear mixed effects models of the slopes of review quality scores over time.

Results

We studied 490 manuscript reviews, with similar baseline characteristics between the 24 mentees who completed the trial and the 22 control reviewers. Mean quality scores for the first 3 reviews on our 1 to 5 point scale were similar between control and mentee groups (3.4 versus 3.5), as were slopes of change of review scores (-0.229 versus -0.549) and all other secondary measures of reviewer performance.

Conclusions

A structured training intervention of pairing newly recruited medical journal peer reviewers with senior reviewer mentors did not improve the quality of their subsequent reviews.

Keywords: Mentoring, Peer review, Scientific publication, Critical analysis, Journal peer reviewer

Articles from BMC Medical Education are provided here courtesy of
BioMed Central