Document Detail

Longitudinal trends in the performance of scientific peer reviewers.
MedLine Citation:
PMID:  21074894     Owner:  NLM     Status:  In-Data-Review    
STUDY OBJECTIVE: We characterize changes in review quality by individual peer reviewers over time.
METHODS: Editors at a specialty journal in the top 11% of Institute of Scientific Information journals rated the quality of every review, using a validated 5-point quality score. Linear mixed-effect models were used to analyze rating changes over time, calculating within-reviewer trends plus predicted slope of change in score for each reviewer. Reviewers at this journal have been shown comparable to those at other journals.
RESULTS: Reviews (14,808) were performed by 1,499 reviewers and rated by 84 editors during the 14-year study. Ninety-two percent of reviewers demonstrated very slow but steady deterioration in their scores (mean -0.04 points [-0.8%] per year). Rate of deterioration was unrelated to duration of reviewing but moderately correlated with mean reviewer quality score (R=0.52). The mean score of each reviewer's first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%. Scores of the group stayed constant over time despite deterioration because newly recruited reviewers initially had higher mean quality scores than their predecessors.
CONCLUSION: This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.
Michael Callaham; Charles McCulloch
Related Documents :
285724 - Disseminated histoplasmosis presenting with oral lesions - report of a case.
21328634 - Patient views and experiences of making adverse drug reaction reports to the yellow car...
2383054 - Synchronous caverno-spongious thrombosis and fournier's gangrene.
7116214 - Macular coccidioidomycosis.
16698494 - Keratitis after intracorneal ring segment insertion for keratoconus.
10932744 - A review on the use of microorganisms as probiotics.
Publication Detail:
Type:  Journal Article     Date:  2010-11-12
Journal Detail:
Title:  Annals of emergency medicine     Volume:  57     ISSN:  1097-6760     ISO Abbreviation:  Ann Emerg Med     Publication Date:  2011 Feb 
Date Detail:
Created Date:  2011-01-21     Completed Date:  -     Revised Date:  -    
Medline Journal Info:
Nlm Unique ID:  8002646     Medline TA:  Ann Emerg Med     Country:  United States    
Other Details:
Languages:  eng     Pagination:  141-8     Citation Subset:  AIM; IM    
Copyright Information:
Copyright © 2010 American College of Emergency Physicians. Published by Mosby, Inc. All rights reserved.
Annals of Emergency Medicine, Dallas, TX; Department of Emergency Medicine, University of California, San Francisco, San Francisco, CA.
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Previous Document:  Effect of oligogalacturonides on root length, extracellular alkalinization and O(2)(-)-accumulation ...
Next Document:  The relationship between hand area and hand contamination.