Document Detail


A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.
MedLine Citation:
PMID:  21179459     Owner:  NLM     Status:  MEDLINE    
Abstract/OtherAbstract:
BACKGROUND: This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree.
METHODOLOGY/PRINCIPAL FINDINGS: Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2=.34, mean Cohen's Kappa=.17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed.
CONCLUSIONS/SIGNIFICANCE: Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
Authors:
Lutz Bornmann; Rüdiger Mutz; Hans-Dieter Daniel
Related Documents :
1436459 - Infarction in the territory of the anterior inferior cerebellar artery: report of five ...
19917959 - Granulomatous dermatitis with pseudoxanthoma elasticum-like changes: report of a case i...
2404659 - Speech-prompt catatonia: a case report and review of the literature.
19050869 - Atlanto-axial segmentation defects and os odontoideum in two male siblings with opsismo...
24314899 - Ondansetron and the risk of cardiac arrhythmias: a systematic review and postmarketing ...
21179459 - A reliability-generalization study of journal peer reviews: a multilevel meta-analysis ...
24650169 - Pacemaker lead malpositioning led to subsequent ischemic strokes despite antiplatelet a...
14768469 - Evaluation of percutaneous endoscopic gastrostomy in elderly patients with silicosis an...
3769939 - Radiological aspects of aspergillosis in the paranasal sinuses.
Publication Detail:
Type:  Journal Article; Meta-Analysis     Date:  2010-12-14
Journal Detail:
Title:  PloS one     Volume:  5     ISSN:  1932-6203     ISO Abbreviation:  PLoS ONE     Publication Date:  2010  
Date Detail:
Created Date:  2010-12-23     Completed Date:  2011-07-05     Revised Date:  2013-07-03    
Medline Journal Info:
Nlm Unique ID:  101285081     Medline TA:  PLoS One     Country:  United States    
Other Details:
Languages:  eng     Pagination:  e14331     Citation Subset:  IM    
Affiliation:
Max Planck Society, Munich, Germany. bornmann@gv.mpg.de
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms
Descriptor/Qualifier:
Bayes Theorem
Humans
Models, Statistical
Observer Variation
Peer Review*
Periodicals as Topic
Publications
Regression Analysis
Reproducibility of Results
Research Design
Comments/Corrections

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine


Previous Document:  Resveratrol induces growth arrest and apoptosis through activation of FOXO transcription factors in ...
Next Document:  Shift work is not associated with high blood pressure or prevalence of hypertension.