| A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. | |
| | |
MedLine Citation:
|
PMID: 21179459 Owner: NLM Status: In-Process |
Abstract/OtherAbstract:
|
BACKGROUND: This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. METHODOLOGY/PRINCIPAL FINDINGS: Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2=.34, mean Cohen's Kappa=.17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. CONCLUSIONS/SIGNIFICANCE: Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system). |
| | |
Authors:
|
Lutz Bornmann; Rüdiger Mutz; Hans-Dieter Daniel |
Related Documents
:
|
19346939 - The microcirculation as a clinical concept: work in progress. 19050869 - Atlanto-axial segmentation defects and os odontoideum in two male siblings with opsismo... 20836899 - Bench-to-bedside review: molecular pharmacology and clinical use of inert gases in anes... 7633589 - The ball-and-socket ankle: a case history and literature review. 19917959 - Granulomatous dermatitis with pseudoxanthoma elasticum-like changes: report of a case i... 1436459 - Infarction in the territory of the anterior inferior cerebellar artery: report of five ... 2825149 - Malignant fibrous histiocytoma in childhood: a report of two cases and review of the li... 16317419 - Male erectile dysfunction following spinal cord injury: a systematic review. 23056979 - Lymphoma presenting as cancer of the glans penis: a case report. |
Publication Detail:
|
Type: Journal Article Date: 2010-12-14 |
Journal Detail:
|
Title: PloS one Volume: 5 ISSN: 1932-6203 ISO Abbreviation: PLoS ONE Publication Date: 2010 |
Date Detail:
|
Created Date: 2010-12-23 Completed Date: - Revised Date: - |
Medline Journal Info:
|
Nlm Unique ID: 101285081 Medline TA: PLoS One Country: United States |
Other Details:
|
Languages: eng Pagination: e14331 Citation Subset: IM |
Affiliation:
|
Max Planck Society, Munich, Germany. bornmann@gv.mpg.de |
Export Citation:
|
APA/MLA Format Download EndNote Download BibTex |
| MeSH Terms | |
Descriptor/Qualifier:
|
|
From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine
Previous Document: Resveratrol induces growth arrest and apoptosis through activation of FOXO transcription factors in ...
Next Document: Shift work is not associated with high blood pressure or prevalence of hypertension.