Paper Info
Title
Anomalies in the peer-review system: A case study of the journal of High Energy Physics
Abstract
Peer-review system has long been relied upon for bringing quality research to the notice of the scientific community and also preventing flawed research from creeping into the literature. The need for the peer-review system has often been debated as in numerous cases it has failed in its task and in most of these cases editors and the reviewers were thought to be responsible for not being able to correctly judge the quality of the work. This raises a question “Can the peer- review system be improved?” Since editors and reviewers are the most important pillars of a reviewing system, we in this work, attempt to address a related question - given the editing/reviewing history of the editors or reviewers “can we identify the under-performing ones?”, with citations received by the edited/reviewed papers being used as proxy for quantifying performance. We term such reviewers and editors as anomalous and we believe identifying and removing them shall improve the performance of the peer-review system. Using a massive dataset of Journal of High Energy Physics (JHEP) consisting of 29k submitted between 1997 and 2015 with 95 editors and 4035 reviewers and the review history, we identify several factors which point to anomalous behavior of referees and editors. We observe that for the editors, (i) frequent assignments, (ii) selecting reviewers from only a small set of reviewers and (iii) allocating specific referees to the submissions of specific authors are anomalous and leads to under-performance. Similarly for reviewers we observe that (i) frequent assignments, (ii) delay in sending reports, (iii) high proportion of acceptance, (iv) reviewing on diverse topics and (v) assignments from a single or a very small set of editors, are indicative of anomalous behavior. Using the above features and k-nearest neighbor anomaly detection technique, we are also able to exactly point out the anomalous editors and reviewers and show that they are largely responsible for cases where the peer-review system fails. We believe that our findings can be helpful in removing anomalous referees and editors from the system and, thereby, improve its performance.
Year
DOI
Venue
2016
10.1145/2983323.2983675
ACM International Conference on Information and Knowledge Management
Keywords
DocType
Volume
Peer-review system,Editor,Reviewer,Citation
Conference
abs/1608.04875
Citations 
PageRank 
References 
4
0.45
1
Authors
4
Name
Order
Citations
PageRank
Sandipan Sikdar1504.86
Matteo Marsili214917.65
Niloy Ganguly31306121.03
Animesh Mukherjee439262.78