Paper Info
Title
Just an artifact? The concordance between peer review and bibliometrics in economics and statistics in the Italian research assessment exercise
Abstract
During the Italian research assessment exercise (2004-2010), the governmental agency (ANVUR) in charge of its realization performed an experiment on the concordance between peer review and bibliometrics at an individual article level. The computed concordances were at most weak for science, technology, engineering, and mathematics. The only exception was the moderate concordance found for the area of economics and statistics. In this paper, the disclosed raw data of the experiment are used to shed light on the anomalous results obtained for economics and statistics. In particular, the data permit us to document that the protocol of the experiment adopted for economics and statistics was different from the one used in the other areas. Indeed, in economics and statistics the same group of scholars developed the bibliometric ranking of journals for evaluating articles, managing peer reviews and forming the consensus groups for deciding the final scores of articles after having received the referee's reports. This paper shows that the highest level of concordance in economics and statistics was an artifact mainly due to the role played by consensus groups in boosting the agreement between bibliometrics and peer review.
Year
DOI
Venue
2022
10.1162/qss_a_00172
QUANTITATIVE SCIENCE STUDIES
Keywords
DocType
Volume
bibliometrics, economics and statistics, Italy, peer review, replication study, research assessment exercise
Journal
3
Issue
Citations 
PageRank 
1
0
0.34
References 
Authors
0
2
Name
Order
Citations
PageRank
Alberto Baccini100.68
Giuseppe De Nicolao273876.26