Paper Info
Title
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Abstract
ABSTRACTLeaderboards are a ubiquitous part of modern research in applied machine learning. By design, they sort entries into some linear order, where the top-scoring entry is recognized as the "state of the art" (SOTA). Due to the rapid progress being made today, particularly with neural models, the top entry in a leaderboard is replaced with some regularity. These are touted as improvements in the state of the art. Such pronouncements, however, are almost never qualified with significance testing. In the context of the MS MARCO document ranking leaderboard, we pose a specific question: How do we know if a run is significantly better than the current SOTA? Against the backdrop of recent IR debates on scale types, our study proposes an evaluation framework that explicitly treats certain outcomes as distinct and avoids aggregating them into a single-point metric. Empirical analysis of SOTA runs from the MS MARCO document ranking leaderboard reveals insights about how one run can be "significantly better" than another that are obscured by the current official evaluation metric ([email protected]).
Year
DOI
Venue
2021
10.1145/3404835.3463034
Research and Development in Information Retrieval
Keywords
DocType
Citations 
Significance Testing, Evaluation Metrics
Conference
2
PageRank 
References 
Authors
0.40
0
5
Name
Order
Citations
PageRank
Jimmy Lin14800376.93
daniel filipe barros campos2288.61
Nick Craswell33942279.60
Bhaskar Mitra444126.26
Emine Yilmaz5145996.39