Name
Abstract | ||
|---|---|---|
In the traditional evaluation of information retrieval systems, assessors are asked to determine the relevance of a document on a graded scale, independent of any other documents. Such judgments are absolute judgments. Learning to rank brings some new challenges to this traditional evaluation methodology, especially regarding absolute relevance judgments. Recently preferences judgments have been investigated as an alternative. Instead of assigning a relevance grade to a document, an assessor looks at a pair of pages and judges which one is better. In this paper, we generalize pairwise preference judgments to relative judgments. We formulate the problem of relative judgments in a formal way and then propose a new strategy called Select-the-Best-Ones to solve the problem. Through user studies, we compare our proposed method with a pairwise preference judgment method and an absolute judgment method. The results indicate that users can distinguish by about one more relevance degree when using relative methods than when using the absolute method. Consequently, the relative methods generate 15-30% more document pairs for learning to rank. Compared to the pairwise method, our proposed method increases the agreement among assessors from 95% to 99%, while halving the labeling time and the number of discordant pairs to experts' judgments. |
Year | DOI | Venue |
|---|---|---|
2011 | 10.1016/j.ipm.2010.02.005 | Inf. Process. Manage. |
Keywords | Field | DocType |
relative method,pairwise preference judgment method,relevance degree,preference judgment,absolute method,pairwise method,relative judgment,relative relevance,absolute judgment,human judgment,absolute relevance judgment,absolute judgment method,relative relevance judgment,information retrieval system,learning to rank | Data mining,Learning to rank,Pairwise comparison,Information retrieval,Computer science,Human judgment,User studies | Journal |
Volume | Issue | ISSN |
47 | 1 | Information Processing and Management |
Citations | PageRank | References |
9 | 0.89 | 17 |
Authors | ||
7 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Ruihua Song | 1 | 1138 | 59.33 |
| Qingwei Guo | 2 | 19 | 3.09 |
| Ruochi Zhang | 3 | 9 | 0.89 |
| Guomao Xin | 4 | 107 | 6.85 |
| Ji-Rong Wen | 5 | 4431 | 265.98 |
| Yong Yu | 6 | 7637 | 380.66 |
| Hsiao-Wuen Hon | 7 | 1719 | 354.37 |