Name
Abstract | ||
|---|---|---|
Document-level relevance judgments are a major component in the calculation of effectiveness metrics. Collecting high-quality judgments is therefore a critical step in information retrieval evaluation. However, the nature of and the assumptions underlying relevance judgment collection have not received much attention. In particular, relevance judgments are typically collected for each document in isolation, although users read each document in the context of other documents. In this work, we aim to investigate the nature of relevance judgment collection. We collect relevance labels in both isolated and conditional setting, and ask for judgments in various dimensions of relevance as well as overall relevance. Then we compare the relevance metrics based on various types of judgments with other metrics of quality such as user preference. Our analyses illuminate how these settings for judgment collection affect the quality and the characteristics of the judgments. We also find that the metrics based on conditional judgments show higher correlation with user preference than isolated judgments. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1145/2566486.2568015 | WWW |
Keywords | Field | DocType |
relevance judgment collection,overall relevance,conditional judgment,relevance judgment,dimensional relevance judgment,high-quality judgment,document-level relevance judgment,user preference,relevance metrics,reusable serp-level evaluation,effectiveness metrics,relevance label,reusability,evaluation | Data mining,Ask price,Information retrieval,Computer science | Conference |
Citations | PageRank | References |
2 | 0.37 | 26 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Peter B. Golbus | 1 | 27 | 3.01 |
| Imed Zitouni | 2 | 612 | 46.39 |
| Jin Young Kim | 3 | 497 | 81.76 |
| Ahmed Hassan | 4 | 943 | 57.64 |
| Fernando Diaz | 5 | 1985 | 97.72 |