Name
Title | ||
|---|---|---|
Investigating Passage-level Relevance and Its Role in Document-level Relevance Judgment |
Abstract | ||
|---|---|---|
The understanding of the process of relevance judgment helps to inspire the design of retrieval models. Traditional retrieval models usually estimate relevance based on document-level signals. Recent works consider a more fine-grain, passage-level relevance information, which can further enhance retrieval performance. However, it lacks a detailed analysis of how passage-level relevance signals determine or influence the relevance judgment of the whole document. To investigate the role of passage-level relevance in the document-level relevance judgment, we construct an ad-hoc retrieval dataset with both passage-level and document-level relevance labels. A thorough analysis reveals that: 1) there is a strong correlation between the document-level relevance and the fractions of irrelevant passages to highly relevant passages; 2) the position, length and query similarity of passages play different roles in the determination of document-level relevance; 3) The sequential passage-level relevance within a document is a potential indicator for the document-level relevance. Based on the relationship between passage-level and document-level relevance, we also show that utilizing passage-level relevance signals can improve existing document ranking models. This study helps us better understand how users perceive relevance for a document and inspire the designing of novel ranking models leveraging fine-grain, passage-level relevance signals.
|
Year | DOI | Venue |
|---|---|---|
2019 | 10.1145/3331184.3331233 | Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval |
Keywords | Field | DocType |
passage-level relevance aggregation, relevance judgment, relevance model | Information retrieval,Computer science | Conference |
ISBN | Citations | PageRank |
978-1-4503-6172-9 | 4 | 0.40 |
References | Authors | |
0 | 5 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Zhijing Wu | 1 | 25 | 4.15 |
| Jiaxin Mao | 2 | 164 | 26.30 |
| Yiqun Liu | 3 | 1592 | 136.51 |
| Min Zhang | 4 | 1658 | 134.93 |
| Shaoping Ma | 5 | 1544 | 126.00 |