Name
Title | ||
|---|---|---|
Feeling lucky?: multi-armed bandits for ordering judgements in pooling-based evaluation. |
Abstract | ||
|---|---|---|
Evaluation is crucial in Information Retrieval. The Cranfield paradigm allows reproducible system evaluation by fostering the construction of standard and reusable benchmarks. Each benchmark or test collection comprises a set of queries, a collection of documents and a set of relevance judgements. Relevance judgements are often done by humans and thus expensive to obtain. Consequently, relevance judgements are customarily incomplete. Only a subset of the collection, the pool, is judged for relevance. In TREC-like campaigns, the pool is formed by the top retrieved documents supplied by systems participating in a certain evaluation task. With multiple retrieval systems contributing to the pool, an exploration/exploitation trade-off arises naturally. Exploiting effective systems could find more relevant documents, but exploring weaker systems might also be valuable for the overall judgement process. In this paper, we cast document judging as a multi-armed bandit problem. This formal modelling leads to theoretically grounded adjudication strategies that improve over the state of the art. We show that simple instantiations of multi-armed bandit models are superior to all previous adjudication strategies.
|
Year | DOI | Venue |
|---|---|---|
2016 | 10.1145/2851613.2851692 | SAC 2016: Symposium on Applied Computing
Pisa
Italy
April, 2016 |
Field | DocType | ISBN |
Information retrieval,Computer science,Pooling,System evaluation,Judgement,Adjudication,Artificial intelligence,Feeling,Machine learning | Conference | 978-1-4503-3739-7 |
Citations | PageRank | References |
12 | 0.57 | 13 |
Authors | ||
3 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| David E. Losada | 1 | 326 | 40.63 |
| Javier Parapar | 2 | 188 | 25.91 |
| Alvaro Barreiro | 3 | 226 | 22.42 |