Name
Abstract | ||
|---|---|---|
We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1145/2348283.2348403 | SIGIR |
Keywords | Field | DocType |
test collection,uncertainty-aware query selection model,trec test collection,associated relevance assessment,subsequent query,information retrieval test collection,queries iteratively,query selection,reliable performance ranking,ir system,adaptive query selection algorithm,relevance judgment,information retrieval,search engine,optimization problem | Query optimization,Web search query,Data mining,Ranking,Information retrieval,Query expansion,Computer science,Selection algorithm,Sargable,Web query classification,Ranking (information retrieval) | Conference |
Citations | PageRank | References |
5 | 0.41 | 17 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Mehdi Hosseini | 1 | 54 | 3.77 |
| Ingemar Cox | 2 | 3652 | 795.60 |
| Natasa Milic-Frayling | 3 | 917 | 75.24 |
| Milad Shokouhi | 4 | 1109 | 50.63 |
| Emine Yilmaz | 5 | 1459 | 96.39 |