Paper Info
Title
Untangling Result List Refinement and Ranking Quality: a Framework for Evaluation and Prediction
Abstract
Traditional batch evaluation metrics assume that user interaction with search results is limited to scanning down a ranked list. However, modern search interfaces come with additional elements supporting result list refinement (RLR) through facets and filters, making user search behavior increasingly dynamic. We develop an evaluation framework that takes a step beyond the interaction assumption of traditional evaluation metrics and allows for batch evaluation of systems with and without RLR elements. In our framework we model user interaction as switching between different sublists. This provides a measure of user effort based on the joint effect of user interaction with RLR elements and result quality. We validate our framework by conducting a user study and comparing model predictions with real user performance. Our model predictions show significant positive correlation with real user effort. Further, in contrast to traditional evaluation metrics, the predictions using our framework, of when users stand to benefit from RLR elements, reflect findings from our user study. Finally, we use the framework to investigate under what conditions systems with and without RLR elements are likely to be effective. We simulate varying conditions concerning ranking quality, users, task and interface properties demonstrating a cost-effective way to study whole system performance.
Year
DOI
Venue
2015
10.1145/2766462.2767740
International Conference on Research an Development in Information Retrieval
Keywords
Field
DocType
Simulation,Search behavior,Faceted search,Evaluation
Data mining,Information retrieval,Faceted search,Ranking,Computer science,Positive correlation
Conference
Citations 
PageRank 
References 
1
0.37
29
Authors
5
Name
Order
Citations
PageRank
Jiyin He132926.45
Marc Bron228617.47
Arjen P. de Vries3136484.36
Leif Azzopardi41919133.10
Maarten de Rijke56516509.76