Paper Info
Title
Measuring the Utility of Search Engine Result Pages: An Information Foraging Based Measure.
Abstract
Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised "answers'') positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the \CWL framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over~1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).
Year
Venue
Field
2018
SIGIR
Information foraging theory,Information system,Web search engine,Information foraging,Units of measurement,Search engine,Ranking,Information retrieval,Computer science,Foraging
DocType
ISBN
Citations 
Conference
978-1-4503-5657-2
6
PageRank 
References 
Authors
0.50
31
3
Name
Order
Citations
PageRank
Leif Azzopardi11919133.10
Paul Thomas270551.00
Nick Craswell33942279.60