Paper Info
Title
Counterfactual Estimation and Optimization of Click Metrics in Search Engines: A Case Study
Abstract
Optimizing an interactive system against a predefined online metric is particularly challenging, especially when the metric is computed from user feedback such as clicks and payments. The key challenge is the counterfactual nature: in the case of Web search, any change to a component of the search engine may result in a different search result page for the same query, but we normally cannot infer reliably from search log how users would react to the new result page. Consequently, it appears impossible to accurately estimate online metrics that depend on user feedback, unless the new engine is actually run to serve live users and compared with a baseline in a controlled experiment. This approach, while valid and successful, is unfortunately expensive and time-consuming. In this paper, we propose to address this problem using causal inference techniques, under the contextual-bandit framework. This approach effectively allows one to run potentially many online experiments offline from search log, making it possible to estimate and optimize online metrics quickly and inexpensively. Focusing on an important component in a commercial search engine, we show how these ideas can be instantiated and applied, and obtain very promising results that suggest the wide applicability of these techniques.
Year
DOI
Venue
2015
10.1145/2740908.2742562
WWW (Companion Volume)
Keywords
Field
DocType
Experimental design, counterfactual analysis, Web search, query correction/rewriting, information retrieval, contextual bandits
Data mining,Causal inference,Web search query,World Wide Web,Search engine,Query expansion,Information retrieval,Computer science,Web query classification,Counterfactual thinking,Search analytics,Payment
Conference
Citations 
PageRank 
References 
23
0.93
28
Authors
4
Name
Order
Citations
PageRank
Lihong Li12390128.53
Shunbao Chen2291.45
Jim Kleban31989.81
Ankur Gupta486055.56