Name
Abstract | ||
|---|---|---|
Evaluation metric is an essential and integral part of a ranking system. In the past, several evaluation metrics have been proposed in information retrieval and web search, among them Discounted Cumulative Gain (DCG) has emerged as one that is widely adopted for evaluating the performance of ranking functions used in web search. However, the two sets of parameters, the gain values and discount factors, used in DCG are usually determined in a rather ad-hoc way, and their impacts have not been carefully analyzed. In this paper, we first show that DCG is generally not coherent, i.e., comparing the performance of ranking functions using DCG very much depends on the particular gain values and discount factors used. We then propose a novel methodology that can learn the gain values and discount factors from user preferences over rankings, modeled as a special case of learning linear utility functions. We also discuss how to extend our methods to handle tied preference pairs and how to explore active learning to reduce preference labeling. Numerical simulations illustrate the effectiveness of our proposed methods. Moreover, experiments are also conducted over a side-by-side comparison data set from a commercial search engine to validate the proposed methods on real-world data. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/TKDE.2012.252 | IEEE Trans. Knowl. Data Eng. |
Keywords | Field | DocType |
dcg,linear utility functions learning,discount factor,particular gain value,web search,user preferences,learning (artificial intelligence),utility function,discount factors,ranking function,information retrieval,commercial search engine,ranking functions,discounted cumulative gains,user preference,gain value,evaluation metric,comparison data,gain values,preference labeling,machine learning,discounted cumulative gain,numerical simulations,side-by-side comparison data,ranking system,search engines,learning artificial intelligence,production,measurement,vectors,optimization | Data mining,Active learning,Search engine,Ranking,Computer science,Artificial intelligence,Machine learning,Discounted cumulative gain,Special case | Journal |
Volume | Issue | ISSN |
26 | 2 | 1041-4347 |
Citations | PageRank | References |
3 | 0.41 | 10 |
Authors | ||
4 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Ke Zhou | 1 | 27 | 1.54 |
| Hongyuan Zha | 2 | 6703 | 422.09 |
| Yi Chang | 3 | 1463 | 86.17 |
| Gui-rong Xue | 4 | 2728 | 126.58 |