Name
Title | ||
|---|---|---|
Measuring system performance and topic discernment using generalized adaptive-weight mean |
Abstract | ||
|---|---|---|
Standard approaches to evaluating and comparing information retrieval systems compute simple averages of performance statistics across individual topics to measure the overall system performance. However, topics vary in their ability to differentiate among systems based on their retrieval performance. At the same time, systems that perform well on discriminative queries demonstrate notable qualities that should be reflected in the systems' evaluation and ranking. This motivated research on alternative performance measures that are sensitive to the discriminative value of topics and the performance consistency of systems. In this paper we provide a mathematical formulation of a performance measure that postulates the dependence between the system and topic characteristics. We propose the Generalized Adaptive-Weight Mean (GAWM) measure and show how it can be computed as a fixed point of a function for which the Brouwer Fixed Point Theorem applies. This guarantees the existence of a scoring scheme that satisfies the starting axioms and can be used for ranking of both systems and topics. We apply our method to TREC experiments and compare the GAWM with the standard averages used in TREC. |
Year | DOI | Venue |
|---|---|---|
2009 | 10.1145/1645953.1646294 | CIKM |
Keywords | Field | DocType |
performance consistency,information retrieval system,retrieval performance,discriminative query,performance statistic,performance measure,discriminative value,trec experiment,topic discernment,overall system performance,alternative performance measure,measurement system,fixed point theorem,system performance,satisfiability,fixed point | Data mining,Information retrieval,Ranking,Computer science,Brouwer fixed-point theorem,Axiom,Theoretical computer science,Artificial intelligence,Fixed point,Discriminative model,Machine learning,Discernment | Conference |
Citations | PageRank | References |
2 | 0.39 | 11 |
Authors | ||
6 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Chung Tong Lee | 1 | 19 | 1.12 |
| Vishwa Vinay | 2 | 245 | 15.94 |
| Eduarda Mendes Rodrigues | 3 | 350 | 21.40 |
| Gabriella Kazai | 4 | 1151 | 97.35 |
| Natasa Milic-Frayling | 5 | 917 | 75.24 |
| Aleksandar Ignjatovic | 6 | 556 | 49.24 |