Name
Abstract | ||
|---|---|---|
The use of sampling, randomized algorithms, or training based on the unpredictable inputs of users in Information Retrieval often leads to non-deterministic outputs. Evaluating the effectiveness of systems incorporating these methods can be challenging since each run may produce different effectiveness scores. Current IR evaluation techniques do not address this problem. Using the context of distributed information retrieval as a case study for our investigation, we propose a solution based on multivariate linear modeling. We show that the approach provides a consistent and reliable method to compare the effectiveness of non-deterministic IR algorithms, and explain how statistics can safely be used to show that two IR algorithms have equivalent effectiveness. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1145/2600428.2609472 | SIGIR |
Keywords | Field | DocType |
experimentation,statistical analysis,information retrieval,effectiveness evaluation,measurement,experimental design,performance evaluation | Randomized algorithm,Data mining,IR evaluation,Linear model,Computer science,Multivariate statistics,Sampling (statistics),Artificial intelligence,Machine learning,Statistical analysis | Conference |
Citations | PageRank | References |
3 | 0.49 | 5 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Gaya K. Jayasinghe | 1 | 20 | 2.18 |
| William Webber | 2 | 585 | 31.30 |
| Mark Sanderson | 3 | 3751 | 341.56 |
| Lasitha S. Dharmasena | 4 | 6 | 1.25 |
| Shane Culpepper | 5 | 519 | 47.52 |