Name
Abstract | ||
|---|---|---|
A key step in building multi-agent systems is to gather data reported by the agents (people), in either cardinal (numeric ratings) or ordinal (rankings) form. Cardinal scores collected from people are well known to sufer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood. It is widely believed that in the absence of simplifying assumptions on the miscalibration, the only useful information in practice from the cardinal scores is the induced ranking. In this paper we address the fundamental question of whether this widespread folklore belief is actually true. We consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations that is only required to be consistent with the induced ranking. We design rating-based estimators and prove that despite making no assumptions on the ratings, they strictly and uniformly outperform all possible estimators that rely on only the ranking. These estimators can be used as a plug-in to show the superiority of cardinal scores over ordinal rankings for a variety of applications, and we provide examples for A/B testing and ranking as a proof of concept. Our results thus provide novel fundamental insights in the eternal debate between cardinal and ordinal data. |
Year | DOI | Venue |
|---|---|---|
2018 | 10.5555/3306127.3331778 | adaptive agents and multi-agents systems |
Keywords | Field | DocType |
Miscalibration,crowdsourcing,data collection methodologies,preference aggregation,multi-agent systems | Ranking,Crowdsourcing,Ordinal data,CONTEST,Artificial intelligence,Mathematics,Machine learning,Estimator,Bayes' theorem | Journal |
Volume | Citations | PageRank |
abs/1806.05085 | 1 | 0.36 |
References | Authors | |
5 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Jingyan Wang | 1 | 37 | 1.74 |
| Nihar B. Shah | 2 | 1202 | 77.17 |