Paper Info
Title
Dynamic Estimation of Rater Reliability in Subjective Tasks Using Multi-armed Bandits
Abstract
Many application areas that use supervised machine learning make use of multiple raters to collect target ratings for training data. Usage of multiple raters, however, inevitably introduces the risk that a proportion of them will be unreliable. The presence of unreliable raters can prolong the rating process, make it more expensive and lead to inaccurate ratings. The dominant, "static" approach of solving this problem in state-of-the-art research is to estimate the rater reliability and to calculate the target ratings when all ratings have been gathered. However, doing it dynamically while raters rate training data can make the acquisition of ratings faster and cheaper compared to static techniques. We propose to cast the problem of the dynamic estimation of rater reliability as a multi-armed bandit problem. Experiments show that the usage of multi-armed bandits for this problem is worthwhile, providing that each rater can rate any asset when asked. The purpose of this paper is to outline the directions of future research in this area.
Year
DOI
Venue
2012
10.1109/SocialCom-PASSAT.2012.50
PASSAT), 2012 International Conference and 2012 International Confernece Social Computing
Keywords
Field
DocType
rater reliability,dynamic estimation,multi-armed bandit,multiple raters,static technique,raters rate training data,target rating,multi-armed bandit problem,unreliable raters,state-of-the-art research,multi-armed bandits,risk analysis,learning artificial intelligence,training data,crowdsourcing,reliability theory
Training set,Risk analysis (business),Crowdsourcing,Computer science,Human computation,Artificial intelligence,Machine learning,Reliability theory
Conference
ISBN
Citations 
PageRank 
978-1-4673-5638-1
0
0.34
References 
Authors
4
3
Name
Order
Citations
PageRank
Alexey Tarasov1283.54
Sarah Jane Delany244629.95
Brian Mac Namee312224.28