Paper Info
Title
Gaussian Processes Regression with Multiple Annotators - When the Annotator Performance Is Not Homogeneous.
Abstract
In supervised learning problems, the right label (also known as the gold standard or the ground truth) is not available because the label acquisition can be expensive or infeasible. Instead of that gold standard, we have access to some annotations provided by multiple annotators with different levels of expertise. Hence, trivial methods such as majority voting (or average in regression problems) are not suitable since they assume homogeneity between the expertise of the labelers. In this work, we introduce a regression approach based on Gaussian processes, where we consider that the expertise of the labelers is non-homogeneous across the input space–(GPR-MANH). The idea is to assume that the input space can be represented by a defined number of regions where each annotator exhibit a particular level of expertise. Experimental results show that our methodology can estimate the performance of annotators even if the gold standard is not available, defeating state-of-the-art techniques.
Year
DOI
Venue
2018
10.1007/978-3-030-13469-3_11
CIARP
Field
DocType
Citations 
Homogeneity (statistics),Pattern recognition,Regression,Computer science,Homogeneous,Supervised learning,Ground truth,Gaussian process,Artificial intelligence,Regression problems,Majority rule
Conference
0
PageRank 
References 
Authors
0.34
7
3
Name
Order
Citations
PageRank
Julián Gil González100.34
Andrés Marino Álvarez-Meza29316.89
Álvaro Á. Orozco31612.88