Name
Title | ||
|---|---|---|
The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy |
Abstract | ||
|---|---|---|
Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of their work. In this paper, we ask how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits. Our findings show that the face of a crowd is in fact indicative of the quality of their work. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1145/2396761.2398697 | CIKM |
Keywords | Field | DocType |
crowdsourcing relevance label,information retrieval system,quality assurance,possible relationship,task completion,poorly-performing crowd,personality trait,specific task condition,control technique,varied output quality,relevance label,personality traits,design,crowdsourcing,human factors | Data mining,Big Five personality traits,Crowds,Ask price,Computer science,Crowdsourcing,Demographics,Task completion,Personality,Quality assurance | Conference |
Citations | PageRank | References |
29 | 1.04 | 14 |
Authors | ||
3 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Gabriella Kazai | 1 | 1151 | 97.35 |
| Jaap Kamps | 2 | 2078 | 178.56 |
| Natasa Milic-Frayling | 3 | 917 | 75.24 |