Paper Info
Title
Improving Reliability of Crowdsourced Results by Detecting Crowd Workers with Multiple Identities.
Abstract
Quality control in crowdsourcing marketplaces plays a vital role in ensuring useful outcomes. In this paper, we focus on tackling the issue of crowd workers participating in tasks multiple times using different worker-ids to maximize their earnings. Workers attempting to complete the same task repeatedly may not be harmful in cases where the aim of a requester is to gather data or annotations, wherein more contributions from a single worker are fruitful. However, in several cases where the outcomes are subjective, requesters prefer the participation of distinct crowd workers. We show that traditional means to identify unique crowd workers such as worker-ids and ip-addresses are not sufficient. To overcome this problem, we propose the use of browser fingerprinting in order to ascertain the unique identities of crowd workers in paid crowdsourcing microtasks. By using browser fingerprinting across 8 different crowdsourced tasks with varying task difficulty, we found that 6.18% of crowd workers participate in the same task more than once, using different worker-ids to avoid detection. Moreover, nearly 95% of such workers in our experiments pass gold-standard questions and are deemed to be trustworthy, significantly biasing the results thus produced.
Year
Venue
Field
2017
ICWE
Earnings,Data mining,Crowdsourcing,Trustworthiness,Computer science
DocType
Citations 
PageRank 
Conference
1
0.35
References 
Authors
14
2
Name
Order
Citations
PageRank
Ujwal Gadiraju1698.42
Ricardo Kawase2152.72