Paper Info
Title
What Ingredients Make For An Effective Crowdsourcing Protocol For Difficult Nlu Data Collection Tasks?
Abstract
Crowdsourcing is widely used to create data for common natural language understanding tasks. Despite the importance of these datasets for measuring and refining model understanding of language, there has been little focus on the crowdsourcing methods used for collecting the datasets. In this paper, we compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We use multiple-choice question answering as a testbed and run a randomized trial by assigning crowdworkers to write questions under one of four different data collection protocols. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. However, we find that training crowdworkers, and then using an iterative process of collecting data, sending feedback, and qualifying workers based on expert judgments is an effective means of collecting challenging data. But using crowdsourced, instead of expert judgments, to qualify workers and send feedback does not prove to be effective. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures. Notably, the human-model gap on the unanimous agreement portion of this data is, on average, twice as large as the gap for the baseline protocol data.
Year
DOI
Venue
2021
10.18653/v1/2021.acl-long.98
59TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS AND THE 11TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING, VOL 1 (ACL-IJCNLP 2021)
DocType
Volume
Citations 
Conference
2021.acl-long
0
PageRank 
References 
Authors
0.34
0
6
Name
Order
Citations
PageRank
Nikita Nangia11326.56
Saku Sugawara2102.94
Harsh Trivedi343.51
Alex Warstadt4184.84
Clara Vania500.34
Samuel R. Bowman690644.99