Title | ||
---|---|---|
Integrating Crowdsourcing and Active Learning for Classification of Work-Life Events from Tweets |
Abstract | ||
---|---|---|
Social media, especially Twitter, is being increasingly used for research with predictive analytics. In social media studies, natural language processing (NLP) techniques are used in conjunction with expert-based, manual and qualitative analyses. However, social media data are unstructured and must undergo complex manipulation for research use. The manual annotation is the most resource and time-consuming process that multiple expert raters have to reach consensus on every item, but is essential to create gold-standard datasets for training NLP-based machine learning classifiers. To reduce the burden of the manual annotation, yet maintaining its reliability, we devised a crowdsourcing pipeline combined with active learning strategies. We demonstrated its effectiveness through a case study that identifies job loss events from individual tweets. We used Amazon Mechanical Turk platform to recruit annotators from the Internet and designed a number of quality control measures to assure annotation accuracy. We evaluated 4 different active learning strategies (i.e., least confident, entropy, vote entropy, and Kullback-Leibler divergence). The active learning strategies aim at reducing the number of tweets needed to reach a desired performance of automated classification. Results show that crowdsourcing is useful to create high-quality annotations and active learning helps in reducing the number of required tweets, although there was no substantial difference among the strategies tested. |
Year | DOI | Venue |
---|---|---|
2020 | 10.1007/978-3-030-55789-8_30 | IEA/AIE |
DocType | Citations | PageRank |
Conference | 0 | 0.34 |
References | Authors | |
0 | 5 |
Name | Order | Citations | PageRank |
---|---|---|---|
Yunpeng Zhao | 1 | 8 | 5.83 |
Mattia C. F. Prosperi | 2 | 99 | 22.97 |
Lyu Tianchen | 3 | 0 | 0.34 |
Yi Guo | 4 | 43 | 6.19 |
Bian Jing | 5 | 0 | 0.34 |