Name
Abstract | ||
|---|---|---|
The advent of crowdsourcing is revolutionizing data annotation, evaluation, and other traditionally manual-labor intensive processes by dramatically reducing the time, cost, and effort involved. This in turn is driving a disruptive shift in search and data mining methodology in areas such as: Evaluation: the Cranfield paradigm for search evaluation requires manually assessing document relevance to search queries. Recent work on stochastic evaluation has reduced but not removed this need for manual assessment. Supervised Learning: while traditional costs associated with data annotation have driven recent machine learning work (e.g. Learning to Rank) toward greater use of unsupervised and semi-supervised methods, the emergence of crowdsourcing has made labeled data far easier to acquire, thereby driving a potential resurgence in fully-supervised methods. Applications: Crowdsourcing has introduced exciting new opportunities to integrate human labor into automated systems: handling difficult cases where automation fails, exploiting the breadth of backgrounds, geographic dispersion, real-time response, etc. |
Year | DOI | Venue |
|---|---|---|
2011 | 10.1145/1935826.1935828 | SIGIR Forum |
Keywords | DocType | Volume |
data mining,automated system,hong kong,crowdsourcing,difficult case,data annotation,empirical method,refereed paper,fourth acm international conference,workshop proceeding,recent machine,presentation slide,data mining methodology,recent advance,onweb search,search evaluation,search,cranfield paradigm,recent work,novel application,stochastic evaluation,supervised learning,real time,machine learning,learning to rank | Conference | 45 |
Issue | Citations | PageRank |
1 | 6 | 0.59 |
References | Authors | |
11 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Vitor R. Carvalho | 1 | 672 | 36.38 |
| Matthew Lease | 2 | 1326 | 84.06 |
| Emine Yilmaz | 3 | 1459 | 96.39 |