Paper Info
Title
Is it worth the effort? Assessing the benefits of partial automatic pre-labeling for frame-semantic annotation
Abstract
Corpora with high-quality linguistic annotations are an essential component in many NLP applications and a valuable resource for linguistic research. For obtaining these annotations, a large amount of manual effort is needed, making the creation of these resources time-consuming and costly. One attempt to speed up the annotation process is to use supervised machine-learning systems to automatically assign (possibly erroneous) labels to the data and ask human annotators to correct them where necessary. However, it is not clear to what extent these automatic pre-annotations are successful in reducing human annotation effort, and what impact they have on the quality of the resulting resource. In this article, we present the results of an experiment in which we assess the usefulness of partial semi-automatic annotation for frame labeling. We investigate the impact of automatic pre-annotation of differing quality on annotation time, consistency and accuracy. While we found no conclusive evidence that it can speed up human annotation, we found that automatic pre-annotation does increase its overall quality.
Year
DOI
Venue
2012
10.1007/s10579-011-9170-z
Language Resources and Evaluation
Keywords
Field
DocType
Linguistic annotation,Semantic role labelling,Frame semantics,Semi-automatic annotation
Minimum information required in the annotation of models,Ask price,Annotation,Information retrieval,Temporal annotation,Semantic annotation,Computer science,Image retrieval,Artificial intelligence,Frame semantics,Natural language processing,Speedup
Journal
Volume
Issue
ISSN
46
1
1574-020X
Citations 
PageRank 
References 
1
0.44
9
Authors
3
Name
Order
Citations
PageRank
Ines Rehbein114219.21
Josef Ruppenhofer223030.43
Caroline Sporleder345331.84