Paper Info
Title
Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank
Abstract
In this paper, we present several ways to measure and evaluate the annotation and annotators, proposed and used during the building of the Czech part of the Prague Czech-English Dependency Treebank. At first, the basic principles of the treebank annotation project are introduced (division to three layers: morphological, analytical and tectogrammatical). The main part of the paper describes in detail one of the important phases of the annotation process: three ways of evaluation of the annotators - inter-annotator agreement, error rate and performance. The measuring of the inter-annotator agreement is complicated by the fact that the data contain added and deleted nodes, making the alignment between annotations non-trivial. The error rate is measured by a set of automatic checking procedures that guard the validity of some invariants in the data. The performance of the annotators is measured by a booking web application. All three measures are later compared and related to each other.
Year
Venue
Field
2010
LREC 2010 - SEVENTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION
Czech,Computer science,Artificial intelligence,Treebank,Natural language processing
DocType
Citations 
PageRank 
Conference
2
0.40
References 
Authors
2
2
Name
Order
Citations
PageRank
Marie Mikulová1203.97
Jan Stepánek2867.37