Paper Info
Title
DO WE AGREE? MEASURING AGREEMENT ON THE HUMAN JUDGMENTS IN EMOTION ANNOTATION OF NEWS SENTENCES
Abstract
An emotional text may be judged to belong to multiple emotion categories because it may evoke different emotions with varying degrees of intensity. For emotion analysis of text in a supervised manner, it is required to annotate text corpus with emotion categories. Because emotion is a very subjective entity, producing reliable annotation is of prime requirement for developing a robust emotion analysis model, so it is wise to have the data set annotated by multiple human judges and generate an aggregated data set provided that the emotional responses provided by different annotators over the data set exhibit substantial agreement. In reality, multiple emotional responses for an emotional text are common. So, the data set is a multilabel one where a single data item may belong to more than one category simultaneously. This article presents a new agreement measure to compute interannotator reliability in multilabel annotation. The new reliability coefficient has been applied to measure the quality of an emotion text corpus. The procedure for generating aggregated data and some corpus cleaning techniques are also discussed.
Year
DOI
Venue
2010
10.1080/01969722.2010.511525
Cybernetics and Systems
Keywords
Field
DocType
different emotion,news sentences,multiple emotion category,emotion text corpus,measuring agreement,robust emotion analysis model,emotion annotation,emotion analysis,single data item,emotion category,human judgments,aggregated data,emotional text,natural language processing
Prime (order theory),Annotation,Reliability (statistics),Text corpus,Speech recognition,Natural language processing,Artificial intelligence,Mathematics
Journal
Volume
Issue
ISSN
41
7
0196-9722
Citations 
PageRank 
References 
0
0.34
7
Authors
3
Name
Order
Citations
PageRank
Plaban Kumar Bhowmick1208.62
Pabitra Mitra21729126.79
Anupam Basu340561.68