Paper Info
Title
A Study of Realtime Summarization Metrics
Abstract
Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their evaluation is not well understood as classic methods for text summarization are inappropriate for real-time and streaming conditions. The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems.
Year
DOI
Venue
2016
10.1145/2983323.2983653
ACM International Conference on Information and Knowledge Management
Keywords
Field
DocType
evaluation,summarization,realtime,stream filtering,temporal summarization
Text graph,Data mining,Automatic summarization,Multi-document summarization,Social media,Best practice,Information retrieval,Computer science,News media,Dynamic text
Conference
Citations 
PageRank 
References 
3
0.53
14
Authors
4
Name
Order
Citations
PageRank
Matthew Ekstrand-Abueg1335.35
Richard Mccreadie240332.43
Virgiliu Pavlu353544.07
Fernando Diaz4198597.72