Paper Info
Title
Measuring variability in sentence ordering for news summarization
Abstract
The issue of sentence ordering is an important one for natural language tasks such as multi-document summarization, yet there has not been a quantitative exploration of the range of acceptable sentence orderings for short texts. We present results of a sentence reordering experiment with three experimental conditions. Our findings indicate a very high degree of variability in the orderings that the eighteen subjects produce. In addition, the variability of reorderings is significantly greater when the initial ordering seen by subjects is different from the original summary. We conclude that evaluation of sentence ordering should use multiple reference orderings. Our evaluation presents several metrics that might prove useful in assessing against multiple references. We conclude with a deeper set of questions: (a) what sorts of independent assessments of quality of the different reference orderings could be made and (b) whether a large enough test set would obviate the need for such independent means of quality assessment.
Year
Venue
Keywords
2007
ENLG
quality assessment,large enough test set,multiple reference,multiple reference ordering,different reference ordering,independent mean,acceptable sentence ordering,sentence reordering experiment,news summarization,independent assessment,deeper set,computer science,information technology
Field
DocType
Citations 
Automatic summarization,Information retrieval,Computer science,Information technology,Natural language,Artificial intelligence,Natural language processing,Sentence,Test set
Conference
10
PageRank 
References 
Authors
0.59
10
8
Name
Order
Citations
PageRank
Nitin Madnani159745.44
Rebecca Passonneau234227.13
Necip Fazil Ayan333923.36
John M. Conroy4100.59
Bonnie J. Dorr52150176.78
Judith L. Klavans6754188.15
O'Leary, Dianne P.71064222.93
Judith D. Schlesinger820112.43