Paper Info
Title
Translating Video Content to Natural Language Descriptions
Abstract
Humans use rich natural language to describe and communicate visual perceptions. In order to provide natural language descriptions for visual content, this paper combines two important ingredients. First, we generate a rich semantic representation of the visual content including e.g. object and activity labels. To predict the semantic representation we learn a CRF to model the relationships between different components of the visual input. And second, we propose to formulate the generation of natural language as a machine translation problem using the semantic representation as source language and the generated sentences as target language. For this we exploit the power of a parallel corpus of videos and textual descriptions and adapt statistical machine translation to translate between our two languages. We evaluate our video descriptions on the TACoS dataset, which contains video snippets aligned with sentence descriptions. Using automatic evaluation and human judgments we show significant improvements over several baseline approaches, motivated by prior work. Our translation approach also shows improvements over related work on an image description task.
Year
DOI
Venue
2013
10.1109/ICCV.2013.61
ICCV
Keywords
Field
DocType
natural language description,machine translation problem,visual input,source language,visual perception,semantic representation,target language,visual content,rich natural language,natural language,translating video content,natural language descriptions,language translation,natural language processing,computer vision
Rule-based machine translation,Computer vision,Cache language model,Language translation,Computer science,Machine translation,Natural language programming,Natural language,Universal Networking Language,Artificial intelligence,Language identification,Natural language processing
Conference
ISSN
Citations 
PageRank 
1550-5499
115
4.28
References 
Authors
24
6
Search Limit
100115
Name
Order
Citations
PageRank
Marcus Rohrbach13138107.83
Wei Qiu21154.28
Ivan Titov3148481.98
Stefan Thater475638.54
Manfred Pinkal5111669.77
Bernt Schiele612901971.29