Paper Info
Title
Boosting Video Description Generation by Explicitly Translating from Frame-Level Captions.
Abstract
Automatically describing video content with natural language is a fundamental challenge of computer vision. The recent advanced technique that approaches this problem is Recurrent Neural Networks (RNN). The need to train RNN on large-scale complex and diverse videos and their associated language, however, makes the task human-labeling intensive and computationally expensive. Moreover, the results can suffer from robustness problem, especially when there are rich of temporal dynamics in the sequence of video frames. We demonstrate in this paper that the above two limitations can be mitigated by jointly exploring the largely available data from image domain and representing each frame by high-level attributes rather than visual features. The former leverages the learnt models on image captioning benchmark to generate caption for each video frame, while the latter explicitly incorporates the obtained captions which are regarded as the attributes of each frame. Specifically, we propose a novel sequence to sequence architecture to generate descriptions for videos, in a sense that the inputs are the captions of sequential frames and it outputs words sequentially. On a widely used YouTube2Text dataset, our proposal is shown to be powerful with superior performance over several state-of-the-art methods including both architectures that are purely developed on video data and RNN-based models which translate directly from visual features to language.
Year
DOI
Venue
2016
10.1145/2964284.2967298
ACM Multimedia
Keywords
Field
DocType
Video Captioning,Image Captioning,Recurrent Neural Networks,Deep Convolutional Neural Networks
Computer vision,Architecture,Closed captioning,Computer science,Recurrent neural network,Robustness (computer science),Speech recognition,Natural language,Boosting (machine learning),Artificial intelligence,Multimedia,Machine learning
Conference
Citations 
PageRank 
References 
7
0.46
11
Authors
2
Name
Order
Citations
PageRank
Yuan Liu121511.43
Shi Zhongchao2468.98