Paper Info
Title
Linking the Characters: Video-oriented Social Graph Generation via Hierarchical-cumulative GCN
Abstract
ABSTRACTRecent years have witnessed the booming of online video platforms. Along this line, a graph to illustrate social relation among characters has been long expected to not only benefit the audiences for better understanding the story, but also support the fine-grained video analysis task in a semantic way. Unfortunately, though we humans could easily infer the social relations among characters, it is still an extremely challenging task for intelligent systems to automatically capture the social relation by absorbing multi-modal cues. Besides, they fail to describe the relations among multiple characters in a graph-generation perspective. To that end, inspired by the human inference ability on social relationship, we propose a novel Hierarchical- Cumulative Graph Convolutional Network (HC-GCN) to generate the social relation graph for multiple characters in the video. Specifically, we first integrate the short-term multi-modal cues, including visual, textual and audio information, to generate the frame-level graphs for part of characters via multimodal graph convolution technique. While dealing with the video-level aggregation task, we design an end-to-end framework to aggregate all frame-level subgraphs along the temporal trajectory, which results in a global video-level social graph with various social relationships among multiple characters. Extensive validations on two real-world large-scale datasets demonstrate the effectiveness of our proposed method compared with SOTA baselines.
Year
DOI
Venue
2021
10.1145/3474085.3475684
International Multimedia Conference
DocType
Citations 
PageRank 
Conference
1
0.37
References 
Authors
0
7
Name
Order
Citations
PageRank
Shiwei Wu110.37
Chen Joya211.72
Tong Xu321836.15
Liyi Chen441.84
Lingfei Wu511632.05
Yao Hu610.37
Enhong Chen72106165.57