Paper Info
Title
Three-stream spatio-temporal attention network for first-person action and interaction recognition
Abstract
The problem of action and interaction recognition of human activities from the perspective of first-person view-point is an interesting area of research in the field of human action recognition (HAR). This paper presents a data-driven spatio-temporal network to combine different modalities computed from first-person videos using a temporal attention mechanism. First, our proposed approach uses three-stream inflated 3D ConvNet (I3D) to extract low-level features from RGB frame difference (FD), optical flow (OF) and magnitude-orientation (MO) streams. An I3D network has the advantage to directly learn spatio-temporal features over short video snippets (like 16 frames). Second, the extracted features are fused together and fed to a Bidirectional long short-term memory (BiLSTM) network to model high-level temporal feature sequences. Third, we propose to incorporate attention mechanism with our BiLSTM network to automatically select the most relevant temporal snippets in the given video sequence. Finally, we conducted extensive experiments and achieve state-of-the-art results on JPL (98.5%), NUS (84.1%), UTK (91.5%) and DogCentric (83.3%) datasets. These results show that features extracted from three-stream network are complementary to each other, and attention mechanism further improves the results by a large margin than previous attempts based on handcrafted and deep features.
Year
DOI
Venue
2022
10.1007/s12652-021-02940-4
JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING
Keywords
DocType
Volume
First-person action recognition, 3D convolutional neural network, Recurrent neural network, Feature fusion, Soft attention
Journal
13
Issue
ISSN
Citations 
2
1868-5137
0
PageRank 
References 
Authors
0.34
0
2
Name
Order
Citations
PageRank
Javed Imran1152.26
Balasubramanian Raman267970.23