Paper Info
Title
Enhanced Attention Tracking With Multi-Branch Network for Egocentric Activity Recognition
Abstract
The emergence of wearable devices has opened up new potentials for egocentric activity recognition. Although some methods integrate attention mechanisms into deep neural networks to capture fine-grained human-object interactions in a weak-supervision manner, they either ignore exploiting the temporal consistency or generate attention based on considering appearance cues only. To address these limitations, in this paper, we propose an enhanced attention-tracking method, combined with multi-branch network (EAT-MBNet), for egocentric activity recognition. Specifically, we propose class-aware attention maps (CAAMs) by employing a self-attention-based module to refine the class activation maps (CAMs). Our proposed method can enhance the semantic dependency between the activity categories and the feature maps. To highlight the discriminative features from the regions of interest across frames, we propose a flow-guided attention-tracking (F-AT) module, by simultaneously leveraging historical attention and motion patterns. Furthermore, we propose a cross-modality modeling branch based on an interactive GRU module, which captures the time-synchronized long-term relationships between the appearance and motion branches. Experimental results on four egocentric activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance.
Year
DOI
Venue
2022
10.1109/TCSVT.2021.3104651
IEEE Transactions on Circuits and Systems for Video Technology
Keywords
DocType
Volume
Egocentric activity recognition,attention tracking,multi-branch network,fine-grained hand-object interactions
Journal
32
Issue
ISSN
Citations 
6
1051-8215
0
PageRank 
References 
Authors
0.34
11
4
Name
Order
Citations
PageRank
Tianshan Liu194.27
Kin-Man Lam21911141.72
Rui Zhao301.35
Jun Kong411118.94