Name
Title | ||
|---|---|---|
Affect recognition from facial movements and body gestures by hierarchical deep spatio-temporal features and fusion strategy. |
Abstract | ||
|---|---|---|
Affect presentation is periodic and multi-modal, such as through facial movements, body gestures, and so on. Studies have shown that temporal selection and multi-modal combinations may benefit affect recognition. In this article, we therefore propose a spatio-temporal fusion model that extracts spatio-temporal hierarchical features based on select expressive components. In addition, a multi-modal hierarchical fusion strategy is presented. Our model learns the spatio-temporal hierarchical features from videos by a proposed deep network, which combines a convolutional neural networks (CNN), bilateral long short-term memory recurrent neural networks (BLSTM-RNN) with principal component analysis (PCA). Our approach handles each video as a “video sentence.” It first obtains a skeleton with the temporal selection process and then segments key words with a certain sliding window. Finally, it obtains the features with a deep network comprised of a video-skeleton and video-words. Our model combines the feature level and decision level fusion for fusing the multi-modal information. Experimental results showed that our model improved the multi-modal affect recognition accuracy rate from 95.13% in existing literature to 99.57% on a face and body (FABO) database, our results have been increased by 4.44%, and it obtained a macro average accuracy (MAA) up to 99.71%. |
Year | DOI | Venue |
|---|---|---|
2018 | 10.1016/j.neunet.2017.11.021 | Neural Networks |
Keywords | Field | DocType |
Affect recognition,Deep learning,Convolutional neural network,Bilateral long short-term memory recurrent neural network,Deep spatio-temporal hierarchical feature,Multi-modal feature fusion strategy | Sliding window protocol,Decision level,Pattern recognition,Convolutional neural network,Gesture,Fusion,Recurrent neural network,Artificial intelligence,Macro,Machine learning,Mathematics,Principal component analysis | Journal |
Volume | Issue | ISSN |
105 | 1 | 0893-6080 |
Citations | PageRank | References |
7 | 0.49 | 37 |
Authors | ||
4 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Bo Sun | 1 | 104 | 21.35 |
| Siming Cao | 2 | 14 | 2.59 |
| Jun He | 3 | 71 | 11.24 |
| Lejun Yu | 4 | 36 | 3.28 |