Paper Info
Title
Arousal Recognition Using Audio-Visual Features and FMRI-based Brain Response
Abstract
As the indicator of emotion intensity, arousal is a significant clue for users to find their interested content. Hence, effective techniques for video arousal recognition are highly required. In this paper, we propose a novel framework for recognizing arousal levels by integrating low-level audio-visual features derived from video content and human brain's functional activity in response to videos measured by functional magnetic resonance imaging (fMRI). At first, a set of audio-visual features which have been demonstrated to be correlated with video arousal are extracted. Then, the fMRI-derived features that convey the brain activity of comprehending videos are extracted based on a number of brain regions of interests (ROIs) identified by a universal brain reference system. Finally, these two sets of features are integrated to learn a joint representation by using a multimodal deep Boltzmann machine (DBM). The learned joint representation can be utilized as the feature for training classifiers. Due to the fact that fMRI scanning is expensive and time-consuming, our DBM fusion model has the ability to predict the joint representation of the videos without fMRI scans. The experimental results on a video benchmark demonstrated the effectiveness of our framework and the superiority of integrated features.
Year
DOI
Venue
2015
10.1109/TAFFC.2015.2411280
IEEE Trans. Affective Computing
Keywords
Field
DocType
Arousal recognition, affective computing, fMRI-derived features, multimodal DBM
Arousal,Boltzmann machine,Functional magnetic resonance imaging,Sentiment analysis,Psychology,Brain activity and meditation,Speech recognition,Feature extraction,Affective computing,Electroencephalography
Journal
Volume
Issue
ISSN
PP
99
1949-3045
Citations 
PageRank 
References 
8
0.46
37
Authors
5
Name
Order
Citations
PageRank
Junwei Han13501194.57
Xiang Ji280.46
Xintao Hu311813.53
Lei Guo41661142.63
Tianming Liu51033112.95