Paper Info
Title
Multimodal Fusion of BERT-CNN and Gated CNN Representations for Depression Detection
Abstract
Depression is a common, but serious mental disorder that affects people all over the world. Besides providing an easier way of diagnosing the disorder, a computer-aided automatic depression assessment system is demanded in order to reduce subjective bias in the diagnosis. We propose a multimodal fusion of speech and linguistic representation for depression detection. We train our model to infer the Patient Health Questionnaire (PHQ) score of subjects from AVEC 2019 DDS Challenge database, the E-DAIC corpus. For the speech modality, we use deep spectrum features extracted from a pretrained VGG-16 network and employ a Gated Convolutional Neural Network (GCNN) followed by a LSTM layer. For the textual embeddings, we extract BERT textual features and employ a Convolutional Neural Network (CNN) followed by a LSTM layer. We achieved a CCC score equivalent to 0.497 and 0.608 on the E-DAIC corpus development set using the unimodal speech and linguistic models respectively. We further combine the two modalities using a feature fusion approach in which we apply the last representation of each single modality model to a fully-connected layer in order to estimate the PHQ score. With this multimodal approach, it was possible to achieve the CCC score of 0.696 on the development set and 0.403 on the testing set of the E-DAIC corpus, which shows an absolute improvement of 0.283 points from the challenge baseline.
Year
DOI
Venue
2019
10.1145/3347320.3357694
Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop
Keywords
Field
DocType
CNN, affective computing, bert, deep learning, depression detection, gated CNN, multimodal systems
Computer science,Fusion,Speech recognition
Conference
ISBN
Citations 
PageRank 
978-1-4503-6913-8
0
0.34
References 
Authors
0
4
Name
Order
Citations
PageRank
Mariana Rodrigues Makiuchi100.34
Tifani Warnita210.77
Kuniaki Uto33210.40
Koichi Shinoda446365.14