Name
Abstract | ||
|---|---|---|
In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of 41% under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of 35.83% demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of 34.03%. |
Year | Venue | Keywords |
|---|---|---|
2015 | 2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP) | Audio-Visual Automatic Speech Recognition (AV-ASR), Multimodal Learning, Deep Neural Networks |
Field | DocType | Volume |
Speech processing,Speech coding,Pattern recognition,Audio mining,Voice activity detection,Computer science,Word error rate,Speech recognition,Audio-visual speech recognition,Artificial intelligence,Multimodal learning,Acoustic model | Journal | abs/1501.05396 |
ISSN | Citations | PageRank |
1520-6149 | 27 | 0.84 |
References | Authors | |
13 | 3 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Youssef Mroueh | 1 | 105 | 16.63 |
| Etienne Marcheret | 2 | 100 | 11.15 |
| Vaibhava Goel | 3 | 376 | 41.25 |