Name
Title | ||
|---|---|---|
Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition |
Abstract | ||
|---|---|---|
ABSTRACTAutomatic audio-visual expression recognition can play an important role in communication services such as tele-health, VOIP calls and human-machine interaction. Accuracy of audio-visual expression recognition could benefit from the interplay between the two modalities. However, most audio-visual expression recognition systems, trained in ideal conditions, fail to generalize in real world scenarios where either the audio or visual modality could be missing due to a number of reasons such as limited bandwidth, interactors' orientation, caller initiated muting. This paper studies the performance of a state-of-the art transformer when one of the modalities is missing. We conduct ablation studies to evaluate the model in the absence of either modality. Further, we propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic real world scenarios. Results conducted on in-the-wild data, indicate significant generalization in proposed models trained on missing cues, with gains up to 17% for frame level ablations, showing that these training strategies cope better with the loss of input modalities. |
Year | DOI | Venue |
|---|---|---|
2020 | 10.1145/3395035.3425202 | ICMI-MLMI |
DocType | Citations | PageRank |
Conference | 2 | 0.36 |
References | Authors | |
0 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| S. Parthasarthy | 1 | 60 | 5.25 |
| Shiva Sundaram | 2 | 142 | 16.01 |