Paper Info
Title
Towards Explainable Deep Learning for Image Captioning through Representation Space Perturbation
Abstract
Most vision-language tasks, such as image captioning, require complex deep learning (DL) models/architectures. However, the high complexity of such models often leads their inner functioning to be regarded as black-boxes, preventing humans from understanding their decision process. Representation space is a key element in DL models, understanding this space and how information is encoded constitute an interesting approach for interpretable DL. Indeed, as far as we are aware, this paper is the first one presenting a novel method based on perturbation principle of representation space, to study the components that influence DL architectures used in image captioning. The core idea is to isolate and identify the importance of each component (element of the architecture) involved in the captioning pipeline by perturbing, by means of Gaussian functions, the representation space rather than the original space of inputs. We experimentally demonstrate that those components differ in their influence and relevance. The results show that the visual modality would constitute a critical explanation target in captioning models in contrast to language modality, thus leading to more fine-grained explanations. We also propose MSICE, an automatic evaluation metric for image captioning which addresses two important yet overlooked linguistic aspects, morphology and semantic. Our code will be publicly accessible to support future research in this field.
Year
DOI
Venue
2022
10.1109/IJCNN55064.2022.9892275
2022 International Joint Conference on Neural Networks (IJCNN)
Keywords
DocType
ISSN
Explainability,Image Captioning,Deep Learning (DL),Perturbation,Vision-Language
Conference
2161-4393
ISBN
Citations 
PageRank 
978-1-6654-9526-4
0
0.34
References 
Authors
9
4
Name
Order
Citations
PageRank
Sofiane Elguendouze100.34
Marcilio C. P. de Souto200.34
Adel Hafiane300.34
Anaïs Halftermeyer400.34