Paper Info
Title
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos.
Abstract
Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities. In this context, this paper proposes a self-supervised training framework that learns a common multimodal embedding space that, in addition to sharing representations across different modalities, enforces a grouping of semantically similar instances. To this end, we extend the concept of instance-level contrastive learning with a multimodal clustering step in the training pipeline to capture semantic similarities across modalities. The resulting embedding space enables retrieval of samples across all modalities, even from unseen datasets and different domains. To evaluate our approach, we train our model on the HowTo100M dataset and evaluate its zero-shot retrieval capabilities in two challenging domains, namely text-to-video retrieval, and temporal action localization, showing state-of-the-art results on four different datasets.
Year
DOI
Venue
2021
10.1109/ICCV48922.2021.00791
ICCV
DocType
Citations 
PageRank 
Conference
0
0.34
References 
Authors
0
13
Name
Order
Citations
PageRank
Brian Chen101.35
Andrew Rouditchenko202.03
Kevin Duarte300.34
Hilde Kuehne401.01
Samuel Thomas553646.88
Angie Boggust600.68
Rameswar Panda78514.02
B. Kingsbury84175335.43
Rogério Feris9152989.95
David F. Harwath10638.34
James Glass113123413.63
Michael Picheny121461920.15
Shih-Fu Chang1300.68