Name
Abstract | ||
|---|---|---|
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior. |
Year | DOI | Venue |
|---|---|---|
2022 | 10.1109/CVPR52688.2022.01553 | IEEE Conference on Computer Vision and Pattern Recognition |
Keywords | DocType | Volume |
Representation learning, Self-& semi-& meta- & unsupervised learning | Conference | 2022 |
Issue | Citations | PageRank |
1 | 0 | 0.34 |
References | Authors | |
0 | 6 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Kaiming He | 1 | 21469 | 696.72 |
| Xinlei Chen | 2 | 853 | 38.03 |
| Saining Xie | 3 | 231 | 12.45 |
| Yanghao Li | 4 | 0 | 0.34 |
| Piotr Dollár | 5 | 7999 | 307.07 |
| Ross B. Girshick | 6 | 21921 | 927.22 |