Abstract

Masked image modeling has been demonstrated as a powerful pretext task for generating robust representations that can be effectively generalized across multiple downstream tasks. Typically, this approach involves randomly masking patches (tokens) in input images, with the masking strategy remaining unchanged during training. In this paper, we propose a curriculum learning approach that updates the masking strategy to continually increase the complexity of the self-supervised reconstruction task. We conjecture that, by gradually increasing the task complexity, the model can learn more sophisticated and transferable representations. To facilitate this, we introduce a novel learnable masking module that possesses the capability to generate masks of different complexities, and integrate the proposed module into masked autoencoders (MAE). Our module is jointly trained with the MAE, while adjusting its behavior during training, transitioning from a partner to the MAE (optimizing the same reconstruction loss) to an adversary (optimizing the opposite loss), while passing through a neutral state. The transition between these behaviors is smooth, being regulated by a factor that is multiplied with the reconstruction loss of the masking module. The resulting training procedure generates an easy-to-hard curriculum. We train our Curriculum-Learned Masked Autoencoder (CL-MAE) on ImageNet and show that it exhibits superior representation learning capabilities compared to MAE. The empirical results on five downstream tasks confirm our conjecture, demonstrating that curriculum learning can be successfully used to self-supervise masked autoencoders. We release our code at https://github.com/ristea/cl-mae.
OriginalsprogEngelsk
Titel2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Antal sider11
ForlagIEEE
Publikationsdato2024
Sider2480-2490
Artikelnummer10484320
ISBN (Trykt)979-8-3503-1893-7
ISBN (Elektronisk)979-8-3503-1892-0
DOI
StatusUdgivet - 2024
Begivenhed2024 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024 - Waikoloa Beach Marriott Resort, Waikoloa, USA
Varighed: 4 jan. 20248 jan. 2024
https://wacv2024.thecvf.com/

Konference

Konference2024 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024
LokationWaikoloa Beach Marriott Resort
Land/OmrådeUSA
ByWaikoloa
Periode04/01/202408/01/2024
Internetadresse
NavnIEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
ISSN2642-9381

Fingeraftryk

Dyk ned i forskningsemnerne om 'CL-MAE: Curriculum-Learned Masked Autoencoders'. Sammen danner de et unikt fingeraftryk.

Citationsformater