Masked Autoencoders Are Scalable Vision Learners (MAE)¶

Authors: Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, Ross Girshick (Facebook AI Research)

Published: November 2021 (arXiv 2111.06377), CVPR 2022

Paper Link: https://arxiv.org/abs/2111.06377

Why This Paper Matters¶

Masked Autoencoders (MAE) brought BERT-style self-supervised learning to vision and made it actually scale. The trick was deceptively simple: randomly hide 75% of an image's patches, then train a Transformer to reconstruct the missing pixels. The result was the first self-supervised vision pretraining method that cleanly beat supervised pretraining on ImageNet - and it did so while training 3x faster than contrastive methods like SimCLR and MoCo.

MAE became the new default for vision pretraining. It also vindicated the broader bet that the same recipe powering NLP (mask things, reconstruct them, scale up) would work for any modality if you got the details right.

The Problem Before MAE¶

In NLP, BERT had shown by 2018 that masked language modeling - predict missing words from context - was a powerful self-supervised objective that produced models you could fine-tune for almost any task. Three years later, vision was still struggling to find an equivalent:

Supervised pretraining on ImageNet was the default but required millions of labels
Contrastive methods (SimCLR, MoCo, BYOL) were strong but complex - needed careful augmentations, big batch sizes, momentum encoders, queues
Earlier masked-image methods (iGPT, BEiT) worked but were slow, complex, or required tokenizers
ViT had brought Transformers to vision but typically still trained with supervised classification

The natural question: why doesn't BERT-style masking just work for images?

Earlier attempts had tried, and the answer was that it kind of did - but not as cleanly as expected. The challenges turned out to be more about engineering than concept:

Information density - pixels have lots of redundancy, so masking a small fraction is too easy a task
Decoder design - what should reconstruct the pixels matters a lot
Compute waste - naively running the encoder on all patches (including masked ones) is expensive

MAE's contribution was solving all three with simple, principled design choices.

The Core Innovation¶

MAE has three key ideas:

Mask a lot - 75% of patches instead of BERT's 15%. Images are redundant, so the task must be harder
Asymmetric encoder-decoder - a heavy encoder that processes only the visible patches, plus a lightweight decoder that handles reconstruction
Reconstruct raw pixels with a simple MSE loss - no tokenizer, no perceptual loss, no adversarial training

Together these make the pretraining task hard enough to learn good representations, fast enough to train at scale, and simple enough to implement in a few hundred lines.

How MAE Works¶

The Pretraining Pipeline¶

Patchify the image into 16x16 patches (like ViT)
Randomly sample 25% of patches as "visible"; mask the other 75%
Encode only the visible patches through a large ViT encoder
Insert learnable mask tokens at the masked positions in the encoder output
Decode the full sequence (visible patches + mask tokens) through a small Transformer decoder
Predict the raw pixel values for each masked patch
Compute MSE loss on the masked patches only

def mae_forward(image):
    patches = patchify(image)                        # (N_patches, patch_dim)

    # 1. Random mask
    visible_idx, masked_idx = random_mask(N_patches, ratio=0.75)
    visible_patches = patches[visible_idx]

    # 2. Encode only visible (huge speedup!)
    visible_tokens = encoder(visible_patches + pos_emb[visible_idx])

    # 3. Re-insert mask tokens at masked positions
    full_sequence = scatter(visible_tokens, mask_token, visible_idx, masked_idx)
    full_sequence = full_sequence + pos_emb

    # 4. Light decoder reconstructs pixels
    pred_patches = decoder(full_sequence)

    # 5. Loss on masked patches only
    loss = mse(pred_patches[masked_idx], patches[masked_idx])
    return loss

Asymmetric Design - The Key to Speed¶

Here's the magic of MAE's asymmetric design:

Encoder sees only 25% of tokens -> 4x less compute, 4x less memory
Decoder sees the full sequence but is much smaller (typically 8 layers vs 24)
Total cost is roughly ⅓ of a symmetric design

This is what makes MAE practical to scale. A ViT-Huge with MAE pretraining costs less than a contrastive ViT-Large.

Why 75% Masking?¶

The paper's most striking ablation is the masking ratio sweep. In BERT, 15% is optimal for text. For images, the optimal is 75% - and even 80-85% works well. Reconstruction is still possible because:

Adjacent patches are highly correlated
Global structure can be inferred from a small subset
The task remains genuinely hard enough that the encoder must learn semantic features, not just local interpolation

This high masking ratio is what makes the pretraining task non-trivial despite pixel redundancy.

Reconstruction Target¶

MAE reconstructs raw normalized pixels. The authors found that per-patch normalization (subtract mean, divide by std within each patch) helps stabilize training. They also tested: - Predicting tokenizer-discretized targets (like BEiT) - slightly worse - Predicting features from another network - comparable but more complex - Predicting raw pixels - simplest, works best

Simplicity won.

Key Results¶

ImageNet Classification (Fine-Tuned)¶

Method	Backbone	Top-1 Accuracy
Supervised (DeiT)	ViT-B	81.8
MoCo v3	ViT-B	83.2
BEiT	ViT-B	83.2
MAE	ViT-B	83.6
Supervised	ViT-L	82.6
MAE	ViT-L	85.9
MAE	ViT-H	86.9
MAE	ViT-H (448)	87.8

MAE-pretrained ViT-Huge reached 87.8% on ImageNet - surpassing all previous methods and showing clear scaling with model size.

Transfer Learning¶

MAE pretraining transferred strongly to: - COCO object detection / instance segmentation: New SOTA when paired with ViT backbones - ADE20K semantic segmentation: Substantial gains over supervised pretraining - iNaturalist, Places: Strong fine-tuning results

A consistent pattern: the bigger the model, the more MAE outperformed alternatives. ViT-B gains were modest; ViT-H gains were dramatic. This is the hallmark of a true scaling method.

Training Efficiency¶

MAE pretrains roughly 3x faster than contrastive methods (MoCo, DINO) with comparable or better results. For ViT-Huge: - 800-epoch MAE pretrain on ImageNet-1K: ~31 hours on 128 TPU v3 cores - Equivalent contrastive pretrain: ~3x longer

Scaling pretraining for free is a huge practical advantage.

Why It Works: The Authors' Hypotheses¶

The paper offers several explanations for MAE's success:

Images are different from text. Words carry dense semantic meaning; pixels are highly redundant. So masking ratios must differ dramatically (75% vs 15%)
The asymmetric design lets the encoder learn full-image semantics without wasting compute on masked positions
Pixel reconstruction is a strong target - much like image-level supervision, but free
Mask tokens stay out of the encoder - they only appear in the lightweight decoder, so the encoder never learns to rely on mask-token shortcuts (a problem that plagues BERT-style image methods that put mask tokens in the encoder)

The encoder is forced to learn representations of visible patches that contain enough information for a lightweight decoder to reconstruct everything else. This pressure produces semantically meaningful features.

Impact and Legacy¶

Self-supervised vision became practical¶

After MAE, generative pretraining became the default approach for large vision Transformers. The contrastive era effectively ended for foundation-scale vision models.

A blueprint for other modalities¶

The MAE recipe - mask aggressively, encode visible only, lightweight decoder, simple reconstruction loss - was quickly adapted to:

Video (VideoMAE, MAE-ST): Mask spacetime tubes; reconstruction yields strong action recognition features
Audio (Audio-MAE): Mask spectrogram patches
Point clouds (Point-MAE): Mask 3D points
Multimodal (MultiMAE, VL-MAE): Mask across modalities simultaneously

Foundation for SAM and modern vision¶

MAE-pretrained ViTs became the standard backbone for many downstream systems, most famously the Segment Anything Model (SAM). The combination of MAE pretraining + large-scale fine-tuning powered much of the next wave of vision research.

Validated the "BERT for X" pattern¶

After GPT and BERT for text, MAE was the cleanest demonstration that the masked-prediction recipe could be ported to any modality with the right adaptations. This pattern now defines self-supervised learning across vision, audio, video, and beyond.

Limitations¶

Reconstruction quality is not the goal - MAE reconstructions are blurry and not photorealistic, but the representations are excellent. The goal is downstream performance, not generation
Pixel target may not be optimal for all tasks - high-frequency texture is hard to predict and may not be the most useful supervision
Linear probing is weak - MAE features need fine-tuning to shine; their linear-probe performance lags contrastive methods
No cross-image learning signal - unlike contrastive methods, MAE doesn't directly learn invariances between augmented views
Compute still substantial - 800-epoch pretraining is expensive even with the asymmetric design

Connections to Other Papers¶

Attention Is All You Need (#1): MAE's encoder and decoder are both Transformers
BERT (#3): The conceptual parent - MAE is "BERT for images" with carefully adapted design choices
Vision Transformer (#11): MAE uses the ViT backbone; without ViT, MAE wouldn't exist
GPT-2 (#64) and GPT-3 (#4): Show generative pretraining for text; MAE is the generative-pretraining-for-vision equivalent
Scaling Laws (#12): MAE's gains compound with model size, consistent with scaling-law expectations
VAE (#57) and VQ-VAE (#89): Both are autoencoder-style methods; MAE is a masked autoencoder that doesn't try to model a latent distribution - just reconstructs
Diffusion Models (#6) and DDPM (#6): Both reconstruct from corrupted inputs; MAE's masking is a discrete, structured corruption rather than continuous noise
DALL-E and CLIP: MAE is sometimes used as the vision tower in multimodal systems; its strong features transfer well

Key Takeaways¶

Mask 75% of patches - far more aggressive than text masking, because images are highly redundant
Asymmetric encoder-decoder - heavy encoder on visible patches only, lightweight decoder on the full sequence; this is what makes scaling cheap
Reconstruct raw pixels - simple MSE works; no tokenizer or adversarial loss needed
Strong scaling - MAE's advantage over alternatives grows with model size, the hallmark of a true foundation pretraining method
A general recipe - MAE's design principles transferred immediately to video, audio, point clouds, and multimodal pretraining, becoming the dominant self-supervised paradigm