CM3Leon: Autoregressive Multimodal

Chapter 1: CM3 Foundation

CM3Leon builds on the CM3 objective — Causally Masked Multimodal Modeling. This is a training objective that combines autoregressive generation with infilling (masked span prediction), enabling the model to both generate and complete multimodal content.

The CM3 objective

Standard autoregressive models predict left-to-right: given all previous tokens, predict the next one. CM3 adds infilling: randomly mask spans of tokens, move them to the end, and predict them. This trains the model to generate content conditioned on context from BOTH sides.

This is crucial for image generation: given surrounding text ("A sunset over the ocean [IMAGE] Beautiful colors"), the model can generate the image tokens by infilling the masked span. Without infilling, the model could only generate images at the end of a sequence.

Architecture

python
# CM3 masking strategy
def cm3_mask(tokens, mask_ratio=0.15):
    # Randomly select spans to mask
    n_mask = int(len(tokens) * mask_ratio)
    mask_start = random.randint(0, len(tokens) - n_mask)
    masked_span = tokens[mask_start:mask_start + n_mask]

    # Replace masked tokens with sentinel
    input_tokens = tokens[:mask_start] + [MASK_TOKEN] + tokens[mask_start + n_mask:]

    # Target: predict the masked span
    target_tokens = masked_span
    return input_tokens, target_tokens

Chapter 2: Retrieval Augmentation

CM3Leon's biggest innovation over previous autoregressive multimodal models is retrieval-augmented pretraining. During training, each document is augmented with retrieved similar documents from a large corpus. This dramatically improves data efficiency.

How retrieval works in training

The retriever uses CLIP embeddings for both text and images. For each training document, CM3Leon retrieves the top-2 most similar documents from a 1.4 billion example corpus. The retrieved documents are prepended to the training input, providing the model with relevant examples it can attend to.

Why retrieval helps so much

python
# CM3Leon retrieval-augmented training
def create_training_example(doc, retriever, corpus):
    # Encode document with CLIP
    doc_embedding = clip.encode(doc)  # text + image features

    # Retrieve top-2 similar documents
    retrieved = retriever.search(doc_embedding, k=2)

    # Prepend retrieved docs to training input
    input_seq = (
        tokenize(retrieved[0]) +   # retrieved doc 1
        tokenize(retrieved[1]) +   # retrieved doc 2
        tokenize(doc)              # original document
    )

    # Apply CM3 masking and predict
    masked_input, target = cm3_mask(input_seq)
    return masked_input, target

Chapter 3: Training Recipe

CM3Leon's training follows the scaling laws from the previous paper in this series (Aghajanyan et al. 2023). The mix ratio and training details are carefully chosen based on those findings.

Pretraining data

CM3Leon uses licensed data from Shutterstock, not scraped web data. This is a notable ethical choice — the model is trained on data where creators were compensated. The quality of licensed data is also generally higher than web-scraped data.

Training details

python
# CM3Leon training configuration
config = {
    "model": "decoder-only transformer",
    "params": "7B",
    "optimizer": "AdamW",
    "lr": 1e-4,
    "warmup": 1500,
    "schedule": "cosine",
    "tokens": 1_400_000_000_000,  # 1.4T total tokens seen
    "context": 4096,
    "batch_size": 8192,  # sequences
    "retrieval_k": 2,   # 2 retrieved docs per training example
    "image_tokens": 256,
    "data_mix": "~70% text, ~30% image (following scaling laws)",
}

Chapter 4: Decoding Strategies

CM3Leon's second key innovation is bringing classifier-free guidance (CFG) to autoregressive generation. CFG was previously exclusive to diffusion models, where it dramatically improves sample quality at the cost of diversity. CM3Leon shows it works for autoregressive models too.

What is classifier-free guidance?

The idea: during generation, compute two predictions — one conditioned on the prompt (conditional) and one without the prompt (unconditional). Then amplify the difference:

Where α is the guidance scale. At α = 1, you get standard conditional generation. At α > 1, the model is pushed to generate outputs more strongly aligned with the prompt. At α = 0, you get unconditional generation.

CFG for autoregressive models

To enable CFG, CM3Leon occasionally drops the conditioning information during training (sets the prompt to empty with 10% probability). This trains the model to generate both conditionally and unconditionally. At inference time, it runs two forward passes per token and combines them:

python
# Classifier-free guidance for autoregressive generation
def generate_with_cfg(model, prompt, alpha=3.0):
    for step in range(n_tokens):
        # Conditional: model sees the text prompt
        logits_cond = model(prompt + generated_so_far)

        # Unconditional: model sees empty prompt
        logits_uncond = model("" + generated_so_far)

        # Guide: amplify the conditional signal
        logits = logits_uncond + alpha * (logits_cond - logits_uncond)

        # Sample next token from guided logits
        next_token = sample(softmax(logits / temperature))
        generated_so_far.append(next_token)
    return generated_so_far

TopP + TopK filtering

CM3Leon also uses sophisticated decoding filters. The paper finds that combining TopP (nucleus sampling) with TopK gives the best results:

Chapter 5: Instruction Tuning

After pretraining, CM3Leon undergoes supervised fine-tuning (SFT) on a diverse set of multimodal tasks. This is the same idea as instruction tuning for text LLMs (InstructGPT, FLAN), but extended to mixed-modal tasks.

Task diversity

CM3Leon is fine-tuned on tasks that cover all four modality combinations:

Instruction format

Each fine-tuning example is formatted as a natural language instruction:

python
# Instruction tuning examples

# Text→Image
{"instruction": "Generate an image of a sunset over mountains",
 "output": "[IMAGE_TOKENS]"}

# Image→Text
{"instruction": "Describe this image in detail: [IMAGE_TOKENS]",
 "output": "A golden retriever playing fetch in a park..."}

# Visual QA
{"instruction": "[IMAGE_TOKENS] How many people are in this image?",
 "output": "Three"}

# Image Editing
{"instruction": "[IMAGE_TOKENS] Make this image look like it was painted by Monet",
 "output": "[EDITED_IMAGE_TOKENS]"}

Chapter 6: Results & Showcase

CM3Leon achieves state-of-the-art results for autoregressive image generation and competitive results across a wide range of multimodal tasks — all from a single 7B model.

Image generation

Multimodal understanding

After instruction tuning, CM3Leon also performs well on vision-language tasks:

CM3Leon is competitive on understanding tasks despite being primarily optimized for generation. It trails Flamingo on some benchmarks but outperforms it on TextVQA (which requires reading text in images).

Chapter 7: Connections

CM3Leon sits between RA-CM3 (which introduced retrieval for multimodal models) and Chameleon (which took the early-fusion approach to scale). Its techniques directly influenced both Chameleon and Transfusion.