Scaling Laws for Mixed-Modal Models

Chapter 1: Scaling Laws Primer

Before we can understand mixed-modal scaling, we need to understand what scaling laws are and why they're so useful.

The Chinchilla revelation

In 2022, Hoffmann et al. (DeepMind) showed that language model loss follows a remarkably clean power law:

Where N is model parameters, D is training tokens, A, B, E are constants, and α, β are scaling exponents. This says: loss decreases as a power of both model size and data size, with an irreducible entropy floor E.

On a log-log plot, power laws appear as straight lines. This is what makes them so useful — you can fit a line through small-scale experiments and extrapolate to predict large-scale performance.

What this paper asks

Do the same power laws hold when you mix text and images? Specifically:

python
# Power law: L = A * N^(-alpha) + E
# On a log-log plot, this is a straight line (slope = -alpha)

import numpy as np

# Example: text-only scaling
N_values = [125e6, 350e6, 760e6, 1.3e9, 6.7e9]  # model sizes
losses =   [3.2,   2.9,   2.7,   2.5,   2.2]   # validation losses

# Fit: log(L - E) = log(A) - alpha * log(N)
# Slope of this line gives alpha ≈ 0.076 for text

Chapter 2: The CM3 Architecture

The paper studies scaling laws on a specific model family called CM3 (Causally Masked Multimodal Model). Understanding CM3 is necessary because the scaling laws are measured on this architecture.

What is CM3?

CM3 is a decoder-only transformer that processes interleaved text and images. Images are tokenized using a VQ-VAE (like Chameleon). The key architectural choice: CM3 uses causal masking with an important twist — it can also do infilling (masked span prediction).

Model sizes studied

For each model size, the paper trains multiple variants with different text/image ratios (from 0% to 100% image data). This gives them enough data points to fit scaling laws across both axes: model size AND data mix.

Chapter 3: Text Scaling Laws

The first major finding: text loss in mixed-modal models still follows a power law in model size, but the exponent and constant depend on the image data proportion.

The text scaling equation

Where ρ is the fraction of image data (0 = text-only, 1 = image-only). Notice that A, α, and E all depend on ρ. This means the scaling behavior changes as you add more images.

Key finding: text loss is U-shaped in ρ

Adding a small amount of image data (up to ~20-30%) actually improves text loss compared to training on text alone! This is surprising. You'd expect that replacing some text data with images would always hurt text quality (less text to learn from). But the model extracts useful information from images that transfers to text understanding.

However, beyond ~30-40% images, text quality degrades. The model spends too many of its parameters on image processing, leaving insufficient capacity for text.

python
# Text loss as a function of image fraction ρ
# Measured at different model sizes

# At 125M params:
#   ρ=0.0: L_text = 3.21 (text-only baseline)
#   ρ=0.1: L_text = 3.18 (BETTER than text-only!)
#   ρ=0.2: L_text = 3.15 (optimal for 125M)
#   ρ=0.3: L_text = 3.19 (slightly worse)
#   ρ=0.5: L_text = 3.35 (significantly worse)

# At 6.7B params:
#   ρ=0.0: L_text = 2.21
#   ρ=0.1: L_text = 2.18
#   ρ=0.3: L_text = 2.14 (optimal for 6.7B — shifted right!)
#   ρ=0.5: L_text = 2.22

Chapter 4: Image Scaling Laws

Image loss also follows power laws, but with different exponents than text. Images scale faster with model size — meaning that bigger models get disproportionately better at images relative to text.

The image scaling equation

The key difference from text: α_img > α_text. The image scaling exponent is larger, meaning image loss decreases faster with model size. Intuitively: images have more "headroom" for improvement. Text models at 6.7B are already quite good; image models at 6.7B are still far from ceiling.

Image scaling exponents

Image quality also depends on text data

Just as images help text, text helps images. Models trained with some text data generate better images than pure image models. The text provides semantic scaffolding that helps the model understand what the image should contain.

Chapter 5: Optimal Mixing

Now we get to the paper's most practical contribution: given a fixed compute budget C, what fraction ρ of image data minimizes the total loss?

The compute-optimal mixing formula

The total loss is a weighted combination of text and image losses:

Where D · (1-ρ) is the number of text tokens and D · ρ is the number of image tokens. The total compute C ≈ 6ND (standard approximation), so fixing C means larger N requires smaller D.

Optimal ρ as a function of N

The paper finds the optimal ρ by minimizing L_total over ρ at each model size. The results:

This is exactly what Chameleon did: at 34B parameters, they used ~40% image data. This paper's scaling law predicted that ~35-40% would be optimal at that scale — and Chameleon's empirical results confirmed it.

Chapter 6: Cross-Modal Transfer

Perhaps the paper's most fascinating finding is cross-modal transfer: training on images makes the model better at text, and training on text makes the model better at images. This isn't just a neutral sharing of capacity — it's a genuine synergy.

Evidence of positive transfer

If cross-modal training were purely competitive (each modality taking capacity from the other), then the optimal image fraction for text loss would be 0%. But empirically, text loss is minimized at ρ ≈ 15-30%, not ρ = 0. Something about image data genuinely helps text modeling.

Hypotheses for why transfer occurs

When transfer breaks down

Beyond the sweet spot, adding more images hurts text performance. This happens when the model runs out of capacity — it literally doesn't have enough parameters to maintain good text features while also learning image features. The capacity bottleneck is in the FFN layers, which is why MoT (separate FFNs per modality) largely eliminates this problem.

Chapter 7: Connections

This paper laid the theoretical foundation for every major mixed-modal model that followed. Its scaling laws directly influenced training recipes at Meta and beyond.