The Llama 3 Herd of Models

Chapter 1: Architecture Decisions

Llama 3 uses a standard decoder-only Transformer — the same fundamental architecture as GPT. But every component has been carefully optimized. Meta's philosophy: "Our design philosophy is to use a relatively standard Transformer architecture with minor modifications, rather than developing a bespoke architecture."

Here are the specific architecture choices, each with the engineering rationale:

Grouped Query Attention (GQA)

Standard multi-head attention (MHA) gives each head its own query, key, and value projections. With 128 heads in the 405B model, that's 128 separate key-value pairs per layer — expensive during inference because all KV pairs must be cached.

Grouped Query Attention (Ainslie et al., 2023) shares key-value heads across groups of query heads. Instead of 128 KV heads, Llama 3 uses only 8 KV groups — each group of 16 query heads shares the same keys and values.

python
# Grouped Query Attention — 8 KV heads shared across 128 query heads
import torch
import torch.nn as nn

class GQA(nn.Module):
    def __init__(self, d_model=16384, n_heads=128, n_kv_heads=8):
        super().__init__()
        self.head_dim = d_model // n_heads  # 128
        self.n_heads = n_heads              # 128 query heads
        self.n_kv_heads = n_kv_heads        # 8 KV groups
        self.n_rep = n_heads // n_kv_heads  # 16 queries per KV group

        self.wq = nn.Linear(d_model, n_heads * self.head_dim, bias=False)
        self.wk = nn.Linear(d_model, n_kv_heads * self.head_dim, bias=False)
        self.wv = nn.Linear(d_model, n_kv_heads * self.head_dim, bias=False)
        self.wo = nn.Linear(n_heads * self.head_dim, d_model, bias=False)

    def forward(self, x):
        B, T, _ = x.shape
        q = self.wq(x).view(B, T, self.n_heads, self.head_dim)
        k = self.wk(x).view(B, T, self.n_kv_heads, self.head_dim)
        v = self.wv(x).view(B, T, self.n_kv_heads, self.head_dim)
        # Repeat KV heads to match query heads
        k = k.repeat_interleave(self.n_rep, dim=2)  # [B,T,128,128]
        v = v.repeat_interleave(self.n_rep, dim=2)  # [B,T,128,128]
        # Standard attention from here
        # ...

SwiGLU Feed-Forward Network

The original Transformer uses a two-layer FFN with ReLU: FFN(x) = ReLU(xW₁)W₂. Llama 3 replaces this with SwiGLU (Shazeer 2020), which uses a gated linear unit with the SiLU (Swish) activation:

Where ⊙ is element-wise multiplication and SiLU(x) = x · σ(x) is the Sigmoid Linear Unit. This introduces a third weight matrix (W_gate) but consistently improves quality. The gating mechanism lets the network learn which dimensions to activate, which is more expressive than a simple ReLU.

RoPE (Rotary Position Embeddings)

Instead of learned absolute position embeddings (like BERT) or sinusoidal embeddings (like the original Transformer), Llama 3 uses RoPE (Su et al., 2021). RoPE encodes position by rotating the query and key vectors in 2D subspaces:

Where θ is a frequency vector and rotate() shifts dimensions by one position. The key property: the attention score between positions i and j depends only on their relative distance (i - j), not their absolute positions. This makes the model naturally length-generalizable — it can handle sequences longer than those seen during training.

Other architecture details

Model sizes at a glance

Chapter 2: Pre-training Data Pipeline

Llama 3 was trained on approximately 15 trillion tokens — roughly 10x more than Llama 2 and 5x more than GPT-3. At the 405B scale, this means the model saw each token roughly 1.5 times on average (15T tokens / ~10T unique tokens). The data pipeline that produces these tokens is arguably as important as the model architecture itself.

Data composition

Data quality pipeline

Raw CommonCrawl is noisy — full of spam, duplicated content, toxic text, and low-quality pages. Llama 3's data pipeline applies multiple cleaning stages:

Data mixing and annealing

Not all data is created equal. Meta uses data annealing — changing the data mixture during training:

The annealing phase is particularly clever: in the final stages of training, the learning rate is reduced to near-zero and the data mixture is shifted to emphasize the highest-quality documents. This is analogous to "polishing" a lens — the final refinement that sharpens quality without the risk of large updates destabilizing the model.

Tokenization: 128K BPE vocabulary

Llama 3 quadrupled the vocabulary from 32K (Llama 2) to 128K tokens. This means each piece of text is encoded in fewer tokens, which:

Chapter 3: Training Infrastructure

Training a 405B parameter model on 15T tokens is one of the largest compute operations ever undertaken. Let's understand the engineering required.

Hardware

Llama 3 405B was trained on 16,384 NVIDIA H100 GPUs, organized into clusters connected by high-bandwidth networking:

Parallelism strategy

A 405B model with fp16 weights requires ~810 GB just for the parameters. A single H100 has 80 GB of memory. You can't fit even the weights on one GPU, let alone the optimizer states and activations. The solution: split the model across thousands of GPUs using multiple parallelism strategies simultaneously.

Training stability

At this scale, hardware failures are the norm, not the exception. With 16K GPUs running for 54 days:

Meta reports that during the 405B training run, they encountered 466 job interruptions, of which 78% were due to unexpected hardware issues. Their automated recovery system was able to restart from the latest checkpoint within minutes, achieving an effective training uptime of ~90%.

Chapter 4: Scaling Laws & Training Recipe

How do you decide whether to train an 8B, 70B, or 405B model? How many tokens should each see? How do you set the learning rate and batch size? Meta used scaling laws — empirical relationships between model size, data size, and compute — to answer these questions before committing to the full training runs.

The scaling law methodology

Meta trained hundreds of small models (40M to 16B parameters) on varying amounts of data and measured their validation loss. They then fit power-law curves to predict the performance of larger models:

Where N is the number of parameters, D is the number of training tokens, A, B, α, β are fitted constants, and L_∞ is the irreducible loss (the best possible loss with infinite data and model size). This equation has three terms:

Training recipe

Learning rate schedule

Chapter 5: Post-Training

Pre-training produces a powerful but "raw" model — it can complete text fluently but doesn't know how to follow instructions, refuse harmful requests, or engage in helpful dialogue. Post-training transforms this base model into an aligned assistant through multiple stages of fine-tuning.

The post-training pipeline

Stage 1: SFT — Learning to follow instructions

SFT uses carefully curated prompt-response pairs. Meta emphasizes quality over quantity: they found that 27,540 high-quality examples outperformed 10x more lower-quality examples. The SFT data covers:

Stage 2: Reward model

The reward model is initialized from the pre-trained base model and trained to predict which of two responses a human annotator would prefer. Given a prompt x and two responses y_w (preferred) and y_l (rejected):

Where r(x, y) is the scalar reward score and σ is the sigmoid function. This is the Bradley-Terry model of pairwise preferences — the same used in chess ELO ratings.

Stage 3: DPO — Direct Preference Optimization

Instead of training a separate reward model and using PPO (as in RLHF), Llama 3 primarily uses DPO (Rafailov et al., 2023), which directly optimizes the policy using preference data:

DPO avoids the instability of PPO training and is simpler to implement. The key idea: instead of fitting a reward model and then optimizing against it, DPO uses the language model itself as an implicit reward model.

Rejection sampling

In addition to DPO, Meta uses rejection sampling: generate K responses from the current model, score each with the reward model, and keep only the best one for SFT. This is simpler than DPO and provides high-quality training examples for the next round of SFT.

python
# Rejection sampling: generate K responses, keep the best
def rejection_sample(model, reward_model, prompt, K=16):
    responses = [model.generate(prompt, temperature=0.7) for _ in range(K)]
    scores = [reward_model.score(prompt, r) for r in responses]
    best_idx = max(range(K), key=lambda i: scores[i])
    return responses[best_idx]  # highest-scoring response

# This is compute-intensive but produces very high quality
# training examples for the next SFT round

Chapter 6: Multimodal Extensions

The Llama 3 paper describes experiments extending the language model to handle images, video, and speech — turning it into a multimodal model. These extensions follow a common pattern: keep the pre-trained language model mostly frozen and train adapter modules that project other modalities into the language model's embedding space.

Image understanding

The vision extension uses a pre-trained image encoder (ViT-H) connected to the language model via a learned projection layer:

The cross-attention adapter is key: instead of simply projecting image features through an MLP (as in LLaVA), Llama 3 uses cross-attention layers inserted between Transformer blocks. The image tokens serve as keys and values, while the text tokens serve as queries. This gives the language model fine-grained access to spatial information in the image.

Video understanding

Video is handled by sampling frames uniformly, encoding each with the image encoder, and concatenating the resulting tokens. For a 30-second video at 1 fps, this produces 30 × N_patches tokens — which quickly consumes the context window. Meta addresses this with temporal aggregation: adjacent frames' features are pooled to reduce the token count.

Speech understanding

Speech is encoded using a separate audio encoder (trained from scratch) that converts waveforms into discrete token sequences. These are interleaved with text tokens, allowing the model to understand and respond to spoken input.

Multimodal results

Chapter 7: Safety

A 405B open-weight model can be used for anything — including harmful purposes. Meta addresses this through multiple layers of safety measures, while acknowledging that open models present fundamentally different safety challenges than API-only models.

Llama Guard

Llama Guard is a separate safety classifier (based on Llama 3 8B) that classifies inputs and outputs into safe/unsafe categories. It acts as a guardrail that can be deployed alongside the main model:

Llama Guard classifies across these safety categories:

Safety training in post-training

Beyond Llama Guard (which is an external classifier), safety is also baked into the model itself during post-training:

Chapter 8: Model Explorer

Let's put it all together. This interactive explorer lets you compare all three Llama 3 model sizes across benchmarks, explore the architecture at different scales, and see how the training pipeline transforms a base model into an aligned assistant.

Benchmark results

Parameter efficiency across scales

How efficiently does each model use its parameters? We can measure this as benchmark score per billion parameters:

The 8B model is by far the most parameter-efficient — each additional billion parameters yields diminishing returns. This is why the 8B model is the most popular for deployment: it offers 80% of the 405B's quality at 2% of the inference cost.

Chapter 9: Connections

Llama 3 represents the culmination of years of research in scaling, training recipes, and alignment. Let's place it in the broader context.

What Llama 3 builds on

The open model ecosystem

Key innovations and their impact

The future trajectory

The Llama series shows a clear trajectory: each generation gets more data (1.4T → 2T → 15T), larger models (65B → 70B → 405B), and more sophisticated post-training (no RLHF → RLHF → iterative DPO). If this trend continues, Llama 4 will likely feature even more training data (50T+?), longer context (1M+?), native multimodality, and potentially mixture-of-experts architectures for efficiency.

"Our experience suggests that it is possible to train models at the frontier of AI capabilities using a standard, dense Transformer architecture. Neither mixture-of-experts models, nor a novel architecture, are required."
— Llama 3 Technical Report, Meta AI