Shrinking the Giant:
Model Compression for LLMs

Chapter 1: The Taxonomy

Faced with "make the model smaller," there are only a handful of fundamentally different things you can do to a weight matrix. The survey organizes the entire field into four families. The trick to remembering them is to ask: what am I attacking?

One cross-cutting distinction: do you have to retrain?

This is the question that decides whether a technique is practical for a 70-billion-parameter model. The survey splits the four families along a second axis: task-agnostic (apply once, works for everything) vs task-based (needs retraining, often per-task).

Chapter 2: Quantization — Fewer Bits Per Weight

Start with the simplest, most-used idea. Quantization reduces the number of bits used to represent each parameter. A weight stored as a 16-bit float becomes an 8-bit, 4-bit, or even 2-bit integer. The survey's one-line definition: "reducing the number of bits (i.e., precision) in the parameters of the model with minimal loss in inference performance."

The mechanics: float → integer

You can't just truncate a float. You map a range of real values onto a small set of integer levels. For symmetric INT-b quantization of a weight block with maximum absolute value max|w|:

Symbol by symbol, with intuition:

• b — the bit-width (e.g. 4). It sets how many levels you have: an INT4 has 2⁴=16 levels.
• max|w| — the largest magnitude in the block. This is the "ruler length" you must fit into the integer grid.
• scale — the size of one integer step in real units. Think of it as the spacing of tick marks on a ruler. Bigger range or fewer bits = coarser ticks = more rounding error.
• w_q — the stored integer. To use it, you dequantize: w ≈ w_q × scale.

A fully worked numerical example

Quantize the weight block [0.10, −0.40, 0.95, 0.02] to INT4 (symmetric, so levels run −7…+7, i.e. 2³−1=7).

Two lessons jump out. The outlier 0.95 is preserved exactly (it defined the scale). But the small weight 0.02 got rounded to zero — it had no level to land on. INT4 gave us a 4× memory cut (16 bits → 4 bits) at the cost of erasing the smallest weights. That is the trade in miniature.

Blue dots are the true weights; the horizontal grid lines are the quantization levels; orange dots show where each weight snaps after rounding. Lower the bits to watch the grid coarsen. Hit Add an outlier and see how one extreme weight stretches the grid and ruins resolution for all the small weights — the motivation for Chapter 3.

QAT vs PTQ — the retraining axis

The survey's first split inside quantization:

• Post-Training Quantization (PTQ): quantize a finished model directly, using only a small calibration set to choose good scales. No gradients. Fast. The workhorse for LLMs.
• Quantization-Aware Training (QAT): simulate quantization during training so the network learns weights robust to the rounding. Recovers more accuracy at very low bits (LLM-QAT, BitDistiller at sub-4-bit, OneBit at literally 1 bit) — but retraining a billion-param model is brutally expensive. The survey's fix: combine QAT with PEFT (e.g. QLoRA) so you only train a tiny adapter.

Chapter 3: PTQ Variants & the Outlier Problem

Chapter 2 ended on a villain: a single outlier weight stretches the scale and crushes resolution for everyone else. In LLMs this is not rare — it is the defining challenge of low-bit quantization, and the survey organizes PTQ around three targets and the outlier fight.

Three things you can quantize

The clever weight-only tricks

SmoothQuant: move the difficulty where it's easy

Here is the most elegant idea in the chapter. Activations have brutal outliers; weights are smooth and easy to quantize. SmoothQuant's insight: a linear layer computes Y = X W, and you can rescale by any diagonal s without changing the math:

Choose s to shrink the activation outliers (dividing X by s) while only mildly growing the weights (multiplying W by s). You have migrated the quantization difficulty from activations to weights — where it is much more tolerable. Same output, easier-to-quantize tensors.

Left column: activation channel magnitudes (note the giant outlier). Right column: weight channel magnitudes (smooth). Drag α to migrate difficulty: the activation outlier shrinks while the weights grow a little. Watch the "max range" readouts — at the right α both are comfortably quantizable, and the product X·W is unchanged.

Concept + Realization: the weight-only compute penalty

Chapter 4: Pruning — Deleting Weights

Quantization keeps every weight but stores it cheaply. Pruning takes the opposite tack: identify weights that don't matter and set them to zero, so you can skip them entirely. The catch is in the word "matter" — and in what shape the surviving weights form, because hardware only goes fast on certain shapes.

Three granularities — the central spectrum

How do you decide which weight is "unimportant"?

Naive answer: prune the smallest-magnitude weights (a small number contributes little). But the breakthrough method, Wanda, found magnitude alone is wrong for LLMs. Its importance score multiplies weight magnitude by the norm of the corresponding input activation:

Decode it: |w_ij| is how big the weight is; ‖X_j‖₂ is how large the activations flowing into that weight typically are. A small weight that always multiplies a huge activation still matters a lot; a large weight that only ever sees tiny activations doesn't. Importance = weight × how-much-it's-used. No retraining, no weight updates — and it matches the far more expensive SparseGPT.

A worked example: 2:4 pruning with Wanda

Take a group of 4 weights with their input-activation norms. Keep the 2 highest scores, zero the rest.

Look at the second row: the smallest weight (0.1) gets the highest score and survives, because it consistently multiplies a giant activation. A pure-magnitude scheme would have deleted it and kept the larger-but-rarely-used 0.6. That activation term is why Wanda works on LLMs where plain magnitude pruning collapses.

A weight matrix split into 2:4 groups (every 4 weights keeps 2). Brightness encodes the activation norm of each weight's input. Toggle the metric: under magnitude only, big weights survive; under Wanda, small weights that see large activations (bright) survive instead. The kept set differs — and Wanda's keeps more of what the model actually uses.

The big result

Chapter 5: Knowledge Distillation

The previous two families shrink an existing model in place. Knowledge distillation (KD) does something different: it trains a brand-new, smaller model — the student — to imitate a large, capable model — the teacher. The student ends up small and fast, but punches above its weight because it learned from the giant.

The two regimes: can you see inside the teacher?

Black-box: distilling emergent abilities

The clever black-box methods don't just copy answers — they copy capabilities that only emerge at scale:

• Chain-of-Thought distillation: prompt the big teacher to produce step-by-step reasoning, then train the small student on those rationales. A 1B student learns to "show its work" by imitating a 175B's reasoning traces.
• In-context-learning distillation: transfer the ability to learn from examples in the prompt.
• Instruction-following distillation: e.g. Lion generates "hard" instructions to stress-train the student; LaMini-LM uses 2.58M instructions.

White-box: the reverse-KL insight (MINILLM)

The standard distillation loss is forward KL, KL(p_teacher ∥ p_student). MINILLM showed this is wrong for generative LLMs and uses reverse KL instead, KL(p_student ∥ p_teacher). The difference is everything:

The teacher (gray) is a two-bump distribution. The student (teal) is a single bump with limited capacity — it cannot cover both modes. Under forward KL it splays out to cover both and lands in the empty valley (bad samples). Under reverse KL it commits to the dominant mode (reliable samples). This is the core MINILLM argument made visual.

Chapter 6: Low-Rank Factorization

The fourth family attacks the matrix's shape. A weight matrix W is m × n. If it has redundant structure — and trained weight matrices usually do — you can approximate it as the product of two much skinnier matrices:

Define the symbols:

• m, n — the original matrix dimensions (e.g. 4096 × 4096).
• k — the rank: the number of "ingredients" you keep. Small k = aggressive compression.
• U, V — the two skinny factors. Storing them costs k(m+n) numbers instead of mn.

The compression math, worked

Take a 4096 × 4096 weight matrix and approximate it with rank k = 256:

An 8× reduction — if rank 256 captures enough of W. And the win is double: you also do fewer multiplies at inference, because Wx becomes U(Vx) — two cheap skinny matmuls instead of one fat one.

The LLM-specific twist: activation-aware (ASVD)

Plain SVD minimizes the error in W itself. But the survey notes ASVD's key finding: what matters is the error in the output Wx, and the activation distribution x is wildly uneven across channels. ASVD scales W by a diagonal matrix reflecting activation magnitudes before the SVD — so the decomposition spends its precious rank on the directions the activations actually probe. Same pattern as AWQ and SmoothQuant: let the data, not the weights alone, decide what to protect.

Left: a 64×64 weight image. Right: its rank-k reconstruction using the top-k singular values. The bar strip shows the singular-value spectrum (which "ingredients" you're keeping in teal vs discarding in gray). Drag k: at low rank you keep only the dominant structure (high compression, blurry); raise it and the reconstruction sharpens while compression drops. The fast decay of the spectrum is why low-rank works.

Chapter 7: The Trade-off — Compare Them Side by Side (the showcase)

You now know all four families. The real engineering question is never "which one is best?" — it's "given my constraint (a memory budget, a latency target, a hardware platform), which technique gets me there with the least accuracy lost?" This chapter is the survey's comparison tables made interactive.

Each dot is a real method (hover-free: labels shown when space allows). X-axis = compression ratio (right = smaller model); Y-axis = perplexity increase (down = better, lower quality loss). Toggle families on/off. Drag the budget line: everything below it meets your quality bar — among those, the rightmost dot is your best compression. This is how you actually pick a method.

Reading the landscape — the rules of thumb

Chapter 8: Metrics & Benchmarks

You cannot pick the rightmost dot in Chapter 7 unless you agree on how to measure both axes. The survey devotes a section to this — and getting the measurement right is where a lot of compression "wins" quietly fall apart.

The efficiency metrics — what "smaller/faster" actually means

The quality benchmarks — what "still good" means

• Perplexity on WikiText-2, C4, PTB — the survey's primary quality axis. Lower = better; compression reports the increase (Δppl). A tiny Δppl is the gold standard.
• Zero-shot ability: LAMBADA, PIQA, OpenBookQA — does the compressed model still answer without examples?
• Reasoning: GSM8K, CommonsenseQA, StrategyQA — the abilities most likely to break under aggressive compression.
• Broad suites: BIG-Bench (200+ tasks), EleutherAI LM Harness (60+ benchmarks) for reproducible, comparable evaluation.
• Unseen instructions (Vicuna-Instructions, User-Oriented): does the compressed model generalize to new tasks?

A simplified roofline. The diagonal is the memory-bandwidth ceiling; the flat top is the compute ceiling. The dot is your workload's arithmetic intensity. When it sits on the diagonal (low intensity, as in LLM decoding) you are bandwidth-bound — cutting FLOPs slides you left along the roof but achievable performance doesn't rise. Lower the bandwidth limit to see the dot drop: that's why reducing memory traffic, not FLOPs, is what speeds up LLMs.

Chapter 9: The Frontier — Open Problems

The survey closes by admitting the field "is still in its early stages." The honest open problems are as instructive as the methods — they tell you where the next breakthroughs (and the current failure modes) live.

Calibration data: the quiet make-or-break

Chapter 10: Connections & Cheat Sheet

This survey is a map of an entire subfield. The deeper you go into any one box, the more you find a dedicated lesson. Here's the whole picture and where to go next.

The cheat sheet — every key idea in one table

The core algorithm as runnable code

# The three weight-touching techniques in one place
import torch

def quantize(W, bits=4):
    """Symmetric per-tensor INT quantization."""
    qmax = 2**(bits-1) - 1
    scale = W.abs().max() / qmax
    Wq = torch.round(W / scale).clamp(-qmax, qmax)
    return Wq * scale            # dequantized (sim'd low-bit)

def wanda_prune_2of4(W, X):
    """2:4 sparsity, Wanda importance = |w| * ||x||."""
    act_norm = X.norm(dim=0)               # per-input-channel
    score = W.abs() * act_norm[None, :]   # [out, in]
    score = score.reshape(W.shape[0], -1, 4)
    keep = score.argsort(dim=-1)[..., 2:]    # top-2 of every 4
    mask = torch.zeros_like(score, dtype=torch.bool).scatter_(-1, keep, True)
    return (W.reshape(W.shape[0], -1, 4) * mask).reshape(W.shape)

def low_rank(W, k=256):
    """Best rank-k via SVD (Eckart-Young)."""
    U, S, Vh = torch.linalg.svd(W, full_matrices=False)
    return (U[:, :k] * S[:k]) @ Vh[:k]   # == U' V', stores k(m+n) numbers

The one-paragraph summary you could give on a whiteboard

An LLM is too big to deploy because its weights eat memory (175B params × 2 bytes = 350 GB). Four families fix this. Quantization stores each weight in fewer bits (16→4), bottlenecked by outlier weights/activations — solved by mixed precision (LLM.int8()), salient-weight protection (AWQ), or difficulty-migration (SmoothQuant). Pruning zeros unimportant weights — importance = |w|·‖activation‖ (Wanda), shaped as unstructured/2:4/structured to trade accuracy against hardware speed (SparseGPT prunes 50% one-shot). Distillation trains a small student to copy a big teacher, using reverse-KL so the capacity-limited student commits to confident modes (MINILLM). Low-rank factorization replaces W with U·V via top-k SVD (8× at rank 256), activation-aware (ASVD). You pick among them by plotting compression vs perplexity-loss under your hardware constraint — and the biggest wins stack several together (QLoRA = quant + low-rank).