Model Compression

The error (0.002) is less than half the scale (0.004). This is guaranteed — the worst-case error for any value is scale/2, achieved when the value falls exactly between two bins.

Symmetric quantization uses 127 levels per side (not 128) to ensure that zero maps exactly to integer 0 with no rounding error. This is important for zero-valued activations (ReLU outputs) and skip connections.

Asymmetric sets the range to exactly [−0.8, 1.2], distributing all 256 levels across 2.0 units of range. Scale = 2.0/255 ≈ 0.00784.

Symmetric would set absmax = max(0.8, 1.2) = 1.2, giving range [−1.2, 1.2]. That's 2.4 units, but values below −0.8 never appear — 0.4 units of range (17% of the negative side) are wasted. Scale = 1.2/127 ≈ 0.00945 — coarser bins, more error.

Asymmetric wins when the distribution is notably skewed. Symmetric wins on simplicity (no zero-point bookkeeping) and is preferred when the distribution is roughly centered around zero, which is common for well-trained weights.

For INT8, the max error is tiny (~0.4% of the range). But for INT4 (only 16 levels), scale = 1.0/7 = 0.143, and max error = 0.071 — that's 7% of the range. This is why INT4 quantization requires more sophisticated techniques (group quantization, calibration) to maintain accuracy.

javascript
function quantize(x, bits, minVal, maxVal) {
  const levels = Math.pow(2, bits) - 1;
  const scale = (maxVal - minVal) / levels;
  let q = Math.round((x - minVal) / scale);
  q = Math.max(0, Math.min(levels, q));  // clamp
  return q * scale + minVal;
}

The clamp step is critical — without it, values slightly outside [minVal, maxVal] would produce out-of-range integers that can't be stored in the target bit-width. In practice, calibration determines minVal and maxVal from real data, and outliers get clipped.

Line 4 is the bug. The dequantize formula is wrong in two ways: it adds zeroPoint instead of subtracting it, and then multiplies. The correct formula is (q − zeroPoint) * scale.

The quantize step on line 2 does: q = round(x/scale + zp). To invert it: x̂ = (q − zp) × scale. The current code does q × scale + zp, which doesn't undo the original transformation.

The overhead is negligible — less than 0.05%. This is why per-channel quantization is essentially free: you pay a rounding error in storage for a massive improvement in accuracy. Every serious PTQ method uses per-channel (or finer) granularity.

Imagine two channels: channel A has weights in [−0.01, 0.01] and channel B has weights in [−2.0, 2.0]. Per-tensor uses one scale for both: scale = 4.0/255 ≈ 0.0157. Channel A, with a range of only 0.02, gets mapped to just ~1 quantization level — almost all information is destroyed.

Per-channel gives channel A its own scale: 0.02/255 ≈ 0.000078, spreading its values across all 256 levels. Channel B gets its own scale: 4.0/255 ≈ 0.0157. Both channels use the full resolution of INT8.

The bit-width is the same. The total storage is nearly the same. But the effective precision per channel improves dramatically.

At group_size=128, the overhead is about 6% — still small, but no longer negligible. Smaller groups (64, 32) give better accuracy but higher overhead. Group size 128 is the standard trade-off used in GPTQ and AWQ. Some aggressive schemes use group_size=32 for 4-bit, accepting ~23% overhead for better accuracy.

GPTQ minimizes layer-wise output reconstruction error: for each layer, it finds quantized weights W̃ that minimize ‖WX − W̃X‖², where X is the calibration data flowing through that layer.

The key insight: not all weight errors matter equally. A small error in a high-sensitivity weight damages the output more than a large error in a low-sensitivity weight. GPTQ uses the Hessian H = 2X^TX of the reconstruction loss to measure each weight's sensitivity.

It quantizes weights one column at a time, using the Hessian inverse to redistribute the quantization error of each column to the remaining (not-yet-quantized) columns. This "error compensation" is what makes GPTQ dramatically better than naive round-to-nearest PTQ.

javascript
function perChannelScale(weights) {
  return weights.map(channel => {
    const mn = Math.min(...channel);
    const mx = Math.max(...channel);
    return (mx - mn) / 255;
  });
}

In practice, you'd also compute the zero-point for each channel: zp = min. For a constant channel (all values equal), scale = 0 and any quantized value dequantizes to that constant. Libraries handle this edge case by setting scale to a tiny epsilon to avoid division by zero during quantization.

Correct order: Run calibration data → Compute scales & zero-points → Quantize weights → Evaluate accuracy

Calibration must come first because you need to observe actual value ranges before you can compute scales. Scales must come before quantization because quantization uses them. Evaluation comes last to verify that the quantized model didn't lose too much accuracy. If accuracy drops too much, you'd try a finer granularity (per-channel → per-group), more calibration data, or a smarter method like GPTQ.

The gradient passes through the fake quantizer unchanged. This is the "straight-through" part — the backward pass acts as if the quantizer isn't there. The forward pass still applies quantization noise, so the model learns to be robust to it. If x were outside the clipping range, the STE returns 0, telling the optimizer "this weight is so far out of range that nudging it won't help — clip it."

The round() function creates a staircase: its output is constant (flat) between integer boundaries. The derivative of a constant is zero. At the integer boundaries, the function jumps discontinuously, so the derivative is undefined.

If you used the true derivative, every weight would receive a gradient of exactly 0. Gradient descent with zero gradients means no learning. The STE is a deliberate, principled approximation: by pretending the staircase is the identity, you get useful (if noisy) gradients that point the weights toward quantization-friendly values.

Bengio et al. (2013) showed this works well empirically, and it has become the standard approach in all QAT frameworks (PyTorch's FX quantization, TensorFlow's QAT toolkit, etc.).

QAT is much cheaper than full training (3.6 hours vs ~28 hours for the original 100 epochs). But it's far more expensive than PTQ, which takes minutes. This is the fundamental trade-off: PTQ is cheap but loses more accuracy; QAT is expensive but recovers most of the lost accuracy. For models going to production with tight accuracy requirements, QAT is worth the cost.

javascript
function fakeQuantize(x, scale, zeroPoint, qmin, qmax) {
  let q = Math.round(x / scale + zeroPoint);
  q = Math.max(qmin, Math.min(qmax, q));
  return (q - zeroPoint) * scale;
}

Let's trace the clamped case: x=1.6, q = round(1.6/0.01 + 100) = round(260) = 260. Clamp to [0,255]: q=255. Dequantize: (255−100)×0.01 = 1.55. The value 1.6 was beyond the representable range and got clipped to 1.55. This is how the model learns to keep its activations within the quantization range — gradients from clipping push values back in-range.

Line 4 is the bug. It uses grad + mask (addition) instead of grad * mask (multiplication).

When x is in range, mask=1: the current code returns grad + 1 (corrupted), but it should return grad × 1 = grad (passthrough). When x is out of range, mask=0: the code returns grad + 0 = grad (should be grad × 0 = 0, i.e., blocked).

The fix: return grad * mask;. The mask acts as a gate — multiply to pass or block the gradient.

Notice that the pruned weights (sum = −0.1 − 0.02 + 0.3 + 0.05 = 0.23) are indeed the ones contributing least to the overall sum. Magnitude pruning assumes that small weights have small effects on the output — often true for well-trained networks, but not always (some small weights sit on high-curvature loss landscape regions).

Structured pruning removes entire structures: a row of a weight matrix (removes an output neuron), a column (removes an input feature), or an entire attention head. The result is a smaller but still dense matrix. Dense matmul on a [3072, 4096] matrix instead of [4096, 4096] is genuinely 25% faster.

Unstructured pruning at 90% sparsity creates a [4096, 4096] matrix where 90% of entries are zero, but the matrix shape is unchanged. Standard GEMM kernels still iterate over all positions. You need special sparse kernels (e.g., NVIDIA's cuSPARSE, 2:4 structured sparsity on Ampere) to skip zeros, and these only help at very high sparsity (>95%) or with hardware-supported patterns (2:4).

10× is the theoretical ceiling. In practice, unstructured 90% sparsity on a GPU typically yields only 1.5-3× speedup (or sometimes no speedup at all) due to irregular memory access patterns, sparse format overhead, and poor hardware utilization. NVIDIA's Ampere architecture supports 2:4 structured sparsity (50% sparsity in a specific pattern) with near-2× speedup — the only widely-deployed hardware sparse acceleration.

GPUs achieve their massive throughput through coalesced memory access — reading contiguous chunks of memory in parallel. A dense matmul reads weight rows sequentially, hitting every cache line once. Sparse matrices require indirect indexing: the GPU reads an index array to find where the nonzero values are, then gathers them from scattered memory locations.

This scatter-gather pattern causes cache misses (the nonzeros are spread across memory), branch mispredictions (the sparsity pattern is unpredictable), and low SIMD utilization (some lanes in a warp have data, others don't). Add the overhead of storing the sparse format (CSR needs row pointers + column indices), and the "free" skipped multiplications cost more than they save.

This is why the industry has converged on N:M structured sparsity (e.g., 2:4 on Ampere): a regular pattern that the hardware can exploit with dedicated logic.

javascript
function magnitudePrune(weights, sparsity) {
  const mags = weights.map(w => Math.abs(w));
  const sorted = [...mags].sort((a, b) => a - b);
  const k = Math.ceil(sparsity * weights.length);
  const threshold = sorted[k - 1];  // k-th smallest magnitude
  let pruned = 0;
  return weights.map((w, i) => {
    if (mags[i] <= threshold && pruned < k) {
      pruned++;
      return 0;
    }
    return w;
  });
}

The pruned < k guard handles ties: if multiple weights share the threshold magnitude, we only prune enough to reach the desired sparsity. In real frameworks (PyTorch's torch.nn.utils.prune), the threshold is computed as a percentile and ties are broken arbitrarily.

Correct order: Train dense model → Score weight importance → Create binary mask → Apply mask → Fine-tune surviving weights

You must start with a fully-trained dense model (random initialization + pruning = disaster). Scoring comes next because you need trained weights to judge importance. The mask is derived from the scores (threshold at desired sparsity). Applying the mask zeroes out the pruned weights. Fine-tuning lets the surviving weights adapt to compensate for their removed neighbors.

In iterative pruning (the Lottery Ticket Hypothesis approach), you repeat steps 2-5 multiple times, pruning a small fraction each round: prune 20% → fine-tune → prune 20% of remaining → fine-tune → ... This is gentler than one-shot pruning and typically recovers more accuracy at high sparsity.

This is the fundamental problem with IMP: finding a winning ticket costs 8× more than a single training run. Later work (Rewinding to iteration k, SNIP, SynFlow) tries to reduce this cost by pruning earlier in training or at initialization.

The critical word is original. A winning ticket is defined by both its structure (which weights survive pruning) and its initialization (the exact values those weights had at epoch 0). Re-initializing with new random weights destroys the ticket — the subnetwork typically fails to train to the same accuracy. This is the key insight: the lottery isn't just about which connections matter, but about those connections starting in the right place in the loss landscape.

After 5 rounds of 20%-per-round pruning, about 67% of parameters have been removed. The exponential decay means early rounds remove many more absolute parameters than later rounds (20M in round 1 vs. 8.2M in round 5).

This is the most surprising result in the paper. The winning ticket is not just the topology (which neurons connect to which) — it's the topology plus the specific initial weights. Re-initializing the same sparse structure with new random values produces a network that trains no better than a random sparse network of the same size.

Think of it like a key and a lock. The structure (mask) is the shape of the key, but the initialization values are the precise tooth heights. You need both to open the lock (reach good accuracy). This suggests that certain fortuitous initial weight configurations are essential for trainability — a deeply non-obvious result.

javascript
function iterativePrune(totalParams, pruneRate, rounds) {
  const result = [];
  let remaining = totalParams;
  for (let i = 0; i < rounds; i++) {
    remaining *= (1 - pruneRate);
    result.push(remaining);
  }
  return result;
}

Step 1: Softmax at T=1

Step 2: Softmax at T=5

Step 3: Entropy

The maximum entropy for 3 classes is ln(3) ≈ 1.099. At T=5, we're already at 1.087 — nearly uniform. Temperature acts as a "softening knob" that reveals relationships hidden by the peaked T=1 distribution.

At T=1, the teacher's softmax is highly peaked — often 95%+ on the top class. The small probabilities on other classes (0.01% dog, 0.003% tiger) are numerically negligible and carry almost no gradient signal. But these tiny probabilities encode exactly the information we want: which wrong answers are less wrong.

Raising T "flattens" the distribution, amplifying these small probabilities into meaningful training signal. The student learns not just "this is a cat" but "if it's not a cat, it's probably a dog, and definitely not a car." This relational structure is the dark knowledge.

The distillation and hard-label terms are roughly balanced here. In practice, α = 0.7–0.9 works well — we lean heavily on the teacher's soft knowledge while still anchoring to ground truth. The T² factor is crucial for keeping gradients balanced (see next exercise).

When you divide logits by T before softmax, the chain rule introduces a factor of 1/T in the gradients. Since KL divergence involves two softmaxes (teacher and student), the gradients are scaled by 1/T². Without correction, higher temperatures would produce vanishingly small gradient updates from the distillation term, making it irrelevant compared to the hard-label loss.

Multiplying by T² exactly compensates this scaling, ensuring that the relative contribution of the distillation and hard-label terms is controlled solely by α, not accidentally by T. This is not a heuristic — it falls directly out of the calculus.

javascript
function softmaxWithTemp(logits, T) {
  const scaled = logits.map(z => z / T);
  const maxVal = Math.max(...scaled);
  const exps = scaled.map(z => Math.exp(z - maxVal));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map(e => e / sum);
}

Subtracting the max before exponentiation prevents overflow (exp of large numbers). This doesn't change the result because softmax(z − c) = softmax(z) for any constant c.

Line 5 is the bug. The KL divergence term is missing the T * T scaling factor. It should be:

javascript
return alpha * T * T * klDiv + (1 - alpha) * hardLoss;

Without T², the gradients from the distillation term are scaled down by 1/T². At T=10, that's a 100× reduction — the distillation signal becomes negligible and the student effectively ignores the teacher, learning only from hard labels. The whole point of distillation is lost.

For a square [d, d] matrix, the break-even rank is always d/2. Any rank below d/2 saves parameters. LoRA typically uses r = 4–64, which is 32–512× below break-even — the savings are enormous because real weight matrices have very low effective rank.

0.24% of parameters — yet LoRA at rank 16 typically achieves 95–100% of full fine-tuning quality on most tasks. The reason: weight updates during fine-tuning are inherently low-rank. The model has already learned general representations; fine-tuning only needs to make a small, structured adjustment.

At initialization, ΔW = A · B = A · 0 = 0. So the adapted weight is W + 0 = W — the model starts exactly at its pre-trained state. This is crucial: it means LoRA training begins from a known-good point in the loss landscape, not a random perturbation of it.

If both A and B were randomly initialized, the initial ΔW would be a random matrix scaled by √r, which would immediately corrupt the pre-trained features. The model would need to "un-learn" this random perturbation before it could start fine-tuning. Zeroing B avoids this entirely.

With just 4 out of 8 singular values (50% of full rank), we capture 99.79% of the matrix's energy. The remaining 4 singular values contribute only 0.21%. This is typical of neural network weight matrices — they're highly low-rank. It's why LoRA works: the "important" information lives in a low-dimensional subspace.

javascript
function loraParams(d, r, numMatrices) {
  const paramsPerMatrix = 2 * d * r;  // A[d,r] + B[r,d]
  return paramsPerMatrix * numMatrices;
}

The correct order is: Freeze W → Init adapters → Train A,B → Merge W + AB → Deploy

The merge step is optional but common for deployment: since W' = W + AB is a single matrix of the same shape as W, the merged model has zero inference overhead — no additional latency, no extra memory. You only pay the LoRA cost during training. This is LoRA's killer feature: efficient training with free deployment.

A common misconception: "mixed precision halves training memory." It doesn't — weight memory stays the same because you need FP32 master weights + FP32 optimizer states regardless. But activation memory (which grows with batch size and sequence length) is halved. For large batch training, activations can be 60–80% of total memory, so this saving is crucial for fitting larger batches on a single GPU.

The speedup is exactly 2× because the A100's FP16 Tensor Core throughput is exactly 2× the FP32 throughput. In practice, you often get closer to 1.5–1.8× because real workloads mix compute-bound and memory-bound operations. The H100 has 2× the FP16/BF16 throughput of the A100, plus native FP8 support at 1,979 TOPS.

FP16's 5-bit exponent limits its range to ~65,504. Gradients or loss values outside this range cause overflow (Inf) or underflow (0) — requiring careful loss scaling. BF16 matches FP32's full range, so you can typically drop it in without any loss scaling. The price is lower precision (7 vs 10 mantissa bits), but empirically this barely affects training quality for large models.

FP8 has two variants: E4M3 (4 exponent, 3 mantissa — more precision, max ~448) and E5M2 (5 exponent, 2 mantissa — more range, max ~57,344). E4M3 is typically used for weights and activations in the forward pass, while E5M2 is used for gradients (which need more dynamic range). The H100 was the first GPU with native FP8 Tensor Core support.

Line 5 is the bug. The gradients are still scaled up by lossScale — you must divide them by lossScale before updating the FP32 master weights. Without unscaling:

javascript
// BUG: effective lr = actual_lr * lossScale (e.g., 1024x too large!)
updateWeights(model, grads);

// FIX: unscale gradients before update
const unscaledGrads = grads.map(g => g / lossScale);
updateWeights(model, unscaledGrads);

Loss scaling works in three steps: (1) multiply loss by scale, (2) backward pass with scaled loss produces scaled gradients, (3) divide gradients by scale before the optimizer step. Forgetting step 3 means your effective learning rate is multiplied by lossScale (typically 1024–65536), causing immediate divergence.

Pruning and quantization multiply: 2× from pruning × 4× from FP16→INT4 = 8× total. This is why practical deployments combine techniques — no single method would get 8× compression with acceptable quality loss. The 3.25 GB fits comfortably in an 8 GB consumer GPU with room for KV cache and activations.

The correct order is prune → fine-tune → quantize. Pruning drastically changes the weight distribution (surviving weights may shift to compensate for removed ones, especially after fine-tuning). If you quantize first, the quantization grid (scale factors and zero points) was calibrated for the original weight distribution — which no longer exists after pruning. The quantization error will be much higher.

Conversely, quantizing the final pruned+fine-tuned model lets the calibration data see the actual weight distribution that will be deployed. This consistently gives 1–3% better accuracy than the reverse order.

This is an idealized estimate — real speedups are lower because autoregressive decoding is memory-bandwidth-bound (not compute-bound), so the INT8 hardware speedup may only be 1.3–1.5× in practice. Still, a 7B INT8 student serving at ~10ms/token is realistic and represents a massive cost reduction over a 70B FP16 teacher. At 5 ms/token you could serve 200 tokens/second on a single GPU.

javascript
function compressionPipeline(params, originalBits, pruneFraction, quantBits) {
  const remaining = params * (1 - pruneFraction);
  return remaining * quantBits / 8 / 1e9;
}

Note that originalBits doesn't appear in the calculation — it tells you the starting size for context, but the final size depends only on remaining params and target bits. The compression ratio would be (params * originalBits/8/1e9) / result.

Line 2 is the bug. Quantizing before pruning means the INT4 scale factors and zero points were computed on the original dense weights. After pruning removes 50% of parameters, the surviving weight distribution shifts significantly — but the quantization grid is now wrong for these new values.

The correct order is:

javascript
function compress(model) {
  prune(model, sparsity=0.5);        // prune first
  finetune(model);                    // recover accuracy
  quantize(model, 'int4');           // quantize pruned weights
  calibrate(model, calibrationData); // calibrate final model
  return model;
}

The calibration on line 4 also can't fully fix the damage — it can adjust scale factors, but the quantization grid was already committed at line 2. Always prune first, then quantize.

The correct order is: Profile → Structured prune → Fine-tune → Quantize (PTQ) → Evaluate → Deploy

Profile first: identify which layers contribute least (via sensitivity analysis, activation magnitudes, or importance scores). Prune: remove the least important structures. Fine-tune: let the model recover from pruning damage (typically 1–5% of original training). Quantize: calibrate on representative data after pruning, because the weight distribution has changed. Evaluate: verify quality on held-out benchmarks before deploying. Deploy: serve the compressed model.