One trunk, many answers. Put a small head for each task on one shared trunk, and a task with few labels can borrow what the other tasks learned. Or it can lose the trunk to them. In the toy below, sharing cuts task A's error from 0.28 to 0.08 when the tasks are related, and raises it to 0.92 when they are only partly related. The weights on the losses decide much of the rest.
Learn why sharing helps and why it hurts, derive uncertainty weighting, GradNorm and gradient surgery from zero, see the honest result that a tuned weighted sum often matches them, and build it all in PyTorch with masked losses.
A tiny trunk with two heads trains live in your browser. Task A has 40 labels; task B has 300, in bigger units. Slide relatedness down and watch A's error climb above its alone line. Then try the loss weight, uncertainty weighting and PCGrad.
You need a loss function, gradient descent, and the idea of a network with a backbone and a head. We build the rest from zero.
Synthetic data, illustrative numbers. Inputs have 10 numbers. Each task's target is a smooth function of two directions in the input; relatedness is the cosine between task A's directions and task B's. Task B's targets are 4 times larger, so its squared loss is about 16 times larger. The trunk is one layer of 2 tanh units; each head is linear. Adam, 500 full-batch steps. Error is the validation mean squared error divided by the target variance: 0 is perfect, 1 is as bad as predicting the mean. Cosine is between the two tasks' validation-loss gradients on the trunk.
Chapter 0
Meet the problem, the shape of the block, and what sharing saves before it costs anything
You run a home-audio product. One audio encoder turns each 10-second clip into an embedding. Product wants five answers from every clip: is someone speaking, is a dog barking, is glass breaking, who is the speaker, and how do they sound. You have 200,000 clips labelled for speech, 8,000 for dogs, 600 for glass, and a few hundred for emotion.
You could train five separate models. Or you could train one network with one shared body and five small outputs. The shared body is the trunk: the layers every task uses. Each small output is a head: a few layers that belong to one task. Training several tasks at once through a shared part is multi-task learning (MTL).
The shape is simple. Input x goes through the trunk once and becomes a feature vector z. Each head k reads the same z and writes its own answer. Each task has its own loss. The training signal is a weighted sum of those losses.
| Step | Shape | What it is |
|---|---|---|
| Input | B × Din | A batch of B inputs (clips, images, patient records). |
| Trunk | B × D | Shared features z. Computed once per input, for every task. |
| Head k | B × Ck | Task k's output: 1 logit for "dog or not", 4 for an emotion class, 1 number for loudness. |
| Loss k | scalar | Task k's loss, averaged over the examples that have a label for task k (Chapter 9). |
| Total | scalar | Σ wkLk. One backward pass trains the trunk and every head. |
Sharing saves parameters, memory and compute before it does anything clever. Count them.
Worked: K = 5, PT = 86 M, PH = 0.01 M → separate 5 × 86.01 = 430.05 M; shared 86 + 0.05 = 86.05 M. Five times fewer weights, and one trunk forward pass per clip instead of five.
Add tasks and grow the trunk. Purple blocks are trunks; small coloured blocks are heads. Then freeze the trunk and cache its outputs.
Arithmetic only. Each head is taken as 0.01 M weights. "Trained" counts the weights that get gradients.
The frozen case is worth a second look. If the trunk never changes, you run it once per clip and store z on disk. Then every head trains on the same small cached tensors. Adding a task costs one tiny head. The tasks also cannot hurt each other, because no gradient reaches the shared part. That is the cheapest multi-task model there is.
When the trunk is trained, the story gets interesting. Gradients from every task now pull on the same weights. Sometimes they pull together, and a task with 600 labels learns from 200,000 others. Sometimes they pull apart, and the big task takes the trunk. This lesson is about telling those cases apart and steering between them.
Chapter 1 Why it works
Borrow data from a related task, and derive exactly when the borrowing pays: a trade between less variance and more bias
Caruana put the idea in one sentence in 1997: multi-task learning "improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias" Caruana 1997 · abstract. An inductive bias is a preference for some solutions over others, before you see the test data. Here the preference is: features that also help the other tasks.
His road-following test shows it. The main task was to steer from a 32-pixel road image. He added eight extra outputs, such as where the road's left edge is and whether the road has one or two lanes. With exactly the same training images, the shared network steered with 15 to 30% less error than single-task networks Caruana 1997 · §2.1, Table 1.
Strip a network down to one number. Task A wants a mean μA and has nA noisy samples. Task B wants a mean μB and has nB samples. The tasks are related: μB differs from μA by Δ. "Sharing" means estimating one number from all nA + nB samples.
Alone, A's estimate is its own average. It is unbiased, and its variance is σ2/nA. Pooled, the estimate is (nAx̄A + nBx̄B)/(nA+nB). Its expected value is μA + (nB/(nA+nB))Δ, so it is off by a bias. Its variance is σ2/(nA+nB). The error of an estimate is variance plus bias squared:
Worked: σ = 1, nA = 20, nB = 300. Alone: 1/20 = 0.050. Shared with Δ = 0.2: 1/320 + (0.9375 × 0.2)2 = 0.0031 + 0.0352 = 0.038 (sharing helps). With Δ = 0.5: 0.0031 + 0.2197 = 0.223 (sharing hurts, 4.5 times worse).
That is the whole of Chapters 1 and 2 in one formula. Sharing buys variance with bias. When A has few labels, its variance is big, so the trade is good, unless the tasks differ a lot. A middle road exists too: mix the two estimates with a weight that depends on how much you trust B. That is soft sharing, and Chapter 3 builds it.
The curves show each estimate's error as A's label count grows (log scales). Make the tasks less related and watch the shared curve flatten into a floor of pure bias.
Exact formulas, not a simulation: σ = 1 and nB = 300. "Best mix" is (1−λ)x̄A + λx̄B with the λ that minimises the error, λ* = (σ2/nA) / (σ2/nA + σ2/nB + Δ2).
A network has millions of numbers, not one. But the trunk plays the role of the shared estimate. If both tasks need the same features, the trunk gets nA + nB examples' worth of evidence for them. Ruder's survey lists the ways this helps: the tasks' different noise averages out, the extra task points attention at the features that matter, and one task can learn a feature that is hard to see from the other Ruder 2017 · §4. He also quotes Baxter's 1997 result that the risk of overfitting the shared parameters is smaller by an order of N, the number of tasks Ruder 2017 · §3.1.
Medicine gave Caruana a strong version of this. In a pneumonia database of 14,199 patients, 35 lab results arrive only after a patient is admitted. They cannot be inputs for the admit decision. He used them as extra outputs instead. The risk model then made 5 to 10% fewer errors Caruana 1997 · §2.3, Table 3. The extra heads were thrown away after training. Their job was to shape the trunk.
Chapter 2 Why it works
See negative transfer as a first-order calculation: one step of the summed gradient can raise a task's own loss
Now look at one training step. The trunk's weights are θ. Task A's loss has gradient gA with respect to θ; task B's has gB. With equal weights, gradient descent moves θ by −η(gA + gB), where η is the learning rate.
What does that step do to task A's loss? For a small step, a loss changes by its gradient dotted with the move. So:
Worked: gA = (1, 0), gB = (−2, 0.5), η = 0.1. gA·gB = −2 and ‖gA‖2 = 1, so ΔLA ≈ −0.1(1 − 2) = +0.10: A gets worse. ΔLB ≈ −0.1(4.25 − 2) = −0.225: B improves.
Read the condition. Task A's loss rises when gA·gB < −‖gA‖2. That needs two things at once: the gradients point apart, and B's gradient is big compared with A's. A big gradient comes from a task with a large loss, large units, or many labels in the batch.
Drag the arrow tips. Warm is task A's gradient, blue is task B's, purple is their sum. The step goes the opposite way to the sum. The bars show what one step does to each task's loss: red means it rises.
Two shared weights, so the gradients are 2-D arrows. η = 0.1. First-order changes: exact for a small step, approximate for a big one.
Yu and colleagues named the bad case the tragic triad: gradients that conflict, one gradient much larger than the other, and high curvature, where a first-order step misjudges the true change PCGrad · §2. When all three meet, the summed step serves the dominant task and damages the other.
When sharing makes a task worse than training it alone, we call it negative transfer. Standley and colleagues measured it on Taskonomy indoor scenes with five vision tasks. A 4-task network had 10.69% worse total loss on average than four single-task networks of the same size. Against four networks a quarter of the size each, so the same total compute, it was 4.86% better Standley 2020 · §4, Setting 1. Sharing saved compute; it did not save accuracy.
Chapter 3
One trunk for all, or one model per task held together by a penalty; then how to choose which layers to share
There are two families. Hard parameter sharing is the trunk-and-heads design: the shared layers are literally the same weights. It goes back to Caruana and is still the default Ruder 2017 · §3.1. Soft parameter sharing gives each task its own model and adds a penalty that keeps the models close, for example the squared distance between their weights Ruder 2017 · §3.2.
Soft sharing has a dial. For two linear models with weights wA and wB:
Worked, one number per task: minimise (wA − 1)2 + (wB − 0)2 + λ(wA − wB)2. Setting both derivatives to 0 gives wA = (1+λ)/(1+2λ). λ = 0 → (1, 0); λ = 1 → (0.667, 0.333); λ → ∞ → (0.5, 0.5).
Task A has 8 examples, task B has 30, both in 6 dimensions. The curves are each task's test error as λ grows. Left end: separate models. Right end: one shared model. Pick a relatedness and slide λ.
Solved exactly here (the normal equations, a 12 × 12 system), averaged over 40 random datasets. Noise spread 0.5; true weight vectors of length about 2.4 with the chosen cosine. Error = expected test squared error. Illustrative numbers.
Two lessons. With related tasks, a middle λ beats both ends for the small task. With unrelated tasks, the right end is bad for both, so soft sharing's freedom matters. Cross-stitch networks learn this dial per layer instead of fixing it: each task's next activations are a learned linear mix of both tasks' activations Cross-stitch · abstract.
In a deep trunk, early layers learn generic features (edges, onsets) and late layers learn task-specific ones. So the usual design shares the early layers and splits late. Where exactly to split is an empirical question, and it is not stable. Standley et al. found that which pairs of tasks help each other changed with network size and data size, and did not correlate with transfer-learning affinities (Pearson's r = −0.12 in their first setting) Standley 2020 · §4.
Chapter 4
Weights rescale each task's gradient on the trunk; units and label counts set the scale before you choose anything
The usual objective is a weighted sum. Taking a gradient shows what the weights really do:
Worked, units: a depth error of 0.05 m gives a squared loss of 0.0025 m2. The same error in centimetres is 5 cm, a loss of 25 cm2: 10,000 times bigger. Change the units and you have changed the task's weight by 10,000.
Kendall, Gal and Cipolla showed how much this matters. On a small version of CityScapes they trained one network for semantic segmentation and depth with fixed weights. Segmentation alone scored 59.4% IoU; depth alone had 0.640 px error. Weights 0.85 : 0.15 gave 60.4% and 0.582, better than both alone. Weights 0.5 : 0.5 gave 56.3% and 0.602: worse segmentation than alone Kendall 2018 · Fig. 2a. Same network, same data; only the weights changed.
The device repeats their sweep on the hero's toy. It trains one network for each weight and plots both tasks' validation error.
Each pair of dots is one network trained with weight wA on task A and 1 − wA on task B. Dashed lines are each task alone. Change task B's units and run again: watch the good region move.
Trained here: the hero's toy (task A 40 labels, task B 300), one network per weight, 500 Adam steps, two data seeds averaged. Illustrative numbers.
Three things to see. With related tasks, a wide band of weights beats both tasks alone. With bigger units for B, the good band slides toward a large wA, because the weight must undo the scale. With unrelated tasks, the curves cross: every weight trades one task for the other.
Scale is one reason weights matter. Speed is the other. Tasks learn at different rates, so a weight that is right at step 100 can be wrong at step 10,000. Chapters 5 to 7 are three ways to set weights, or gradients, automatically.
Chapter 5 Why it works
Derive Kendall, Gal and Cipolla's 1/(2σ2)·L + log σ from a Gaussian likelihood, and see why it cancels units
Here is the idea in plain words. If a task's targets are noisy, its errors will be large no matter what. A large error from a noisy task should count for less. So give each task a learned noise level σi, and divide its loss by it.
Derive it. Suppose task i's target is the network's output plus Gaussian noise of spread σ: y ~ N(f(x), σ2). Maximum likelihood means minimising the negative log of the density. The Gaussian density is e−(y−f)2/(2σ2) / (√(2π)σ). Take minus its log:
This is Kendall, Gal and Cipolla's equation 7 Kendall 2018 · §3.2, eq. 7. The log σ term matters. Without it, the network would make every σ huge and every weight zero. With it, a big σ is penalised.
Treat L as fixed and minimise over σ. The derivative of L/(2σ2) + log σ is −L/σ3 + 1/σ. Set it to 0: σ2 = L. The learned variance tracks the task's own loss. Put it back in: the weight is 1/(2L), and the task's term is ½ + ½log L.
Worked: L = 0.04 → σ2 = 0.04, weight 1/0.08 = 12.5, contribution 12.5 × 0.04 = 0.5. Same task in units 100 times smaller: L = 400 → weight 1/800 = 0.00125, contribution again 0.5.
So at balance, each task contributes ½ + ½log Li. Its gradient is ∇Li/(2Li) = ½∇log Li. Uncertainty weighting is, near balance, minimising the sum of log losses. A log does not care about units: multiplying L by 10,000 only adds a constant. That is why it fixes Chapter 4's units problem.
The curve is one task's term, L/(2σ2) + log σ, against log σ2. Its lowest point is where training settles. Change the loss, then switch to units 100 times smaller.
Exact curve. In practice the network learns s = log σ2 rather than σ, for numerical stability, as Kendall et al. do Kendall 2018 · §3.2.
On their Tiny CityScapes test, learned weights beat every hand-tuned pair: 62.7% IoU and 0.533 depth error, against the best fixed pair's 60.4% and 0.582 Kendall 2018 · Fig. 2a. For classification they derive a similar form with a softmax scaled by 1/σ2, using an approximation that is exact when σ = 1 Kendall 2018 · §3.2.
Chapter 6
Learn the weights so that each task's gradient on the trunk has a target size, set by how fast that task is training
Chen and colleagues looked at the problem from the gradient side. A task controls the trunk through the size of its weighted gradient. So measure that size, decide what it should be, and nudge the weight until it gets there GradNorm · §3.
Four definitions, from the paper's §3.1. Let W be the last shared layer. For each task i at step t:
The weights are then trained on Lgrad = Σi|Gi − targeti|, with the targets held constant. After each step the weights are rescaled to sum to the number of tasks GradNorm · §3.2, eq. 2. Because Gi = wi‖∇WLi‖, the gradient for wi is just sign(Gi − targeti) × ‖∇WLi‖.
Worked: GA = 2, GB = 6, so Ḡ = 4. L̃A = 0.8, L̃B = 0.4, mean 0.6, so rA = 1.333 and rB = 0.667. With α = 1.5: rα = 1.540 and 0.544, targets 6.16 and 2.18. A pulls too weakly (2 < 6.16), so wA rises; B pulls too hard, so wB falls.
Lines are the two task weights over training (they always sum to 2). Pick α and a relatedness, then Run. The bars compare the final errors with plain equal weights.
Trained here: the hero's toy with task B in bigger units. GradNorm's W here is the whole one-layer trunk. Weight learning rate 0.025, as in the paper's §5.1. Illustrative numbers.
On NYUv2 with segmentation, depth and surface normals, GradNorm with α = 1.5 beat equal weights on depth (0.925 against 0.944 m RMS error) and segmentation (67.8 against 70.1, error as 100 − IoU), and matched or beat the single-task networks GradNorm · Table 1. The authors also retrained with the time-averaged GradNorm weights held fixed, and got similar results (0.939 depth, 67.5 segmentation). So one GradNorm run can be used to find good static weights GradNorm · §5.2.
Chapter 7 Why it works
When two task gradients conflict, remove from each the part that points against the other; derive why both losses then fall
Weights change how hard each task pulls. PCGrad changes the direction. Yu and colleagues' rule: if task i's gradient conflicts with task j's (their dot product is negative), project gi onto the plane at right angles to gj PCGrad · §2.3, Algorithm 1.
Worked: g1 = (1, 0), g2 = (−1, 1). g1·g2 = −1 < 0. g1′ = (1, 0) − (−1/2)(−1, 1) = (0.5, 0.5). g2′ = (−1, 1) − (−1/1)(1, 0) = (0, 1). Update direction: (0.5, 1.5), against the plain sum (0, 1).
Now check what the plain sum did. g1·(0, 1) = 0: by Chapter 2's formula, task 1 made no progress at all. With PCGrad, g1·(0.5, 1.5) = 0.5 > 0: task 1's loss falls. And task 2: g2·(0.5, 1.5) = 1 > 0, it falls too.
Take the conflicting case. The update is g1′ + g2′. Task 1's first-order progress is g1·(g1′ + g2′). Expand g1·g2′ = g1·g2 − (g2·g1/‖g1‖2)‖g1‖2 = 0. And g1·g1′ = ‖g1‖2 − (g1·g2)2/‖g2‖2 ≥ 0, by the Cauchy-Schwarz inequality. So task 1's loss does not rise to first order. The same holds for task 2 by symmetry.
Drag the tips. With PCGrad on, the green dashed arrows are the projected gradients, and their sum is the purple step. Make B large and opposite, then switch PCGrad off and on.
Two shared weights, η = 0.1, first-order loss changes. With more than two tasks PCGrad projects against the others one at a time, in a random order.
Reported results: on CelebA's 40 face attributes, PCGrad's average error was 8.69% against 8.95% for the MGDA method of Sener and Koltun PCGrad · Table 3. On the Meta-World MT10 robot benchmark, PCGrad with SAC solved all 10 tasks, and about 70% of the 50 in MT50 PCGrad · §5.2.
Chapter 8
Two careful studies found that a well-tuned weighted sum matches them; see why on the trade-off curve
You now have three clever balancing methods. In 2022 two groups asked whether they beat the plain weighted sum when that baseline is tuned properly. Both said no.
Xin, Ghorbani, Garg, Firat and Gilmer ran large experiments on translation and vision. Their conclusion: "despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance improvements beyond what is achievable via traditional optimization approaches" Xin 2022 · abstract. Kurin, De Palma, Kostrikov, Whiteson and Kumar found that the plain sum with equal weights, "coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon" the complex optimisers Kurin 2022 · abstract.
Plot every trained model as a point: task A's error across, task B's error up. Lower-left is better. A model is Pareto optimal if no other model is better on both tasks. The set of such models is the Pareto front: the best trade-offs on offer Xin 2022 · §2.
Any minimiser of a positive weighted sum is Pareto optimal: if another model were better on every task, it would have a smaller weighted sum. For convex losses the converse also holds: every point on the front is reached by some weights. So for convex problems, "no algorithm can outperform properly chosen scalarization that has been trained to convergence" Xin 2022 · §2. Networks are not convex, which is why it had to be tested.
Worked: two models with errors (A, B) = (0.30, 0.10) and (0.25, 0.20). With weights (0.5, 0.5) their weighted sums are 0.20 and 0.225: the first wins. With (0.9, 0.1): 0.28 and 0.245: the second wins. Each weight picks a different point on the front.
Purple dots: one network per fixed weight. The purple line joins the ones no other dot beats. Green marks: the balancing methods. Dashed lines: each task alone. Lower-left is better.
Trained here: the hero's toy with B in bigger units; 9 fixed weights plus 4 methods; 500 Adam steps; two data seeds averaged. Illustrative numbers.
In the toy, as in Xin et al., the methods land on or behind the front that the weight sweep traces. Xin et al. also looked inside: for most runs, the weights that GradNorm, PCGrad, MGDA and IMTL effectively assigned "do not move significantly" during training, so the methods behave like static weights Xin 2022 · §4.1, Fig. 3.
Note what they did not say. The weights still matter a great deal. On CityScapes the segmentation loss was an order of magnitude larger than depth, and most of the test-error front came from runs with segmentation weight below 0.2 Xin 2022 · §4.3. The finding is that a tuned fixed weight is enough, not that equal weights are.
Chapter 9 How
A trunk with a ModuleDict of heads, masked losses for missing labels, uncertainty weights, PCGrad in about 20 lines, and the frozen-trunk case
Real multi-task data is ragged. A clip has a speech label but no emotion label. A patient has a mortality label but only some lab results. So every example carries, per task, a label and a mask: true if the label exists. Each task's loss is averaged over its own labelled examples only.
Worked: batch of 4; task k labelled on 2 examples with squared errors 1 and 9. Masked mean = 10/2 = 5. Divided by the batch size instead: 10/4 = 2.5, which silently halves task k's weight. Filling the missing labels with 0 instead adds two fake errors that pull predictions toward 0.
200 examples of one regression task; hollow points have no label. Slide the missing fraction and switch strategies. The purple line is the fitted head; the dashed line is the truth.
Computed here: least squares on y = 3 + 2x + noise (spread 0.5). "Effective weight" is this task's weight relative to a fully labelled task with the same nominal weight.
An nn.ModuleDict holds the heads by task name, so adding a task is one line and every head's weights are registered with the optimiser. The forward pass runs the trunk once.
import torch import torch.nn as nn import torch.nn.functional as F class MultiTaskNet(nn.Module): """One shared trunk, one small head per task.""" def __init__(self, d_in, d_hid, tasks): # tasks: {'name': n_outputs} super().__init__() self.trunk = nn.Sequential( # shared: every task trains it nn.Linear(d_in, d_hid), nn.ReLU(), nn.Linear(d_hid, d_hid), nn.ReLU()) self.heads = nn.ModuleDict({ # private: one per task name: nn.Linear(d_hid, n_out) for name, n_out in tasks.items()}) def forward(self, x): # x: (B, d_in) z = self.trunk(x) # (B, d_hid), computed once return {name: head(z) for name, head in self.heads.items()} def masked_loss(pred, target, mask, kind): """Mean loss over the examples that HAVE a label for this task.""" # pred, target: (B,) ; mask: (B,) bool, False = label missing if mask.sum() == 0: return pred.sum() * 0.0 # keeps the graph, adds nothing if kind == 'bce': per = F.binary_cross_entropy_with_logits(pred, target, reduction='none') else: per = F.mse_loss(pred, target, reduction='none') return (per * mask).sum() / mask.sum() # divide by labels, not by B net = MultiTaskNet(32, 64, {'event': 1, 'loudness': 1}) x = torch.randn(8, 32) y = {'event': torch.randint(0, 2, (8,)).float(), 'loudness': torch.randn(8)} m = {'event': torch.tensor([1, 1, 0, 0, 1, 0, 1, 1]).bool(), 'loudness': torch.tensor([0, 1, 1, 1, 0, 0, 0, 1]).bool()} kinds = {'event': 'bce', 'loudness': 'mse'} out = net(x) # {'event': (8,1), 'loudness': (8,1)} losses = {k: masked_loss(out[k].squeeze(-1), y[k], m[k], kinds[k]) for k in out} total = sum(losses.values()) # equal weights; see the next block total.backward()
Why each choice. The trunk runs once, so K tasks cost one trunk pass. Each loss is divided by its own label count, so a rare label does not get a silently smaller weight. An empty task returns a zero that keeps the graph, so a batch with no glass labels does not crash and does not divide by zero. Two tasks with different loss types (BCE and MSE) sit side by side: that is exactly the scale mismatch of Chapter 4.
import torch import torch.nn as nn class UncertaintyWeights(nn.Module): """Kendall, Gal and Cipolla (2018): learn s_i = log sigma_i^2 per task.""" def __init__(self, n_tasks): super().__init__() self.s = nn.Parameter(torch.zeros(n_tasks)) # start at sigma^2 = 1 def forward(self, losses): # losses: list of scalars L = torch.stack(losses) # (n_tasks,) # 1/(2 sigma^2) * L + log sigma == 0.5 * exp(-s) * L + 0.5 * s return (0.5 * torch.exp(-self.s) * L + 0.5 * self.s).sum() uw = UncertaintyWeights(2) params = [torch.randn(4, requires_grad=True)] # stand-in for the network opt = torch.optim.Adam(params + list(uw.parameters()), lr=0.05) for step in range(300): loss_a = (params[0] ** 2).sum() * 0.01 # a task with small loss loss_b = (params[0] - 3).pow(2).sum() * 100.0 # a task in bigger units total = uw([loss_a, loss_b]) opt.zero_grad(); total.backward(); opt.step() print('weights =', (0.5 * torch.exp(-uw.s)).detach().numpy().round(3)) # [1.61 0.056]
The s parameters go into the same optimiser as the network. Learning s = log σ2 instead of σ keeps it unconstrained and avoids dividing by zero, as in the paper. The printed weights are the real output: the task in bigger units ends up with a weight about 29 times smaller.
import random import torch def pcgrad_step(losses, shared, opt): """Yu et al. (2020), Algorithm 1, on the shared parameters.""" grads = [] for L in losses: # one backward pass per task g = torch.autograd.grad(L, shared, retain_graph=True) grads.append(torch.cat([x.flatten() for x in g])) projected = [g.clone() for g in grads] for i, g_i in enumerate(projected): others = [j for j in range(len(grads)) if j != i] random.shuffle(others) # random order, as in the paper for j in others: dot = g_i @ grads[j] if dot < 0: # conflict: remove the clash g_i -= dot / grads[j].norm() ** 2 * grads[j] total = torch.stack(projected).sum(0) # (n_shared_params,) i = 0 for p in shared: # write it back into .grad n = p.numel(); p.grad = total[i:i + n].view_as(p).clone(); i += n opt.step() w = torch.nn.Parameter(torch.zeros(2)) opt = torch.optim.SGD([w], lr=0.1) g1 = torch.tensor([1.0, 0.0]); g2 = torch.tensor([-1.0, 1.0]) pcgrad_step([w @ g1, w @ g2], [w], opt) # losses with gradients g1, g2 print(w.data) # tensor([-0.0500, -0.1500])
The test case is Chapter 7's worked example: the update −0.1 × (0.5, 1.5) comes out exactly. In a real network, the head parameters are not in shared; give them their ordinary gradients with one more backward pass of the summed loss, and zero the trunk's gradient before writing the projected one.
import torch import torch.nn as nn import torch.nn.functional as F torch.manual_seed(0) # Cached embeddings from a FROZEN encoder: computed once, stored on disk. N, D = 2000, 512 emb = torch.randn(N, D) # stand-in for (N, D) cached clip embeddings labels = { # each task labelled on a different subset 'speech': (torch.randint(0, 2, (N,)).float(), torch.rand(N) < 0.9), 'dog': (torch.randint(0, 2, (N,)).float(), torch.rand(N) < 0.3), 'emotion': (torch.randint(0, 4, (N,)), torch.rand(N) < 0.1), } heads = nn.ModuleDict({'speech': nn.Linear(D, 1), 'dog': nn.Linear(D, 1), 'emotion': nn.Linear(D, 4)}) # 3 heads, 3,078 weights in all opt = torch.optim.AdamW(heads.parameters(), lr=1e-3, weight_decay=1e-2) for step in range(200): idx = torch.randint(0, N, (256,)) z = emb[idx] # (256, D): no encoder in the loop total = 0.0 for name, head in heads.items(): y, has = labels[name][0][idx], labels[name][1][idx] if has.sum() == 0: continue out = head(z[has]) # only rows with this label if name == 'emotion': total = total + F.cross_entropy(out, y[has]) else: total = total + F.binary_cross_entropy_with_logits(out.squeeze(-1), y[has]) opt.zero_grad(); total.backward(); opt.step()
With the trunk frozen, each head's gradient touches only its own weights. The sum is just a convenience: the heads train exactly as if trained alone. No transfer, good or bad, happens between them. You get the savings of sharing with none of the interference, and none of the borrowing. To borrow, something shared must train: the trunk, its last block, or an adapter (see Backbone + Head).
There is no single standard library one-liner for multi-task training. The blocks above are the library: nn.ModuleDict for heads, masked means for missing labels, and about 20 lines each for a balancing method. We ran all four blocks with PyTorch 2.10; the printed values in the comments are the real output.
(per * mask).sum() / 64. What did you just do?Chapter 10 Did it work?
Compare every task with its own single-task baseline across seeds, read a transfer matrix, and use gradient cosines with care
A multi-task model can raise the average score while one task quietly gets worse. So never report only an average. Here is the checklist.
Train each task alone, and each pair together. The entry for "task i trained with j" is the relative change in task i's loss:
Worked: task i alone has loss 0.40; with j, 0.30 → T = 0.10/0.40 = +25%. With k, 0.46 → T = −15%.
Here are real entries, from Standley et al.'s larger-network setting. Each row is the task trained alongside; each column is the task measured.
| Trained with ↓ / measured → | SemSeg | Depth | Normals | Keypoints | Edges |
|---|---|---|---|---|---|
| SemSeg | – | +3.00% | −2.79% | −5.20% | +27.80% |
| Depth | +1.72% | – | +1.18% | −3.52% | +25.73% |
| Normals | +10.81% | +7.12% | – | +88.98% | +71.59% |
| Keypoints | +3.12% | −0.41% | −10.12% | – | +61.07% |
| Edges | +0.03% | −1.40% | −4.78% | −3.05% | – |
Relative performance on each column task when trained with each row task, Taskonomy, larger Xception encoder Standley 2020 · Table 5.
Read it. Normals helps almost everything. Edges as a partner helps almost nothing. Keypoints gains 88.98% from normals, while normals loses 10.12% from keypoints: transfer is not symmetric. Pairwise training is expensive for many tasks, so Fifty et al. estimate it in one run. Their inter-task affinity takes a gradient step on the trunk using only task i, and measures task j's loss before and after: Zi→j = 1 − Lj(after)/Lj(before) TAG · §4.1, eq. 1. On Taskonomy, grouping by it cut test loss by 10.0% against training all tasks together TAG · abstract.
Each dot is one seed: alone minus shared, so right of zero means sharing helped. The bar is the mean, the whisker about two standard errors. Pick a relatedness and Run.
Trained here: the hero's toy with equal weights, 5 data seeds, 15 networks. Cosine is the trunk-gradient cosine on validation data, averaged over the second half of training. Illustrative numbers.
Log the cosine between the tasks' gradients on the trunk during training. Positive means they agree; persistently negative means they compete. Yu et al. used it to show conflict in real runs PCGrad · §5.3. Three cautions. At random initialisation the cosine is noise, because the heads are random. On training data it heads toward −1 near the end, because the summed gradient goes to zero. And it is a per-step quantity: smooth it and compute it on held-out batches.
Chapter 11 What elseWhat not
Five fields that use one trunk and many heads, the cases where it loses, and the whole block on one screen
| Field | Trunk → heads | What changes, what stays |
|---|---|---|
| Audio | One audio encoder → tagging, event detection, speaker, emotion | Heads differ in shape: clip-level logits, frame-level logits (event heads), an embedding for speaker, a class for emotion. Labels are very uneven, so masks and per-task baselines matter most. With a frozen encoder, all heads train on one cache. |
| Driving perception | One image encoder → segmentation, depth, instance masks | Kendall et al.'s setting: an encoder with one decoder per task, losses in different units, balanced by learned σ Kendall 2018 · §4. One encoder pass per frame matters for real time. |
| Clinical prediction | Patient record → mortality, plus future lab results | Caruana's pneumonia model: 35 later lab tests as extra outputs, unavailable at decision time, discarded after training, and 5 to 10% fewer errors Caruana 1997 · §2.3. Outcomes are rare and imbalanced. |
| Recommenders | Shared embeddings → click (CTR) and conversion (CVR) | ESMM shares the embedding layer and models pCTCVR = pCTR × pCVR over all impressions, because conversion labels exist only after clicks ESMM · §2.3. The tasks are linked by structure, not only by features. |
| Robotics | Observation encoder → action head + auxiliary predictions | Extra heads predict things the robot can measure later, such as where an object will be, to shape the encoder; Caruana gives the same "use the future" idea for vehicles Caruana 1997 · §4.1. Multi-task policies across skills, as in Meta-World MT10/MT50, face strong gradient conflict PCGrad · §5.2. |
What stays the same in every row: one shared trunk, one head per output, a weighted sum of masked losses, a single-task baseline per task. What changes: the head shapes, the loss types and their scales, and how the labels overlap.
An auxiliary task is one you do not need at deployment. You add its head only to shape the trunk, then delete it. Caruana's lab results and the robot's future measurements are both auxiliary. Treat its weight as a regularisation strength: tune it on the main task's validation score, and expect a U-shape, as in Chapter 1's bias-variance trade.
| Situation | Avoid a shared trunk because | Use instead |
|---|---|---|
| Unrelated tasks on a small trunk | They compete for capacity; balancing only picks a point on a bad trade-off (Chapters 4, 8) | Separate models, or share only the earliest layers; group by a transfer matrix |
| One task has far more data or larger loss | Its gradient dominates the trunk; the small task drowns (Chapter 2) | Tune the weights, sample batches per task, or pretrain on the big task then fine-tune for the small one (transfer learning) |
| Deployment needs only one task | Extra heads cost tuning time and can hurt the one that matters | Single-task training; keep other tasks as auxiliary heads only if they beat its validation score |
| You need tasks to update on different schedules | A retrain for one task moves the trunk under all others | Frozen trunk with independent heads, or separate models |
| The trunk is frozen | Nothing to fear, but nothing to gain: no transfer happens | Train heads independently on the cache; unfreeze the last block or add adapters to get transfer |
Choose an answer in each row. The design below updates.
A rule of thumb from this lesson, not a law. Always compare with single-task baselines on held-out data.
| Part | In one line |
|---|---|
| Shape signature | (B, Din) → trunk → (B, D) → head k → (B, Ck); labels (B, K) with a (B, K) mask. |
| Loss | L = Σ wkLk, each Lk a masked mean divided by its own label count. |
| Why it helps | Shared features see nA + nB examples: less variance, plus a bias that grows with how different the tasks are. |
| Why it hurts | ΔLA ≈ −η(‖gA‖2 + gA·gB): rises when B's gradient is large and opposite. |
| Balancing | Uncertainty: L/(2σ2) + log σ, σ2 → L. GradNorm: match ‖∇WwiLi‖ to Ḡriα. PCGrad: drop the conflicting component. |
| Honest default | A weighted sum with tuned weights, learning rate and weight decay often matches all three (Xin 2022, Kurin 2022). |
| Knobs | Where to split, weights per task, batch sampling per task, trunk capacity, frozen or trained trunk. |
| Did it work? | Per-task metric against its own tuned single-task baseline, over seeds; a transfer matrix; held-out gradient cosines. |
| Failure modes | Negative transfer, a dominant task, unmasked or wrongly normalised missing labels, averages hiding a broken task, under-tuned baselines. |
| Alternatives | Separate models; frozen trunk with independent heads; pretrain then fine-tune; soft sharing; adapters per task. |
This lesson is part of the Builder's Toolkit series on reusable heads and blocks. Go on to:
Now press Present or Teach and explain, out loud and from memory, why sharing a trunk can help one task and hurt another, and derive the uncertainty weight. Then go back to the hero and find the relatedness where task A stops gaining.