Rare Classes and Honest Probabilities

One clip in 200 holds the event. A detector that always says “no” scores 99.5% accuracy and catches nothing. Fix the training, the threshold and the probabilities, one at a time.

Learn why a rare class starves the gradient, how the class prior hides inside every logit, what class weights, focal loss and logit adjustment really change, how to pick a threshold from costs, and how to make a probability mean what it says.

Set the imbalance, pick a loss and press Train. A small head learns in your browser. Slide the threshold along the PR curve. Then press Calibrate and watch the reliability curve move to the diagonal.

You need the sigmoid, cross-entropy and the idea of a trained classifier. We build the rest from zero.

Train a rare-event detector

Ready
1 : 200
0.50

Synthetic data, illustrative numbers. Two features per clip. No-event clips come from a round cloud at the origin; event clips from a tighter cloud up and to the right. The head is logistic regression on the two features, their squares and their product, trained here with Adam on 60 events plus 60 × imbalance non-events. Calibrate first undoes the known prior shift of the loss, then fits one temperature on a separate validation set. We know the true probability of every test point, because we made the data, so the PR curve and the reliability diagram use it instead of noisy 0/1 labels. The scatter shows every training event and a sample of the non-events.

Chapter 0

The 99.5% detector that catches nothing

Meet the problem: a rare event, a head that looks excellent, and the numbers that show it is useless

You build a glass-break detector for a home camera. You have a frozen audio model and a small head: the few trainable layers on top that turn its features into a decision. You label 20,000 ten-second clips. Glass breaks in 100 of them. That is one clip in 200.

You train the head and check its accuracy on held-out clips: 99.5%. It looks superb. Then you look at what it predicts. It says “no glass” for every clip. It has learned the cheapest answer, and accuracy rewards it.

This is class imbalance: one class (the positives, the event) is much rarer than the other (the negatives). The ratio of negatives to positives is the imbalance ratio, here 199 to 1. The share of positives is the base rate or prior, here 0.5%.

Four numbers, not one

Accuracy counts every clip the same. With 199 negatives per positive, the negatives own the score. So we need numbers that look at the positives on their own.

recall
Of the real events, the share you caught. TP is a true positive (a caught event); FN is a false negative (a missed event).
precision
Of your alarms, the share that were real. FP is a false positive: a false alarm.
balanced acc
The mean of the recall on each class. TN is a true negative. Each class counts the same, however rare it is.

Worked: 10,000 clips at 1 : 199 hold 50 events. “Always no”: TP = 0, so recall = 0%; accuracy = 9,950 / 10,000 = 99.5%; balanced accuracy = ½(0 + 1) = 50%, a coin flip.

The accuracy trap

Pick a detector and slide the imbalance. Watch accuracy climb toward 100% as the event gets rarer, whatever the detector does.

1 : 199

Detector A catches 90% of events and alarms on 5% of non-events; Detector B catches 60% and alarms on 0.5%. Each is described by its catch rate (recall) and false-alarm rate on negatives. Those two rates do not depend on the imbalance; precision and accuracy do. Illustrative numbers.

Look at Detector A at 1 : 199. It catches 90% of the glass. It also raises about 498 false alarms per 10,000 clips, against 45 real ones. Its precision is about 8%. Most of its alarms are wrong, yet its accuracy is still 95%.

There are three separate problems hiding here, and this lesson takes them one at a time. Training: the loss and its gradient are dominated by the common class. Deciding: where to put the threshold depends on what a miss and a false alarm cost. Trusting: a probability of 0.3 should mean “30% of such clips have the event”. Most fixes for the first problem break the third, unless you undo them.

The lesson in one line. The class prior lives inside the logit as an additive term; every imbalance fix moves that term, so you must know where it went before you set a threshold or trust a probability.
  1. Why the gradient starves: count what each group of examples pushes on. Why
  2. The prior lives in the logit: derive the log-odds offset from Bayes' rule. Why
  3. Class weights and re-sampling: what they do to the logit, and how to undo it.
  4. Focal loss: derive (1 − p)γ and its gradient, and see what it predicts.
  5. Logit adjustment and balanced softmax: the long tail of many classes.
  6. The threshold is a separate decision: costs, per-class thresholds, and why PR beats ROC.
  7. What a calibrated probability means: reliability diagrams, ECE, and overconfidence.
  8. Recalibrate: temperature, Platt and isotonic, fitted on held-out data.
  9. Build it in PyTorch: by hand, from scratch, and the library lines. How
  10. Did it work?: the PR curve, ECE before and after, per-class recall, sanity checks. Did it work?
  11. Where else, when not, and the cheat sheet. What else What not
Events are 1 in 500 clips. A head never alarms. What are its accuracy and balanced accuracy?

Chapter 1 Why it works

Why the gradient starves

Split the gradient of cross-entropy into what the positives, the hard negatives and the easy negatives push on, and see who wins

A detector head ends in one number per clip, the logit u. The sigmoid turns it into a probability p = σ(u) = 1 / (1 + e−u). Training lowers the binary cross-entropy, the loss −log p for an event and −log(1 − p) for a non-event.

Its derivative with respect to the logit is famously simple. For both classes it is the prediction minus the label.

p
The predicted probability of the event for one clip.
y
The label: 1 for an event, 0 for no event.
∂L/∂u
How much this clip pushes its own logit. Positive means “lower it”.
∂L/∂b
The push on the head's bias b, the one number added to every logit. Every clip votes on it.

Now count votes. At the start, a fresh head says p = 0.5 for everything. Each of 199 negatives pushes the bias down by 0.5. The one positive pushes it up by 0.5.

Worked, at the start: 199 × (0.5 − 0) + 1 × (0.5 − 1) = 99.5 − 0.5 = +99. The first steps learn only “say no more often”.

Later, the head is decent. The positive gets p = 0.5. The negatives are easy negatives: clearly not glass, each at p = 0.05. Each easy negative pushes only a little. But there are 199 of them.

Worked, mid-training: the positive pushes 0.5; the easy negatives push 199 × 0.05 = 9.95, about 20 times more. Many small votes outweigh one large one.

Who pushes the bias?

A 1-D world: non-events (blue) and events (warm) along one feature. The green curve is the head's current p. Slide training progress from a fresh head to the best head. Bars show the total push from each group.

60%
1 : 199

Expected values per 1 + (imbalance) clips, computed exactly by numerical integration. Non-events: normal with mean 0; events: normal with mean 3; both with spread 1. “Easy” means correctly classified with pt > 0.9, where pt is the probability given to the true class.

Where the push stops: the base rate

Training stops moving the bias when its gradient is zero: Σ(pi − yi) = 0. Divide by the number of clips. The average predicted probability equals the share of events. A head trained with cross-entropy matches the base rate on average. Under imbalance that average is tiny, so most predictions sit near 0.

This is not a bug. It is cross-entropy doing its job: matching the frequencies it saw. The trouble is that the rare class's share of the loss and of the gradient is small. With limited capacity or few steps, the head spends its effort on the easy majority.

A head trained to convergence with plain cross-entropy on a set with 2% events. What is the average of its predicted probabilities on that training set?

Chapter 2 Why it works

The prior lives in the logit

Derive from Bayes' rule that the best logit is evidence plus log prior odds, and see the boundary walk past the events

Forget networks for a moment. Suppose you knew exactly how clips with glass and clips without glass are spread over feature space. What is the best possible probability for a clip x? Bayes' rule answers it.

Let p(x | 1) be the likelihood of seeing features x in an event clip, and p(x | 0) in a non-event clip. Let π be the base rate. Bayes' rule gives the probability of an event after seeing x, the posterior. Take its log odds, and the fraction splits into two terms:

u*(x)
The best possible logit: the log odds of an event given the features.
evidence
The log-likelihood ratio: how much more typical x is of events than of non-events. It does not depend on how common events are.
prior
The log prior odds: one constant added to every logit. It depends only on the base rate.

Worked: π = 1/200 → log(1/199) = −5.29. Balanced classes, π = 1/2 → log 1 = 0. Going from balanced to 1 : 199 lowers every logit by 5.29.

That is the whole story of this lesson in one line. A trained head learns both terms. The evidence term is about the sounds, and it transfers. The prior term is about your training set, and it moves the moment the base rate changes.

Where the boundary goes

Make it concrete in one dimension. Non-events have feature x spread as a bell curve with mean 0 and spread 1. Events have mean μ and spread 1. The evidence term is then a straight line, μx − μ2/2. The classifier says “event” where the logit is above 0:

x*
The decision boundary: clips above it are called events.
μ
The distance between the two class means, in units of the spread.
π
The base rate of events.

Worked: μ = 3. Balanced: x* = 1.5, halfway, and 93.3% of events are caught. At π = 1/200: x* = 1.5 + 5.29/3 = 3.26, past the event mean. Only 39.6% of events are caught.

Slide the prior, watch the boundary

Top: the logit as a function of the feature. The dashed line is the evidence alone; the green line adds the log prior odds. Bottom: the two class shapes and the boundary. Red marks the events the best classifier misses.

1 : 199
3.0

Exact Bayes-optimal classifier for two normal classes with equal spread. The class shapes are drawn at the same height so you can see both; the prior only enters through the logit.

Read the result carefully. The best classifier for accuracy at 1 : 199 misses 60% of events. That is not a failure of training. It is the correct answer to the question “maximise accuracy”. If a miss costs more than a false alarm, you are asking a different question. Chapter 6 shows how to ask it.

Your head was trained at a 1 : 1 balance. In deployment, events are 1 in 100. Which change to its logits gives the right probabilities, if the features behave the same?

Chapter 3

Class weights and re-sampling

Weight the rare class, derive exactly what that does to the logit, and undo it with one subtraction

The first fix everyone tries: make each positive count more. A class weight w multiplies the loss of every event clip. In PyTorch it is the pos_weight argument of BCEWithLogitsLoss. Setting w to the imbalance ratio makes the two classes carry equal total weight.

Re-sampling does the same thing with data instead of arithmetic. Oversampling repeats each event clip w times per epoch. Undersampling keeps only 1 in w non-event clips. In expectation, both give the same loss as a class weight w.

What the weight does to the answer

Take one point x, where the true event probability is q. With weight w, the expected loss of predicting p there is w·q·(−log p) + (1 − q)·(−log(1 − p)). Set its derivative in p to zero:

pw
What the weighted head learns to predict, with enough data and capacity.
q
The true probability of an event at x. logit(q) = log(q/(1 − q)).
log w
The shift. The weight acts exactly like multiplying the prior odds by w.

Worked: π = 1/200, w = 199. A clip with true q = 0.05 has logit −2.94. The weighted head predicts logit −2.94 + 5.29 = 2.35, so pw = 0.91. Undo it: 2.35 − 5.29 = −2.94 → 0.05 again.

So weighting by the imbalance ratio trains a head that answers “what if the classes were balanced?”. Its 0.5 threshold catches many more events. But its probabilities are inflated. A clip that is glass 5% of the time is reported as 91%.

Weight it, then undo it

A head with one feature trains here on 100 events and 19,900 non-events. Slide the class weight. Dashed: the true probability. Green: what the weighted head learns. Turn on the undo to subtract log w.

199

Trained by Newton's method on a fixed sample (non-events normal with mean 0, events normal with mean 3, spread 1). The learned bias moves by close to log w; sampling noise makes it not exact. Illustrative numbers.

The general fix: prior correction

Any change of base rate between training and use shifts the logit by the change in log prior odds. If the head saw base rate πtrain (after any weights or re-sampling) and the world has πuse, correct it like this Saerens 2002:

πtrain
The effective base rate the loss saw. With weight w on events, the odds are w times the data's odds.
πuse
The base rate where the head runs. A hospital and a screening clinic have different ones.
uuse
The corrected logit. Only the constant moves; the ranking of clips does not.

Worked: trained balanced (odds 1), used at 1 in 1,000 (odds 1/999): add log(1/999) = −6.91 to every logit.

You trained with pos_weight = 50 and the head says p = 0.8 for a clip. Ignoring capacity effects, what is the probability at the data's base rate?

Chapter 4

Focal loss: turn down the easy examples

Derive the focal loss and its gradient, see how γ silences easy negatives, and find what it does to the probabilities

Class weights treat every negative the same. But Chapter 1 showed the real crowd: easy negatives, each pushing a little. Lin, Goyal, Girshick, He and Dollár met this in dense object detection, where an image yields about 100,000 candidate boxes and almost all are background Focal Loss · §4. Their fix scales each example's loss by how wrong it still is.

Write pt for the probability the head gives the true class: p for an event, 1 − p for a non-event. Cross-entropy is −log pt. The focal loss multiplies it by a modulating factor:

pt
The probability given to the true class. Near 1 means easy, near 0 means badly wrong.
(1 − pt)γ
The modulating factor. Near 0 for easy examples, near 1 for hard ones. γ ≥ 0 is the focusing parameter; γ = 0 gives plain cross-entropy.
FL
The focal loss for one example. Lin et al. found γ = 2 worked best Focal Loss · §3.2.

Worked, γ = 2: pt = 0.9 → factor 0.12 = 0.01, so 100× less loss than cross-entropy. pt ≈ 0.968 → about 1,000× less. pt ≤ 0.5 → at most 4× less Focal Loss · §3.2.

The gradient, derived

Take an event, so pt = p = σ(u). The sigmoid's derivative is dp/du = p(1 − p). Differentiate the product (1 − p)γ·(−log p) in p, then multiply by p(1 − p):

p − 1
Cross-entropy's own gradient for an event. Set γ = 0 and you get it back.
γ p log p
An extra term from differentiating the factor. It is negative, so it adds to the push on wrong examples.
∂FL/∂u
Matches Lin et al.'s Appendix B, eq. 10 Focal Loss · App. B. A non-event is the mirror image.

Worked, γ = 2, an event at p = 0.9: cross-entropy's gradient is −0.1. Focal: 0.01 × (2 × 0.9 × ln 0.9 − 0.1) = 0.01 × (−0.190 − 0.1) = −0.0029, about 34× smaller.

Turn the focusing knob

Slide γ. The dashed curve is cross-entropy. Switch views to see the gradient, and what a head trained with focal loss would predict at each true probability.

2.0

Computed exactly. The share of the gradient from easy negatives uses the 1-D world of Chapter 1 at 1 : 199 and its best head. “What it predicts” minimises the expected focal loss at each true probability q, numerically.

The price: probabilities pulled toward 0.5

Focal loss changes the answer the head converges to. For a true q, the head minimises q·FL(p) + (1 − q)·FL(1 − p). Cross-entropy's minimum is p = q. Focal's is not. Because confident answers earn almost no reward, the minimum is pulled toward 0.5.

Computed, γ = 2: true q = 0.9 → focal head predicts 0.69. q = 0.99 → 0.85. q = 0.1 → 0.31. The ranking is kept; the scale is squashed.

The squash is close to a constant scale on the logit. So one temperature, fitted afterwards, mostly removes it (Chapter 8). In the hero, focal loss gives the same ranking as cross-entropy but underconfident probabilities; Calibrate fits a temperature below 1 and fixes them.

With γ = 2, how much is the loss of a non-event with p = 0.1 (so pt = 0.9) scaled, compared with cross-entropy?

Chapter 5

Logit adjustment and balanced softmax

Move the prior term on purpose, after training or inside the loss, for a long tail of many classes

Many real problems have not two classes but hundreds. A bird classifier sees thousands of robins and a handful of rare warblers. This shape is the long tail. Overall accuracy rewards the common head classes. What you often want instead is the balanced error: the average of each class's error rate, so every class counts the same.

Chapter 2 generalises to many classes. A softmax head with logits fy(x) learns the evidence plus log πy, the log prior of class y. Menon et al. showed that the best rule for the balanced error is to remove that prior: predict the class with the largest fy(x) − log πy Menon 2021 · §3, eq. 8. They give two ways to do it.

πy
The class prior, estimated by its share of the training set.
τ
How strongly to adjust. τ = 1 is the exact Bayes correction; Menon et al. also tune it.
post-hoc
Train with plain cross-entropy. At test time, subtract τ log πy, which raises rare classes most.
loss
The logit-adjusted loss: add τ log πy inside the training softmax only. The head's own fy then learns the evidence, and you predict with argmax fy directly.

Worked, three classes with 5,000, 500 and 50 examples: log π = (−0.10, −2.41, −4.71). Plain logits (2.0, 0.5, 0.0) pick class 1. Adjusted: (2.10, 2.91, 4.71) pick the rare class 3.

Why does adding the prior during training remove it from f? Because the loss already supplies the prior term. The head does not need to learn it, so fy keeps only the evidence. This is the binary trick of Chapter 3 in reverse: there you saw weights push the prior in; here you add it outside the head so the head can leave it out.

Balanced softmax, by Ren et al., reaches the same place from another direction. It replaces the softmax in training by njeηj / Σi nieηi, where nj is the count of class j Balanced Softmax · eq. 3, 4. Since nj is proportional to πj, this is the logit-adjusted loss with τ = 1.

A long tail of three classes

Head class (blue), middle class (yellow), tail class (warm). Shaded regions show what the head predicts. Pick a method, set the imbalance and press Train. Bars give each class's recall on a balanced test set.

100 : 1
1.00

Trained here: a linear softmax head on two features, 600 head-class points, 600/√ratio middle-class points, 600/ratio tail-class points (at least 6), 300 Adam steps. Recall is measured on 500 new points per class. Post-hoc τ acts live; the adjusted loss uses τ when you train. Illustrative numbers.

What the paper measured

Balanced error (%), lower is betterCIFAR-10-LTImageNet-LTiNaturalist
Plain cross-entropy (ERM)27.1653.1138.66
Post-hoc adjustment, τ = 122.6049.6633.98
Logit-adjusted loss, τ = 122.3348.8933.64

Source: Menon 2021 · Table 3, averages of 5 runs; the CIFAR sets are made long-tailed with a 100 : 1 ratio between the largest and smallest class Menon 2021 · §6.2. Tuning τ post-hoc reached 19.08 on CIFAR-10-LT.

You trained with the logit-adjusted loss (τ = 1). In deployment you want honest probabilities at the training class frequencies. What do you do to the logits?

Chapter 6

The threshold is a separate decision

Choose the threshold from the cost of a miss and a false alarm, give each class its own, and read the PR curve instead of the ROC curve

So far, every fix moved the logit's constant term. That is the same as moving the threshold. A head that says p, thresholded at t, makes the same decisions as a head that says logit(p) − logit(t), thresholded at 0.5. So a cleaner way to think is: train for good scores, then choose the threshold on purpose.

From costs to a threshold

Suppose a missed glass break costs cmiss and a false alarm costs cFA. For a clip with a calibrated probability q, alarming costs cFA(1 − q) on average. Staying silent costs cmiss·q. Alarm when silence costs more:

q
The calibrated probability of an event for this clip.
t*
The threshold that minimises expected cost Elkan 2001.
cmiss
The cost of a missed event.
cFA
The cost of a false alarm.

Worked: a miss costs 50 times a false alarm → t* = 1/51 = 0.0196. Equal costs → t* = 0.5, the accuracy threshold.

Notice what this needs: a calibrated q. If the head's probabilities are inflated by a class weight, the cost rule gives the wrong threshold. This is the link between the two halves of the lesson.

Why PR beats ROC under imbalance

The ROC curve plots recall (the true-positive rate) against the false-positive rate, FP divided by all negatives. The PR curve plots precision against recall. The difference is the denominator. ROC divides false alarms by the huge number of negatives, so thousands of false alarms look like a small rate.

Worked, 10,000 clips at 1 : 199: 50 events, 9,950 non-events. Recall 0.9 → 45 caught. False-positive rate 0.05 → 497.5 false alarms. The ROC point (0.05, 0.90) looks excellent. Precision = 45 / 542.5 = 8.3%.

Same head, two curves

One fixed, calibrated head. Slide the imbalance: the ROC curve does not move; the PR curve sinks. Slide the cost ratio: the teal point moves to the cost-optimal threshold. The hollow point is the 0.5 threshold.

1 : 199
50

Exact curves for the 1-D world with separation 2 and a calibrated head. Counts are per 10,000 clips. Illustrative numbers.

Saito and Rehmsmeier made this argument with simulations and a review of published work: on imbalanced data, the PR plot is more informative than the ROC plot Saito 2015. Summarise a PR curve by its area, the average precision (AP). A useless head scores AP equal to the base rate, 0.005 here, not 0.5 as with ROC.

One threshold per class

With many classes, a single global threshold assumes every class has the same costs, base rate and calibration. They rarely do. Choose each class's threshold on validation data: either from its costs with the rule above, or to reach a target, such as “recall at least 90% for the alarm class”.

A fraud team can review false alarms cheaply; a missed fraud costs 200 times a review. With calibrated probabilities, which threshold minimises expected cost?

Chapter 7

What a calibrated probability means

Define calibration, draw a reliability diagram, compute ECE by hand, and see why it is noisy on small sets and why modern nets fail it

A weather forecaster says “30% chance of rain” on many days. If it rains on about 30% of those days, the forecaster is calibrated. The same holds for a head: among all clips where it says 0.3, about 30% should hold the event.

P(Y = 1 | p̂ = p)
The real share of events among clips that got score p.
p
The score the head gave. Calibration asks the two to match.

Calibration and ranking are different. A head can rank perfectly and be badly calibrated: multiply every logit by 3 and the order does not change, but every probability moves. A head can also be calibrated and useless: say the base rate for every clip.

The reliability diagram and ECE

To check, group clips into M bins by score, say [0, 0.1), [0.1, 0.2) and so on. In each bin, compare the average score (the confidence) with the share of real events (the accuracy of the bin). Plot one against the other: that is a reliability diagram. A calibrated head lies on the diagonal. The expected calibration error averages the gaps, weighted by bin size Guo 2017 · §2, eq. 3:

|Bm|
The number of clips in bin m; n is the total.
acc(Bm)
The share of real events in the bin.
conf(Bm)
The average score in the bin.
ECE
0 for a perfectly calibrated head. Guo et al. used M = 15 bins.

Worked, 200 clips in 3 bins: 120 clips with confidence 0.10, real rate 0.05; 50 with 0.50, real 0.40; 30 with 0.85, real 0.70. ECE = 0.6 × 0.05 + 0.25 × 0.10 + 0.15 × 0.15 = 0.0775, or 7.75%.

Measure calibration, and its noise

Bars are the real event rate in each bin, measured on a sampled validation set; the red parts are the gaps from the diagonal. Make the model overconfident. Then shrink the set and press Resample: even a perfect model shows gaps.

2.00
2,000
10

Base rate 20%, the 1-D world with separation 2. The model's logit is the true logit times the slider. True ECE is computed on 400,000 points; the noise floor is the average ECE of a perfectly calibrated model over 20 validation sets of the chosen size. Illustrative numbers.

Why modern nets are overconfident

Guo, Pleiss, Sun and Weinberger compared a 5-layer LeNet with a 110-layer ResNet on CIFAR-100. The ResNet was more accurate (error 30.6% against 44.9%), yet its average confidence sat well above its accuracy, while LeNet's matched Guo 2017 · Fig. 1. Its ECE was 16.53%, against 4.85% for LeNet Guo 2017 · Table 1.

They traced it to depth, width, Batch Normalization and less weight decay Guo 2017 · §3, Fig. 2. The mechanism is in the loss. Once a network classifies almost every training example correctly, it can still lower its training NLL (the cross-entropy) by raising confidence. So test NLL overfits while test error keeps improving Guo 2017 · §3, Fig. 3.

Among 400 clips your head scored between 0.6 and 0.7 (average 0.65), 180 held the event. What does this bin say?

Chapter 8

Recalibrate: temperature, Platt and isotonic

Fit a small map from scores to probabilities on held-out data, derive why temperature scaling is one easy convex problem, and know when it is not enough

You cannot calibrate on the training set: the head has already overfit its confidence there. Keep a validation set the head never trained on. Freeze the head. Learn a tiny map from its logit to a better probability.

Temperature scaling

The simplest map has one number. Divide every logit by a temperature T > 0 Guo 2017 · §4.2, eq. 9. T > 1 softens overconfident scores; T < 1 sharpens underconfident ones. The order of the logits is kept, so the decisions at the argmax and the PR curve do not change.

z
The frozen head's logit for a validation clip.
T
The temperature, fitted by minimising the validation NLL, the cross-entropy of the rescaled probabilities.
p
The recalibrated probability.
dNLL/dβ
The slope of the NLL in β. Its second derivative, Σ σ(1 − σ) zi2, is never negative, so the problem is convex: one minimum, and Newton's method finds it in a few steps.

Worked: a logit of 4 gives 0.982. With T = 2 it becomes σ(2) = 0.881. The optimal T makes the residuals p − y, weighted by the logits, sum to zero.

Guo et al. found this one-number fix often the best of all the methods they tried. On CIFAR-100, the ResNet-110's ECE fell from 16.53% to 1.26%; on ImageNet, a ResNet-152's fell from 5.48% to 1.86% Guo 2017 · Table 1. A conjugate-gradient solver finds T in about 10 iterations Guo 2017 · §5.

Platt scaling and isotonic regression

Platt scaling fits two numbers: p = σ(a·z + b) Platt 1999. The a is a temperature; the b is a shift. You need the shift when the miscalibration includes a prior change, such as a class weight. Temperature alone cannot move the constant term.

Isotonic regression fits any non-decreasing step function from score to probability Zadrozny 2002. The pool-adjacent-violators algorithm finds it: sort by score, and merge neighbouring groups whose event rates go down. It can fix any monotone distortion, but it has many free steps, so it needs more validation data.

Fit the map

Pick what went wrong with the head and a method, then press Fit. Left: test reliability before (green) and after (purple). Right: the fit itself. Try a small validation set with isotonic.

Base rate 10%, the 1-D world with separation 2. Fitted by Newton's method (temperature, Platt) or pool-adjacent-violators (isotonic) on sampled validation labels; tested against the true probability on 200,000 points. Illustrative numbers.

In Guo et al.'s tests, isotonic regression was less accurate than temperature scaling on most vision sets, for example 4.99% against 1.26% ECE for the CIFAR-100 ResNet-110 Guo 2017 · Table 1. Its flexibility costs data.

A head was trained with pos_weight = 20 and you do not know it. You fit only a temperature on validation data. What happens?

Chapter 9 How

Build it in PyTorch

The same four tools three ways: by hand, from scratch in PyTorch, and as library lines

The data flow is short. A frozen backbone gives a feature per clip (B × D). The head gives logits (B × 1 for one event, or B × C for C classes). The loss sees the logits and the labels (B). Every tool in this lesson acts on the logits, the loss, or the map after the head. None of them touches the backbone.

By hand first

Before any code, compute one example by hand and check that the code agrees. Use the calculator: it runs the formulas of Chapters 1 to 8 for one clip.

One clip, every formula

Set the head's logit for one clip and its label. The rows show each loss and its gradient with respect to the logit, and the probability after the prior fix and the temperature.

2.20
2.0
1/200
1.00

Exact formulas. The logit-adjusted loss adds log(π/(1 − π)) to z inside the loss, as in the binary form of Chapter 5. Set z = 2.20, γ = 2 and an event to match the worked example of Chapter 4.

Focal loss, from scratch

Use binary_cross_entropy_with_logits for −log pt: it is stable for large logits. Then multiply by the factor. Autograd derives the gradient of Chapter 4 for you; the printout checks it.

import torch
import torch.nn.functional as F

def focal_loss(logits, y, gamma=2.0, alpha=None):
    # logits: (B,) raw scores; y: (B,) floats, 1 = event, 0 = no event
    ce = F.binary_cross_entropy_with_logits(logits, y, reduction='none')  # -log p_t
    p = torch.sigmoid(logits)
    p_t = y * p + (1 - y) * (1 - p)           # probability given to the true class
    loss = (1 - p_t) ** gamma * ce             # easy examples (p_t near 1) fade out
    if alpha is not None:                      # optional class weight, as in Lin et al.
        loss = (y * alpha + (1 - y) * (1 - alpha)) * loss
    return loss.mean()

logits = torch.tensor([2.1972, 2.1972, 0.0], requires_grad=True)   # p = 0.9, 0.9, 0.5
y = torch.tensor([1.0, 0.0, 1.0])              # an easy positive, a hard negative, a coin flip
print(focal_loss(logits, y, gamma=0.0).item())  # 1.0337: plain cross-entropy
print(focal_loss(logits, y, gamma=2.0).item())  # 0.6798
focal_loss(logits, y, gamma=2.0).backward()
print(logits.grad * 3)   # dL/dlogit per example: [-0.0029, 1.1020, -0.2983]

The printed gradients match the hand formula: −0.0029 for the easy event (cross-entropy gives −0.1) and 1.102 for the hard non-event, which is more than cross-entropy's 0.9. The library version is one line, torchvision.ops.sigmoid_focal_loss(logits, y, alpha=-1, gamma=2, reduction='mean'); it gives 0.6798 on these inputs. Its default α is 0.25, as in the paper, so pass alpha=-1 to turn the weight off.

Logit adjustment, from scratch

import torch
import torch.nn.functional as F

counts = torch.tensor([5000.0, 500.0, 50.0])     # training examples per class
log_prior = torch.log(counts / counts.sum())    # (C,) = log pi_y

def la_loss(logits, y, tau=1.0):
    # logits: (B, C) from the head; y: (B,) class ids.
    # Add tau * log pi during training only. With tau = 1 this is Balanced Softmax.
    return F.cross_entropy(logits + tau * log_prior, y)

def la_posthoc(logits, tau=1.0):
    # For a head trained with plain cross-entropy: subtract tau * log pi at test time.
    return logits - tau * log_prior

print(log_prior)                                 # tensor([-0.1044, -2.4069, -4.7095])
f = torch.tensor([[2.0, 0.5, 0.0]])              # plain-CE logits for one input
print(f.argmax(1).item(), la_posthoc(f).argmax(1).item())   # 0 before, 2 after
print(la_loss(f, torch.tensor([2])).item())      # 6.6286 if the true class is the rare one

Why each choice. The prior comes from training counts, so it is fixed before training. The adjustment is added outside the head, so the head's weights are a plain linear layer. Use la_loss or la_posthoc, never both (Chapter 5).

Temperature scaling, from scratch

Fit on validation logits from the frozen head. Optimise log T rather than T, so T stays positive. L-BFGS solves this one-number convex problem in a few steps.

import torch
import torch.nn.functional as F
torch.manual_seed(0)

def fit_temperature(val_logits, val_y):
    # val_logits: (N, C) from the frozen, trained model; val_y: (N,) class ids
    log_t = torch.zeros(1, requires_grad=True)          # T = exp(log_t) stays positive
    opt = torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)
    def closure():
        opt.zero_grad()
        loss = F.cross_entropy(val_logits / log_t.exp(), val_y)   # NLL at this T
        loss.backward()
        return loss
    opt.step(closure)
    return log_t.exp().item()

true_logits = torch.randn(5000, 10) * 1.5               # a world where these are honest
val_y = torch.distributions.Categorical(logits=true_logits).sample()
model_logits = 2.5 * true_logits                        # an overconfident model
T = fit_temperature(model_logits, val_y)
print(round(T, 2))                                      # 2.5: it finds the overconfidence
probs = F.softmax(model_logits / T, dim=1)              # calibrated; argmax unchanged

The sklearn one-liner, and undoing a weight

For a classic model, CalibratedClassifierCV wraps the model, trains it on folds, and fits a sigmoid (Platt) or isotonic map on the held-out fold. Here it repairs a class-weighted logistic regression. Undoing the weight by subtracting log w is the exact fix in the limit; on finite data it is close but not perfect.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV
rng = np.random.default_rng(0)
X = np.r_[rng.normal(0, 1, (19900, 2)), rng.normal(2.5, 1, (100, 2))]   # 1 event in 200
y = np.r_[np.zeros(19900), np.ones(100)]
x0 = np.array([[1.5, 1.5]])                      # true P(event | x0) = 0.017

raw = LogisticRegression(class_weight='balanced').fit(X, y)    # weight ratio w = 199
cal = CalibratedClassifierCV(LogisticRegression(class_weight='balanced'),
                             method='sigmoid', cv=5).fit(X, y)  # refits a sigmoid on held-out folds
p_raw = raw.predict_proba(x0)[0, 1]

def undo_weight(p, w):
    # exact fix for a positive-class weight w: subtract log w from the logit
    z = np.log(p / (1 - p)) - np.log(w)
    return 1 / (1 + np.exp(-z))

print(round(p_raw, 3), round(undo_weight(p_raw, 199), 3), round(cal.predict_proba(x0)[0, 1], 3))
# 0.596 0.007 0.023  (truth: 0.017)
In the focal-loss code, why compute −log pt with binary_cross_entropy_with_logits instead of -torch.log(p_t)?

Chapter 10 Did it work?

Did it work?

The report card for an imbalanced head: the PR curve and AP, reliability and ECE before and after, recall per class, and the sanity checks that catch bugs

A single number cannot tell you whether an imbalanced head works. You need a small set of plots, each answering one question.

QuestionWhat to showWhat good looks like
Does it rank events above non-events?PR curve and AP, with the base rate markedAP well above the base rate; stable across seeds.
Is the threshold right?Precision and recall at the chosen threshold, from validationMeets the target recall or minimises cost on validation; similar on test.
Do the probabilities mean what they say?Reliability diagram and ECE before and after calibration, plus NLLPoints near the diagonal; ECE above its noise floor only by a little.
Does every class work?Recall per class, balanced errorNo class near 0 recall; the tail is not sacrificed for the head.
Is the signal real?Shuffled-label runAP falls to the base rate. If not, something leaks.

The report card

This reads the head you last trained in the hero at the top. Pick a panel. Press Shuffle labels to retrain the same head on scrambled labels.

Uses the hero's head, threshold and test set. “After” is the hero's calibration fitted on its validation set; if you have not pressed Calibrate, it is fitted here the same way. Illustrative numbers.

The analysis in code

The same report in a few lines of numpy and sklearn. The ECE function bins the event probability with equal-width bins, as Guo et al. do. The data are the 1-D world at 1 : 199 with the true probability as the score, plus an overconfident copy.

import numpy as np
from sklearn.metrics import average_precision_score, precision_recall_curve, recall_score

def ece(p, y, bins=15):
    # p: (N,) predicted P(event); y: (N,) 0/1. Equal-width bins, as in Guo et al.
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
    total = 0.0
    for b in range(bins):
        m = idx == b
        if m.any():
            total += m.mean() * abs(y[m].mean() - p[m].mean())   # |B|/n * |acc - conf|
    return total

rng = np.random.default_rng(1)
x = np.r_[rng.normal(0, 1, 19900), rng.normal(3, 1, 100)]       # scores of 20,000 clips
y = np.r_[np.zeros(19900), np.ones(100)]
q = 1 / (1 + np.exp(-(3 * x - 4.5 + np.log(1 / 199))))           # the true P(event | x)
p_over = 1 / (1 + np.exp(-2.5 * np.log(q / (1 - q))))            # an overconfident model

print('AP', round(average_precision_score(y, q), 3), 'base rate', y.mean())  # AP 0.6, base rate 0.005
prec, rec, thr = precision_recall_curve(y, q)                    # the full PR curve
print('ECE honest', round(ece(q, y), 4), 'overconfident', round(ece(p_over, y), 4))  # 0.0009, 0.003
print('per-class recall at 0.5', recall_score(y, q > 0.5, average=None).round(3))  # [0.999 0.38]

Look at the ECE numbers: 0.09% for the honest score and 0.3% for one that is 2.5 times too confident. Both look tiny, because 99% of clips sit near 0 and are right. That is the “which ECE?” trap the report card shows.

Common bugs, and the check that catches each

After retraining on shuffled labels, your head's AP is 0.31 with a base rate of 0.005. What is the most likely cause?

Chapter 11 What else What not

Where else, when not, and the cheat sheet

The same three steps in six fields, the cases where each tool hurts, and the whole block on one screen

Where else it works

The recipe never changes: train for a good ranking, put the prior where you want it, choose the threshold from costs, then calibrate on held-out data. What changes between fields is the base rate, the cost ratio, and whether anyone reads the probability.

FieldThe rare classWhat changes
Sound event detectionA glass break or gunshot in hours of home audioReport false alarms per hour; a threshold per event class; see event detection heads.
Medical screeningA rare disease in a screening populationA miss costs far more than a recall visit; the clinic's base rate differs from the training hospital's, so apply prior correction; doctors read the probability, so calibrate.
Fraud detectionA fraudulent payment among thousandsThe threshold is often set by review capacity (the top k per day) rather than a cost; the base rate drifts, so recalibrate often.
Defect detectionA cracked part on a production lineVery few defect images: undersample good parts, weight defects, and pick the threshold for a target recall such as 99%.
Long-tail speciesRare birds or frogs in bioacoustic recordings, rare plants in photosHundreds of classes: logit adjustment or balanced softmax, per-class thresholds, and balanced error; see Perch 2.
LLM classifier confidenceA harmful prompt, or a question the model gets wrongFit a temperature on the classifier's logits; check the reliability of the rare “flag” class; see LLM calibration.

When not to use it

ToolAvoid it whenUse instead
Class weights or re-samplingYou need calibrated probabilities and will not correct themPlain CE plus a chosen threshold; or weights, then subtract log w or refit Platt.
Focal lossData is balanced, or easy examples are not the bulk of the lossPlain cross-entropy with a prior-initialised bias.
One global thresholdClasses differ in base rate, cost or calibrationPer-class thresholds chosen on validation.
ECE as the only checkThe validation set is tiny or almost all scores are near 0Reliability with confidence bands, NLL, Brier score, ECE on flagged clips.
Temperature scalingThe base rate shifted or a class weight was usedPrior correction first, or Platt scaling.
Isotonic regressionYou have few validation eventsTemperature or Platt scaling.
Logit adjustmentYou care about accuracy at the training frequenciesPlain cross-entropy: it already targets that.

Choose a recipe

Which recipe for my problem?

Choose an answer in each row. The pipeline below updates.

A rule of thumb from this lesson, not a law. Always compare against plain cross-entropy with a tuned threshold.

The block on one screen

PartIn one line
Shape signature(B, D) features → head → (B, C) logits → [+ τ log π in the loss] → threshold per class → [÷ T, ± prior] → (B, C) probabilities.
The key factlogit = evidence + log prior odds. Every imbalance fix moves the second term.
Weightspos_weight w adds log w to the logit; undo by subtracting it.
Focal−(1 − pt)γ log pt, γ = 2; silences easy examples; pulls probabilities toward 0.5.
Logit adjustmentTrain on f + τ log π (or predict f − τ log π); targets the balanced error. Balanced softmax = τ = 1.
Thresholdt* = cFA / (cFA + cmiss) on calibrated q; per class; chosen on validation.
CalibrationTemperature (1 number, convex), Platt (adds a shift), isotonic (needs data); fit on held-out data.
Did it work?PR curve and AP vs base rate; reliability and ECE before/after with a noise floor; recall per class; shuffled labels.
Knobsw, γ, α, τ, the threshold per class, T (or a, b), bins M.

Honest limits

Where this sits, and where to go next

This lesson is part of the Builder's Toolkit series on reusable blocks for heads on frozen backbones. Go on to:

The takeaway. A logit is evidence plus log prior odds. Imbalance fixes move the prior term, which raises recall at 0.5 but inflates the probabilities. So train for a good ranking, choose the threshold from costs on validation data, judge the ranking with the PR curve, and recalibrate on held-out data before anyone trusts a number.
A rare-disease screen: 1 in 1,000 patients, a miss costs 100 times a recall visit, and doctors see the probability. Which pipeline fits?

Now press Present or Teach and explain, out loud and from memory, why the prior is an additive term in the logit and what each fix does to it. Then go back to the hero, train with class weights at 1 : 500, and predict where the reliability curve will sit before you press Calibrate.

Rare Classes and Honest Probabilities. Sources: T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, “Focal Loss for Dense Object Detection” (ICCV 2017, arXiv); A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, S. Kumar, “Long-tail learning via logit adjustment” (ICLR 2021, arXiv); J. Ren, C. Yu, S. Sheng, X. Ma, H. Zhao, S. Yi, H. Li, “Balanced Meta-Softmax for Long-Tailed Visual Recognition” (NeurIPS 2020, arXiv); C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, “On Calibration of Modern Neural Networks” (ICML 2017, arXiv); J. Platt, “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods” (Advances in Large Margin Classifiers, 1999); B. Zadrozny, C. Elkan, “Transforming Classifier Scores into Accurate Multiclass Probability Estimates” (KDD 2002); C. Elkan, “The Foundations of Cost-Sensitive Learning” (IJCAI 2001); M. Saerens, P. Latinne, C. Decaestecker, “Adjusting the Outputs of a Classifier to New a Priori Probabilities” (Neural Computation 2002); T. Saito, M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets” (PLOS ONE 2015, DOI).
Back to Gleams