One clip in 200 holds the event. A detector that always says “no” scores 99.5% accuracy and catches nothing. Fix the training, the threshold and the probabilities, one at a time.
Learn why a rare class starves the gradient, how the class prior hides inside every logit, what class weights, focal loss and logit adjustment really change, how to pick a threshold from costs, and how to make a probability mean what it says.
Set the imbalance, pick a loss and press Train. A small head learns in your browser. Slide the threshold along the PR curve. Then press Calibrate and watch the reliability curve move to the diagonal.
You need the sigmoid, cross-entropy and the idea of a trained classifier. We build the rest from zero.
Synthetic data, illustrative numbers. Two features per clip. No-event clips come from a round cloud at the origin; event clips from a tighter cloud up and to the right. The head is logistic regression on the two features, their squares and their product, trained here with Adam on 60 events plus 60 × imbalance non-events. Calibrate first undoes the known prior shift of the loss, then fits one temperature on a separate validation set. We know the true probability of every test point, because we made the data, so the PR curve and the reliability diagram use it instead of noisy 0/1 labels. The scatter shows every training event and a sample of the non-events.
Chapter 0
Meet the problem: a rare event, a head that looks excellent, and the numbers that show it is useless
You build a glass-break detector for a home camera. You have a frozen audio model and a small head: the few trainable layers on top that turn its features into a decision. You label 20,000 ten-second clips. Glass breaks in 100 of them. That is one clip in 200.
You train the head and check its accuracy on held-out clips: 99.5%. It looks superb. Then you look at what it predicts. It says “no glass” for every clip. It has learned the cheapest answer, and accuracy rewards it.
This is class imbalance: one class (the positives, the event) is much rarer than the other (the negatives). The ratio of negatives to positives is the imbalance ratio, here 199 to 1. The share of positives is the base rate or prior, here 0.5%.
Accuracy counts every clip the same. With 199 negatives per positive, the negatives own the score. So we need numbers that look at the positives on their own.
Worked: 10,000 clips at 1 : 199 hold 50 events. “Always no”: TP = 0, so recall = 0%; accuracy = 9,950 / 10,000 = 99.5%; balanced accuracy = ½(0 + 1) = 50%, a coin flip.
Pick a detector and slide the imbalance. Watch accuracy climb toward 100% as the event gets rarer, whatever the detector does.
Detector A catches 90% of events and alarms on 5% of non-events; Detector B catches 60% and alarms on 0.5%. Each is described by its catch rate (recall) and false-alarm rate on negatives. Those two rates do not depend on the imbalance; precision and accuracy do. Illustrative numbers.
Look at Detector A at 1 : 199. It catches 90% of the glass. It also raises about 498 false alarms per 10,000 clips, against 45 real ones. Its precision is about 8%. Most of its alarms are wrong, yet its accuracy is still 95%.
There are three separate problems hiding here, and this lesson takes them one at a time. Training: the loss and its gradient are dominated by the common class. Deciding: where to put the threshold depends on what a miss and a false alarm cost. Trusting: a probability of 0.3 should mean “30% of such clips have the event”. Most fixes for the first problem break the third, unless you undo them.
Chapter 1 Why it works
Split the gradient of cross-entropy into what the positives, the hard negatives and the easy negatives push on, and see who wins
A detector head ends in one number per clip, the logit u. The sigmoid turns it into a probability p = σ(u) = 1 / (1 + e−u). Training lowers the binary cross-entropy, the loss −log p for an event and −log(1 − p) for a non-event.
Its derivative with respect to the logit is famously simple. For both classes it is the prediction minus the label.
Now count votes. At the start, a fresh head says p = 0.5 for everything. Each of 199 negatives pushes the bias down by 0.5. The one positive pushes it up by 0.5.
Worked, at the start: 199 × (0.5 − 0) + 1 × (0.5 − 1) = 99.5 − 0.5 = +99. The first steps learn only “say no more often”.
Later, the head is decent. The positive gets p = 0.5. The negatives are easy negatives: clearly not glass, each at p = 0.05. Each easy negative pushes only a little. But there are 199 of them.
Worked, mid-training: the positive pushes 0.5; the easy negatives push 199 × 0.05 = 9.95, about 20 times more. Many small votes outweigh one large one.
A 1-D world: non-events (blue) and events (warm) along one feature. The green curve is the head's current p. Slide training progress from a fresh head to the best head. Bars show the total push from each group.
Expected values per 1 + (imbalance) clips, computed exactly by numerical integration. Non-events: normal with mean 0; events: normal with mean 3; both with spread 1. “Easy” means correctly classified with pt > 0.9, where pt is the probability given to the true class.
Training stops moving the bias when its gradient is zero: Σ(pi − yi) = 0. Divide by the number of clips. The average predicted probability equals the share of events. A head trained with cross-entropy matches the base rate on average. Under imbalance that average is tiny, so most predictions sit near 0.
This is not a bug. It is cross-entropy doing its job: matching the frequencies it saw. The trouble is that the rare class's share of the loss and of the gradient is small. With limited capacity or few steps, the head spends its effort on the easy majority.
Chapter 2 Why it works
Derive from Bayes' rule that the best logit is evidence plus log prior odds, and see the boundary walk past the events
Forget networks for a moment. Suppose you knew exactly how clips with glass and clips without glass are spread over feature space. What is the best possible probability for a clip x? Bayes' rule answers it.
Let p(x | 1) be the likelihood of seeing features x in an event clip, and p(x | 0) in a non-event clip. Let π be the base rate. Bayes' rule gives the probability of an event after seeing x, the posterior. Take its log odds, and the fraction splits into two terms:
Worked: π = 1/200 → log(1/199) = −5.29. Balanced classes, π = 1/2 → log 1 = 0. Going from balanced to 1 : 199 lowers every logit by 5.29.
That is the whole story of this lesson in one line. A trained head learns both terms. The evidence term is about the sounds, and it transfers. The prior term is about your training set, and it moves the moment the base rate changes.
Make it concrete in one dimension. Non-events have feature x spread as a bell curve with mean 0 and spread 1. Events have mean μ and spread 1. The evidence term is then a straight line, μx − μ2/2. The classifier says “event” where the logit is above 0:
Worked: μ = 3. Balanced: x* = 1.5, halfway, and 93.3% of events are caught. At π = 1/200: x* = 1.5 + 5.29/3 = 3.26, past the event mean. Only 39.6% of events are caught.
Top: the logit as a function of the feature. The dashed line is the evidence alone; the green line adds the log prior odds. Bottom: the two class shapes and the boundary. Red marks the events the best classifier misses.
Exact Bayes-optimal classifier for two normal classes with equal spread. The class shapes are drawn at the same height so you can see both; the prior only enters through the logit.
Read the result carefully. The best classifier for accuracy at 1 : 199 misses 60% of events. That is not a failure of training. It is the correct answer to the question “maximise accuracy”. If a miss costs more than a false alarm, you are asking a different question. Chapter 6 shows how to ask it.
Chapter 3
Weight the rare class, derive exactly what that does to the logit, and undo it with one subtraction
The first fix everyone tries: make each positive count more. A class weight w multiplies the loss of every event clip. In PyTorch it is the pos_weight argument of BCEWithLogitsLoss. Setting w to the imbalance ratio makes the two classes carry equal total weight.
Re-sampling does the same thing with data instead of arithmetic. Oversampling repeats each event clip w times per epoch. Undersampling keeps only 1 in w non-event clips. In expectation, both give the same loss as a class weight w.
Take one point x, where the true event probability is q. With weight w, the expected loss of predicting p there is w·q·(−log p) + (1 − q)·(−log(1 − p)). Set its derivative in p to zero:
Worked: π = 1/200, w = 199. A clip with true q = 0.05 has logit −2.94. The weighted head predicts logit −2.94 + 5.29 = 2.35, so pw = 0.91. Undo it: 2.35 − 5.29 = −2.94 → 0.05 again.
So weighting by the imbalance ratio trains a head that answers “what if the classes were balanced?”. Its 0.5 threshold catches many more events. But its probabilities are inflated. A clip that is glass 5% of the time is reported as 91%.
A head with one feature trains here on 100 events and 19,900 non-events. Slide the class weight. Dashed: the true probability. Green: what the weighted head learns. Turn on the undo to subtract log w.
Trained by Newton's method on a fixed sample (non-events normal with mean 0, events normal with mean 3, spread 1). The learned bias moves by close to log w; sampling noise makes it not exact. Illustrative numbers.
Any change of base rate between training and use shifts the logit by the change in log prior odds. If the head saw base rate πtrain (after any weights or re-sampling) and the world has πuse, correct it like this Saerens 2002:
Worked: trained balanced (odds 1), used at 1 in 1,000 (odds 1/999): add log(1/999) = −6.91 to every logit.
Chapter 4
Derive the focal loss and its gradient, see how γ silences easy negatives, and find what it does to the probabilities
Class weights treat every negative the same. But Chapter 1 showed the real crowd: easy negatives, each pushing a little. Lin, Goyal, Girshick, He and Dollár met this in dense object detection, where an image yields about 100,000 candidate boxes and almost all are background Focal Loss · §4. Their fix scales each example's loss by how wrong it still is.
Write pt for the probability the head gives the true class: p for an event, 1 − p for a non-event. Cross-entropy is −log pt. The focal loss multiplies it by a modulating factor:
Worked, γ = 2: pt = 0.9 → factor 0.12 = 0.01, so 100× less loss than cross-entropy. pt ≈ 0.968 → about 1,000× less. pt ≤ 0.5 → at most 4× less Focal Loss · §3.2.
Take an event, so pt = p = σ(u). The sigmoid's derivative is dp/du = p(1 − p). Differentiate the product (1 − p)γ·(−log p) in p, then multiply by p(1 − p):
Worked, γ = 2, an event at p = 0.9: cross-entropy's gradient is −0.1. Focal: 0.01 × (2 × 0.9 × ln 0.9 − 0.1) = 0.01 × (−0.190 − 0.1) = −0.0029, about 34× smaller.
Slide γ. The dashed curve is cross-entropy. Switch views to see the gradient, and what a head trained with focal loss would predict at each true probability.
Computed exactly. The share of the gradient from easy negatives uses the 1-D world of Chapter 1 at 1 : 199 and its best head. “What it predicts” minimises the expected focal loss at each true probability q, numerically.
Focal loss changes the answer the head converges to. For a true q, the head minimises q·FL(p) + (1 − q)·FL(1 − p). Cross-entropy's minimum is p = q. Focal's is not. Because confident answers earn almost no reward, the minimum is pulled toward 0.5.
Computed, γ = 2: true q = 0.9 → focal head predicts 0.69. q = 0.99 → 0.85. q = 0.1 → 0.31. The ranking is kept; the scale is squashed.
The squash is close to a constant scale on the logit. So one temperature, fitted afterwards, mostly removes it (Chapter 8). In the hero, focal loss gives the same ranking as cross-entropy but underconfident probabilities; Calibrate fits a temperature below 1 and fixes them.
Chapter 5
Move the prior term on purpose, after training or inside the loss, for a long tail of many classes
Many real problems have not two classes but hundreds. A bird classifier sees thousands of robins and a handful of rare warblers. This shape is the long tail. Overall accuracy rewards the common head classes. What you often want instead is the balanced error: the average of each class's error rate, so every class counts the same.
Chapter 2 generalises to many classes. A softmax head with logits fy(x) learns the evidence plus log πy, the log prior of class y. Menon et al. showed that the best rule for the balanced error is to remove that prior: predict the class with the largest fy(x) − log πy Menon 2021 · §3, eq. 8. They give two ways to do it.
Worked, three classes with 5,000, 500 and 50 examples: log π = (−0.10, −2.41, −4.71). Plain logits (2.0, 0.5, 0.0) pick class 1. Adjusted: (2.10, 2.91, 4.71) pick the rare class 3.
Why does adding the prior during training remove it from f? Because the loss already supplies the prior term. The head does not need to learn it, so fy keeps only the evidence. This is the binary trick of Chapter 3 in reverse: there you saw weights push the prior in; here you add it outside the head so the head can leave it out.
Balanced softmax, by Ren et al., reaches the same place from another direction. It replaces the softmax in training by njeηj / Σi nieηi, where nj is the count of class j Balanced Softmax · eq. 3, 4. Since nj is proportional to πj, this is the logit-adjusted loss with τ = 1.
Head class (blue), middle class (yellow), tail class (warm). Shaded regions show what the head predicts. Pick a method, set the imbalance and press Train. Bars give each class's recall on a balanced test set.
Trained here: a linear softmax head on two features, 600 head-class points, 600/√ratio middle-class points, 600/ratio tail-class points (at least 6), 300 Adam steps. Recall is measured on 500 new points per class. Post-hoc τ acts live; the adjusted loss uses τ when you train. Illustrative numbers.
| Balanced error (%), lower is better | CIFAR-10-LT | ImageNet-LT | iNaturalist |
|---|---|---|---|
| Plain cross-entropy (ERM) | 27.16 | 53.11 | 38.66 |
| Post-hoc adjustment, τ = 1 | 22.60 | 49.66 | 33.98 |
| Logit-adjusted loss, τ = 1 | 22.33 | 48.89 | 33.64 |
Source: Menon 2021 · Table 3, averages of 5 runs; the CIFAR sets are made long-tailed with a 100 : 1 ratio between the largest and smallest class Menon 2021 · §6.2. Tuning τ post-hoc reached 19.08 on CIFAR-10-LT.
Chapter 6
Choose the threshold from the cost of a miss and a false alarm, give each class its own, and read the PR curve instead of the ROC curve
So far, every fix moved the logit's constant term. That is the same as moving the threshold. A head that says p, thresholded at t, makes the same decisions as a head that says logit(p) − logit(t), thresholded at 0.5. So a cleaner way to think is: train for good scores, then choose the threshold on purpose.
Suppose a missed glass break costs cmiss and a false alarm costs cFA. For a clip with a calibrated probability q, alarming costs cFA(1 − q) on average. Staying silent costs cmiss·q. Alarm when silence costs more:
Worked: a miss costs 50 times a false alarm → t* = 1/51 = 0.0196. Equal costs → t* = 0.5, the accuracy threshold.
Notice what this needs: a calibrated q. If the head's probabilities are inflated by a class weight, the cost rule gives the wrong threshold. This is the link between the two halves of the lesson.
The ROC curve plots recall (the true-positive rate) against the false-positive rate, FP divided by all negatives. The PR curve plots precision against recall. The difference is the denominator. ROC divides false alarms by the huge number of negatives, so thousands of false alarms look like a small rate.
Worked, 10,000 clips at 1 : 199: 50 events, 9,950 non-events. Recall 0.9 → 45 caught. False-positive rate 0.05 → 497.5 false alarms. The ROC point (0.05, 0.90) looks excellent. Precision = 45 / 542.5 = 8.3%.
One fixed, calibrated head. Slide the imbalance: the ROC curve does not move; the PR curve sinks. Slide the cost ratio: the teal point moves to the cost-optimal threshold. The hollow point is the 0.5 threshold.
Exact curves for the 1-D world with separation 2 and a calibrated head. Counts are per 10,000 clips. Illustrative numbers.
Saito and Rehmsmeier made this argument with simulations and a review of published work: on imbalanced data, the PR plot is more informative than the ROC plot Saito 2015. Summarise a PR curve by its area, the average precision (AP). A useless head scores AP equal to the base rate, 0.005 here, not 0.5 as with ROC.
With many classes, a single global threshold assumes every class has the same costs, base rate and calibration. They rarely do. Choose each class's threshold on validation data: either from its costs with the rule above, or to reach a target, such as “recall at least 90% for the alarm class”.
Chapter 7
Define calibration, draw a reliability diagram, compute ECE by hand, and see why it is noisy on small sets and why modern nets fail it
A weather forecaster says “30% chance of rain” on many days. If it rains on about 30% of those days, the forecaster is calibrated. The same holds for a head: among all clips where it says 0.3, about 30% should hold the event.
Calibration and ranking are different. A head can rank perfectly and be badly calibrated: multiply every logit by 3 and the order does not change, but every probability moves. A head can also be calibrated and useless: say the base rate for every clip.
To check, group clips into M bins by score, say [0, 0.1), [0.1, 0.2) and so on. In each bin, compare the average score (the confidence) with the share of real events (the accuracy of the bin). Plot one against the other: that is a reliability diagram. A calibrated head lies on the diagonal. The expected calibration error averages the gaps, weighted by bin size Guo 2017 · §2, eq. 3:
Worked, 200 clips in 3 bins: 120 clips with confidence 0.10, real rate 0.05; 50 with 0.50, real 0.40; 30 with 0.85, real 0.70. ECE = 0.6 × 0.05 + 0.25 × 0.10 + 0.15 × 0.15 = 0.0775, or 7.75%.
Bars are the real event rate in each bin, measured on a sampled validation set; the red parts are the gaps from the diagonal. Make the model overconfident. Then shrink the set and press Resample: even a perfect model shows gaps.
Base rate 20%, the 1-D world with separation 2. The model's logit is the true logit times the slider. True ECE is computed on 400,000 points; the noise floor is the average ECE of a perfectly calibrated model over 20 validation sets of the chosen size. Illustrative numbers.
Guo, Pleiss, Sun and Weinberger compared a 5-layer LeNet with a 110-layer ResNet on CIFAR-100. The ResNet was more accurate (error 30.6% against 44.9%), yet its average confidence sat well above its accuracy, while LeNet's matched Guo 2017 · Fig. 1. Its ECE was 16.53%, against 4.85% for LeNet Guo 2017 · Table 1.
They traced it to depth, width, Batch Normalization and less weight decay Guo 2017 · §3, Fig. 2. The mechanism is in the loss. Once a network classifies almost every training example correctly, it can still lower its training NLL (the cross-entropy) by raising confidence. So test NLL overfits while test error keeps improving Guo 2017 · §3, Fig. 3.
Chapter 8
Fit a small map from scores to probabilities on held-out data, derive why temperature scaling is one easy convex problem, and know when it is not enough
You cannot calibrate on the training set: the head has already overfit its confidence there. Keep a validation set the head never trained on. Freeze the head. Learn a tiny map from its logit to a better probability.
The simplest map has one number. Divide every logit by a temperature T > 0 Guo 2017 · §4.2, eq. 9. T > 1 softens overconfident scores; T < 1 sharpens underconfident ones. The order of the logits is kept, so the decisions at the argmax and the PR curve do not change.
Worked: a logit of 4 gives 0.982. With T = 2 it becomes σ(2) = 0.881. The optimal T makes the residuals p − y, weighted by the logits, sum to zero.
Guo et al. found this one-number fix often the best of all the methods they tried. On CIFAR-100, the ResNet-110's ECE fell from 16.53% to 1.26%; on ImageNet, a ResNet-152's fell from 5.48% to 1.86% Guo 2017 · Table 1. A conjugate-gradient solver finds T in about 10 iterations Guo 2017 · §5.
Platt scaling fits two numbers: p = σ(a·z + b) Platt 1999. The a is a temperature; the b is a shift. You need the shift when the miscalibration includes a prior change, such as a class weight. Temperature alone cannot move the constant term.
Isotonic regression fits any non-decreasing step function from score to probability Zadrozny 2002. The pool-adjacent-violators algorithm finds it: sort by score, and merge neighbouring groups whose event rates go down. It can fix any monotone distortion, but it has many free steps, so it needs more validation data.
Pick what went wrong with the head and a method, then press Fit. Left: test reliability before (green) and after (purple). Right: the fit itself. Try a small validation set with isotonic.
Base rate 10%, the 1-D world with separation 2. Fitted by Newton's method (temperature, Platt) or pool-adjacent-violators (isotonic) on sampled validation labels; tested against the true probability on 200,000 points. Illustrative numbers.
In Guo et al.'s tests, isotonic regression was less accurate than temperature scaling on most vision sets, for example 4.99% against 1.26% ECE for the CIFAR-100 ResNet-110 Guo 2017 · Table 1. Its flexibility costs data.
Chapter 9 How
The same four tools three ways: by hand, from scratch in PyTorch, and as library lines
The data flow is short. A frozen backbone gives a feature per clip (B × D). The head gives logits (B × 1 for one event, or B × C for C classes). The loss sees the logits and the labels (B). Every tool in this lesson acts on the logits, the loss, or the map after the head. None of them touches the backbone.
Before any code, compute one example by hand and check that the code agrees. Use the calculator: it runs the formulas of Chapters 1 to 8 for one clip.
Set the head's logit for one clip and its label. The rows show each loss and its gradient with respect to the logit, and the probability after the prior fix and the temperature.
Exact formulas. The logit-adjusted loss adds log(π/(1 − π)) to z inside the loss, as in the binary form of Chapter 5. Set z = 2.20, γ = 2 and an event to match the worked example of Chapter 4.
Use binary_cross_entropy_with_logits for −log pt: it is stable for large logits. Then multiply by the factor. Autograd derives the gradient of Chapter 4 for you; the printout checks it.
import torch import torch.nn.functional as F def focal_loss(logits, y, gamma=2.0, alpha=None): # logits: (B,) raw scores; y: (B,) floats, 1 = event, 0 = no event ce = F.binary_cross_entropy_with_logits(logits, y, reduction='none') # -log p_t p = torch.sigmoid(logits) p_t = y * p + (1 - y) * (1 - p) # probability given to the true class loss = (1 - p_t) ** gamma * ce # easy examples (p_t near 1) fade out if alpha is not None: # optional class weight, as in Lin et al. loss = (y * alpha + (1 - y) * (1 - alpha)) * loss return loss.mean() logits = torch.tensor([2.1972, 2.1972, 0.0], requires_grad=True) # p = 0.9, 0.9, 0.5 y = torch.tensor([1.0, 0.0, 1.0]) # an easy positive, a hard negative, a coin flip print(focal_loss(logits, y, gamma=0.0).item()) # 1.0337: plain cross-entropy print(focal_loss(logits, y, gamma=2.0).item()) # 0.6798 focal_loss(logits, y, gamma=2.0).backward() print(logits.grad * 3) # dL/dlogit per example: [-0.0029, 1.1020, -0.2983]
The printed gradients match the hand formula: −0.0029 for the easy event (cross-entropy gives −0.1) and 1.102 for the hard non-event, which is more than cross-entropy's 0.9. The library version is one line, torchvision.ops.sigmoid_focal_loss(logits, y, alpha=-1, gamma=2, reduction='mean'); it gives 0.6798 on these inputs. Its default α is 0.25, as in the paper, so pass alpha=-1 to turn the weight off.
import torch import torch.nn.functional as F counts = torch.tensor([5000.0, 500.0, 50.0]) # training examples per class log_prior = torch.log(counts / counts.sum()) # (C,) = log pi_y def la_loss(logits, y, tau=1.0): # logits: (B, C) from the head; y: (B,) class ids. # Add tau * log pi during training only. With tau = 1 this is Balanced Softmax. return F.cross_entropy(logits + tau * log_prior, y) def la_posthoc(logits, tau=1.0): # For a head trained with plain cross-entropy: subtract tau * log pi at test time. return logits - tau * log_prior print(log_prior) # tensor([-0.1044, -2.4069, -4.7095]) f = torch.tensor([[2.0, 0.5, 0.0]]) # plain-CE logits for one input print(f.argmax(1).item(), la_posthoc(f).argmax(1).item()) # 0 before, 2 after print(la_loss(f, torch.tensor([2])).item()) # 6.6286 if the true class is the rare one
Why each choice. The prior comes from training counts, so it is fixed before training. The adjustment is added outside the head, so the head's weights are a plain linear layer. Use la_loss or la_posthoc, never both (Chapter 5).
Fit on validation logits from the frozen head. Optimise log T rather than T, so T stays positive. L-BFGS solves this one-number convex problem in a few steps.
import torch import torch.nn.functional as F torch.manual_seed(0) def fit_temperature(val_logits, val_y): # val_logits: (N, C) from the frozen, trained model; val_y: (N,) class ids log_t = torch.zeros(1, requires_grad=True) # T = exp(log_t) stays positive opt = torch.optim.LBFGS([log_t], lr=0.1, max_iter=100) def closure(): opt.zero_grad() loss = F.cross_entropy(val_logits / log_t.exp(), val_y) # NLL at this T loss.backward() return loss opt.step(closure) return log_t.exp().item() true_logits = torch.randn(5000, 10) * 1.5 # a world where these are honest val_y = torch.distributions.Categorical(logits=true_logits).sample() model_logits = 2.5 * true_logits # an overconfident model T = fit_temperature(model_logits, val_y) print(round(T, 2)) # 2.5: it finds the overconfidence probs = F.softmax(model_logits / T, dim=1) # calibrated; argmax unchanged
For a classic model, CalibratedClassifierCV wraps the model, trains it on folds, and fits a sigmoid (Platt) or isotonic map on the held-out fold. Here it repairs a class-weighted logistic regression. Undoing the weight by subtracting log w is the exact fix in the limit; on finite data it is close but not perfect.
import numpy as np from sklearn.linear_model import LogisticRegression from sklearn.calibration import CalibratedClassifierCV rng = np.random.default_rng(0) X = np.r_[rng.normal(0, 1, (19900, 2)), rng.normal(2.5, 1, (100, 2))] # 1 event in 200 y = np.r_[np.zeros(19900), np.ones(100)] x0 = np.array([[1.5, 1.5]]) # true P(event | x0) = 0.017 raw = LogisticRegression(class_weight='balanced').fit(X, y) # weight ratio w = 199 cal = CalibratedClassifierCV(LogisticRegression(class_weight='balanced'), method='sigmoid', cv=5).fit(X, y) # refits a sigmoid on held-out folds p_raw = raw.predict_proba(x0)[0, 1] def undo_weight(p, w): # exact fix for a positive-class weight w: subtract log w from the logit z = np.log(p / (1 - p)) - np.log(w) return 1 / (1 + np.exp(-z)) print(round(p_raw, 3), round(undo_weight(p_raw, 199), 3), round(cal.predict_proba(x0)[0, 1], 3)) # 0.596 0.007 0.023 (truth: 0.017)
Chapter 10 Did it work?
The report card for an imbalanced head: the PR curve and AP, reliability and ECE before and after, recall per class, and the sanity checks that catch bugs
A single number cannot tell you whether an imbalanced head works. You need a small set of plots, each answering one question.
| Question | What to show | What good looks like |
|---|---|---|
| Does it rank events above non-events? | PR curve and AP, with the base rate marked | AP well above the base rate; stable across seeds. |
| Is the threshold right? | Precision and recall at the chosen threshold, from validation | Meets the target recall or minimises cost on validation; similar on test. |
| Do the probabilities mean what they say? | Reliability diagram and ECE before and after calibration, plus NLL | Points near the diagonal; ECE above its noise floor only by a little. |
| Does every class work? | Recall per class, balanced error | No class near 0 recall; the tail is not sacrificed for the head. |
| Is the signal real? | Shuffled-label run | AP falls to the base rate. If not, something leaks. |
This reads the head you last trained in the hero at the top. Pick a panel. Press Shuffle labels to retrain the same head on scrambled labels.
Uses the hero's head, threshold and test set. “After” is the hero's calibration fitted on its validation set; if you have not pressed Calibrate, it is fitted here the same way. Illustrative numbers.
The same report in a few lines of numpy and sklearn. The ECE function bins the event probability with equal-width bins, as Guo et al. do. The data are the 1-D world at 1 : 199 with the true probability as the score, plus an overconfident copy.
import numpy as np from sklearn.metrics import average_precision_score, precision_recall_curve, recall_score def ece(p, y, bins=15): # p: (N,) predicted P(event); y: (N,) 0/1. Equal-width bins, as in Guo et al. edges = np.linspace(0, 1, bins + 1) idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1) total = 0.0 for b in range(bins): m = idx == b if m.any(): total += m.mean() * abs(y[m].mean() - p[m].mean()) # |B|/n * |acc - conf| return total rng = np.random.default_rng(1) x = np.r_[rng.normal(0, 1, 19900), rng.normal(3, 1, 100)] # scores of 20,000 clips y = np.r_[np.zeros(19900), np.ones(100)] q = 1 / (1 + np.exp(-(3 * x - 4.5 + np.log(1 / 199)))) # the true P(event | x) p_over = 1 / (1 + np.exp(-2.5 * np.log(q / (1 - q)))) # an overconfident model print('AP', round(average_precision_score(y, q), 3), 'base rate', y.mean()) # AP 0.6, base rate 0.005 prec, rec, thr = precision_recall_curve(y, q) # the full PR curve print('ECE honest', round(ece(q, y), 4), 'overconfident', round(ece(p_over, y), 4)) # 0.0009, 0.003 print('per-class recall at 0.5', recall_score(y, q > 0.5, average=None).round(3)) # [0.999 0.38]
Look at the ECE numbers: 0.09% for the honest score and 0.3% for one that is 2.5 times too confident. Both look tiny, because 99% of clips sit near 0 and are right. That is the “which ECE?” trap the report card shows.
Chapter 11 What else What not
The same three steps in six fields, the cases where each tool hurts, and the whole block on one screen
The recipe never changes: train for a good ranking, put the prior where you want it, choose the threshold from costs, then calibrate on held-out data. What changes between fields is the base rate, the cost ratio, and whether anyone reads the probability.
| Field | The rare class | What changes |
|---|---|---|
| Sound event detection | A glass break or gunshot in hours of home audio | Report false alarms per hour; a threshold per event class; see event detection heads. |
| Medical screening | A rare disease in a screening population | A miss costs far more than a recall visit; the clinic's base rate differs from the training hospital's, so apply prior correction; doctors read the probability, so calibrate. |
| Fraud detection | A fraudulent payment among thousands | The threshold is often set by review capacity (the top k per day) rather than a cost; the base rate drifts, so recalibrate often. |
| Defect detection | A cracked part on a production line | Very few defect images: undersample good parts, weight defects, and pick the threshold for a target recall such as 99%. |
| Long-tail species | Rare birds or frogs in bioacoustic recordings, rare plants in photos | Hundreds of classes: logit adjustment or balanced softmax, per-class thresholds, and balanced error; see Perch 2. |
| LLM classifier confidence | A harmful prompt, or a question the model gets wrong | Fit a temperature on the classifier's logits; check the reliability of the rare “flag” class; see LLM calibration. |
| Tool | Avoid it when | Use instead |
|---|---|---|
| Class weights or re-sampling | You need calibrated probabilities and will not correct them | Plain CE plus a chosen threshold; or weights, then subtract log w or refit Platt. |
| Focal loss | Data is balanced, or easy examples are not the bulk of the loss | Plain cross-entropy with a prior-initialised bias. |
| One global threshold | Classes differ in base rate, cost or calibration | Per-class thresholds chosen on validation. |
| ECE as the only check | The validation set is tiny or almost all scores are near 0 | Reliability with confidence bands, NLL, Brier score, ECE on flagged clips. |
| Temperature scaling | The base rate shifted or a class weight was used | Prior correction first, or Platt scaling. |
| Isotonic regression | You have few validation events | Temperature or Platt scaling. |
| Logit adjustment | You care about accuracy at the training frequencies | Plain cross-entropy: it already targets that. |
Choose an answer in each row. The pipeline below updates.
A rule of thumb from this lesson, not a law. Always compare against plain cross-entropy with a tuned threshold.
| Part | In one line |
|---|---|
| Shape signature | (B, D) features → head → (B, C) logits → [+ τ log π in the loss] → threshold per class → [÷ T, ± prior] → (B, C) probabilities. |
| The key fact | logit = evidence + log prior odds. Every imbalance fix moves the second term. |
| Weights | pos_weight w adds log w to the logit; undo by subtracting it. |
| Focal | −(1 − pt)γ log pt, γ = 2; silences easy examples; pulls probabilities toward 0.5. |
| Logit adjustment | Train on f + τ log π (or predict f − τ log π); targets the balanced error. Balanced softmax = τ = 1. |
| Threshold | t* = cFA / (cFA + cmiss) on calibrated q; per class; chosen on validation. |
| Calibration | Temperature (1 number, convex), Platt (adds a shift), isotonic (needs data); fit on held-out data. |
| Did it work? | PR curve and AP vs base rate; reliability and ECE before/after with a noise floor; recall per class; shuffled labels. |
| Knobs | w, γ, α, τ, the threshold per class, T (or a, b), bins M. |
This lesson is part of the Builder's Toolkit series on reusable blocks for heads on frozen backbones. Go on to:
Now press Present or Teach and explain, out loud and from memory, why the prior is an additive term in the logit and what each fix does to it. Then go back to the hero, train with class weights at 1 : 500, and predict where the reliability curve will sit before you press Calibrate.