Focal Loss: the loss that looks at the hard examples

Why we need it

An object detector such as RetinaNet checks about a hundred thousand “anchors” per image — boxes, each of which may contain an object. There are only a handful of objects in a frame; everything else is background. Most background anchors are trivial: sky, a wall, asphalt. The model quickly learns to call them background with confidence, but cross-entropy still charges a small loss for each one. A hundred thousand small losses add up to a large one — and that sum drowns out the signal from the few hard examples the whole thing was for.

Lin, Goyal, Girshick, He and Dollár (2017) proposed not to discard easy examples and not to resample the data, but to change the loss itself: multiply cross-entropy by $(1-p_t)^\gamma$. For a confidently recognised example the factor is almost zero, for a hard one almost one. Where it is used:

• one-stage detectors — RetinaNet, and after it FCOS, ATSS and other dense detectors;
• segmentation — especially medical, where the region of interest covers a fraction of a percent of the pixels (often paired with Dice);
• tabular problems with a rare class — fraud, churn, anomalies; in boosting, focal loss is set as a custom objective;
• calibrating deep networks — focal loss is sometimes used against the overconfidence of large networks trained with CE (Mukhoti et al., 2020).

Intuition: a volume knob for every example

Picture a class where a teacher answers students’ questions. A thousand students ask questions they already know the answer to, and twenty ask really hard ones. If every question gets the same time, none is left for the hard ones. Focal loss is a volume knob: the more confidently the model already answers an example, the quieter its voice in the total.

Drag the point and change $\gamma$. The grey dashed curve is cross-entropy, $-\ln p_t$. The blue one is focal loss: in the right half, where examples are already classified correctly, it hugs zero much earlier.

The formula, bone by bone

$p=\sigma(z)$

the probability of class 1 that the model obtained from the logit $z$ (via the sigmoid; in the multi-class case via softmax);

$p_t$

the probability of the correct class: how right the model is on this example; $p_t>0.5$ means it is classified correctly;

$-\log p_t$

ordinary cross-entropy: zero at $p_t=1$, infinity as $p_t\to0$;

$(1-p_t)^\gamma$

the modulating factor: “how wrong the model still is”, raised to the power $\gamma$;

$\gamma\ge0$

the focusing parameter: $\gamma=0$ is CE, and the larger it is, the more easy examples are damped; RetinaNet uses $\gamma=2$;

$\alpha_t$

the class weight: $\alpha$ for positives and $1-\alpha$ for negatives; RetinaNet uses $\alpha=0.25$.

Writing it through $p_t$ is the key trick: instead of the two branches of binary cross-entropy $-y\log p-(1-y)\log(1-p)$ there is one, $-\log p_t$. In terms of the logit it is handy to set $z_t=z$ for $y=1$ and $z_t=-z$ for $y=0$; then $p_t=\sigma(z_t)$.

The last column says how many times focal loss is smaller than cross-entropy; it is exactly $1/(1-p_t)^\gamma$. The example from the cheat sheet: with $\gamma=2$ the easy example $p_t=0.9$ gives $0.00105$ versus $0.105$ (100 times less), the hard one $p_t=0.3$ gives $0.590$ versus $1.204$ (about half).

On an ordinary scale the values $0.001$ and $0.000001$ are indistinguishable, so it helps to look at the losses on a logarithmic scale, where the factor turns into a vertical shift.

Properties

• Generalises cross-entropy: $\gamma=0$, $\alpha_t=1$ give $\mathrm{CE}=-\log p_t$.
• Never exceeds CE: $0\le\mathrm{FL}\le\mathrm{CE}$, since $0\le(1-p_t)^\gamma\le1$ for $\gamma\ge0$.
• Decreasing in $p_t$: both factors $(1-p_t)^\gamma$ and $-\log p_t$ are non-negative and decreasing, so the more confident the correct answer, the smaller the loss.
• Behaviour at the edge: as $p_t\to1$, $\mathrm{FL}\approx(1-p_t)^{\gamma+1}$, because $-\log p_t\approx1-p_t$.
• Not convex in the logit for $\gamma>0$: for very hard examples (with $\gamma=2$, for $p_t<0.058$, $z_t<-2.79$) the second derivative with respect to $z_t$ is negative. Cross-entropy is convex in the logit everywhere.
• Not a strictly proper scoring rule: the minimum of the expected FL is not attained at the true probability. Hence probabilities learned with FL need calibration (see the section on $\alpha$).

The derivative: the gradient with respect to the logit

The model learns through the gradient with respect to the logit $z$, so that gradient shows whom training “hears”. For cross-entropy it is simple: $\partial\,\mathrm{CE}/\partial z_t=-(1-p_t)$ — the bigger the error, the stronger the gradient, monotonically. Focal loss adds a second term.

Inside the brackets is the CE gradient, $1-p_t$, plus a non-negative extra term $\gamma p_t|\log p_t|$. Outside is the same factor $(1-p_t)^\gamma$. For easy examples the factor wins, while for very hard ones the extra term can make the gradient even larger than for CE.

The surface FL(p_t, γ) in 3D

Let us gather the whole family of curves into one surface over the $(p_t,\gamma)$ plane. The slice at $\gamma=0$ is cross-entropy; moving along $\gamma$, the surface “sinks”, and the larger $p_t$, the faster.

Why a straight line: $\log\mathrm{FL}=\log\mathrm{CE}+\gamma\log(1-p_t)$. The loss decays exponentially in $\gamma$, at the rate $|\ln(1-p_t)|$: $0.69$ per unit of $\gamma$ for $p_t=0.5$, $2.30$ for $p_t=0.9$, $4.61$ for $p_t=0.99$. The parameter $\gamma$ spreads easy and hard examples apart on a scale of orders of magnitude.

Imbalance: where the loss comes from

Let us model a one-stage detector. There are 1000 negatives per positive. The negatives are mostly easy: the logit of the correct class is $z_t\sim N(4,\,1)$, so a typical $p_t=\sigma(4)\approx0.98$. The positives are hard: $z_t\sim N(0,\,1)$, a typical $p_t=0.5$. Let us compute what share of the total loss comes from each group.

This is exactly the “focusing”. Gradient descent minimises the sum, and under CE that sum consists almost entirely of examples that are already solved. Focal loss redistributes the “attention budget”: the losses of easy examples are damped tens and hundreds of times, and the sum becomes dominated by hard negatives and positives — the ones on which the model can actually improve.

α-balancing and calibration

The factor $\alpha_t$ is ordinary class weighting. Intuition suggests giving the rare class more weight: $\alpha>0.5$. Yet in RetinaNet the best value turned out to be $\alpha=0.25$ — positives weigh less than negatives. The reason is that $\gamma$ has already done the job: easy negatives are damped, and now the balance tilts towards the positives. $\alpha$ and $\gamma$ have to be tuned together; the authors recommend $\gamma=2$, $\alpha=0.25$; in their experiments the gain over CE held for every $\gamma$ from $0.5$ to $5$, and the best $\alpha$ decreased slightly as $\gamma$ grew.

There is also a subtlety that people forget. Take the simplest “model” — a single number $p$, the same for every example. If the share of positives is $\pi$, cross-entropy is minimised exactly at $p=\pi$: CE is a proper scoring rule, it teaches honest probabilities. With focal loss this is no longer true.

The conclusion: the factor $(1-p_t)^\gamma$ pulls predictions towards the middle — it “dislikes” confidence, because confidence stops being rewarded. So the output of a model trained with focal loss cannot be read as a probability: for thresholds and ranking that is fine, but if you need probabilities, calibrate them on validation data (Platt scaling, isotonic regression, temperature scaling). Without $\alpha$ the optimal FL prediction is an increasing function of the true probability that equals $0.5$ at $0.5$ (Charoenphakdee et al., 2021): the order and the decision at threshold $0.5$ are preserved, only the values are distorted. And for large networks that become overconfident under CE, this pull towards the middle sometimes even improves calibration (Mukhoti et al., 2020) — but that is an empirical observation, not a guarantee.

Focal loss in training

Let us train logistic regression on two-dimensional data with a 1 : 20 imbalance (400 negatives, 20 positives) in two ways — with CE and with FL — from the same start (the bias initialised to the share of positives, as in RetinaNet) and with the same step. The size of each dot is proportional to its gradient $|\partial L/\partial z|$: this shows which examples training “hears”.

What the figure shows. With CE large dots are scattered over the whole cloud of negatives: each of the 400 easy examples adds its own small portion of gradient. With FL only the dots near the boundary and the positives stay large — the rest almost vanish. The boundary itself comes out similar for the two models: for a linear model on such data the optimal direction is almost the same. The difference lies elsewhere: FL reaches its boundary in a few dozen steps and stops moving it, while CE keeps growing the weights, making easy examples even more confident. In a deep network with millions of parameters this difference is decisive: gradient spent on “obvious” background is network capacity not spent on hard cases.

In code

Compute focal loss from logits, not from probabilities: $\log p_t=\log\sigma(z_t)=-\operatorname{softplus}(-z_t)$ — no $\log0$ and no overflow. Below is a NumPy version with the gradient and a check on the cheat-sheet example.

pythonimport numpy as np

def log_sigmoid(z):
    return -np.logaddexp(0.0, -z)          # log σ(z) without overflow

def focal_loss(z, y, gamma=2.0, alpha=None):
    """Binary focal loss from logits z and labels y ∈ {0, 1}, element-wise."""
    zt = np.where(y == 1, z, -z)           # logit of the correct class: p_t = σ(z_t)
    log_pt = log_sigmoid(zt)
    pt = np.exp(log_pt)
    at = 1.0 if alpha is None else np.where(y == 1, alpha, 1 - alpha)
    return -at * (1 - pt) ** gamma * log_pt

def focal_grad(z, y, gamma=2.0, alpha=None):
    """dFL/dz = ±α_t (1 − p_t)^γ (γ p_t log p_t + p_t − 1)."""
    zt = np.where(y == 1, z, -z)
    log_pt = log_sigmoid(zt)
    pt = np.exp(log_pt)
    at = 1.0 if alpha is None else np.where(y == 1, alpha, 1 - alpha)
    g = at * (1 - pt) ** gamma * (gamma * pt * log_pt + pt - 1)
    return np.where(y == 1, g, -g)

z = np.array([np.log(9.0), np.log(3 / 7), -np.log(9.0)])  # p_t = 0.9, 0.3, 0.9
y = np.array([1, 1, 0])
print(focal_loss(z, y, gamma=0.0).round(4))   # [0.1054 1.204  0.1054]  — this is CE
print(focal_loss(z, y).round(5))              # [0.00105 0.58995 0.00105]
print(focal_grad(z, y, gamma=0.0).round(4))   # [-0.1 -0.7  0.1]
print(focal_grad(z, y).round(4))              # [-0.0029 -0.697   0.0029]

In PyTorch the binary version ships with torchvision; the multi-class one is easy to write with log_softmax. The last line initialises the bias for a rare class.

pythonimport math
import torch
import torch.nn as nn
from torchvision.ops import sigmoid_focal_loss

# Binary (per anchor and class): α = 0.25, γ = 2, as in RetinaNet
logits = torch.tensor([math.log(9.0), math.log(3 / 7)])
targets = torch.tensor([1.0, 1.0])
loss = sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2.0, reduction="none")
# ≈ [0.00026, 0.14749] = 0.25 · [0.00105, 0.58995]

# Multi-class version via log_softmax
def focal_loss_multiclass(logits, target, gamma=2.0):
    logp = torch.log_softmax(logits, dim=-1)
    logp_t = logp.gather(-1, target[:, None]).squeeze(-1)
    return (-(1 - logp_t.exp()) ** gamma * logp_t).mean()

# Initialise the last layer’s bias for a rare class (π = 0.01)
cls_head = nn.Conv2d(256, 9 * 80, kernel_size=3, padding=1)   # A = 9 anchors × K = 80 classes
nn.init.constant_(cls_head.bias, -math.log((1 - 0.01) / 0.01))   # ≈ −4.595

Common mistakes

1. $\alpha$ “by frequency” together with $\gamma$. Setting $\alpha=0.99$ for a class with a 1% share at $\gamma=2$ compensates twice. Start from $\gamma=2$, $\alpha=0.25$ and tune them together.
2. No initialisation for the rare class. With a zero bias the first iterations drown in background losses; in the RetinaNet paper, training with plain CE without the $\pi=0.01$ initialisation diverged.
3. Normalising by the number of all examples. Averaging over 100,000 anchors makes the loss tiny and dependent on image size; in detection it is divided by the number of positives.
4. Focal loss on probabilities with a clamp. $\log(\max(p,\varepsilon))$ cuts off the gradient of the logarithm for the hardest examples ($p<\varepsilon$) — exactly where it matters most; and $p=\sigma(z)$ also saturates in float32. Compute from logits.
5. Reading the output as a probability. FL shifts predictions towards the middle (Figure 6); for risk estimates and pricing, probabilities have to be calibrated.
6. A large $\gamma$ with noisy labels. Wrong labels look like very hard examples, and FL increases their relative weight. If you suspect noise, lower $\gamma$ or clean the labels.

Links to other formulas

• BCE and cross-entropy: FL with $\gamma=0$, $\alpha_t=1$ coincides with them.
• The sigmoid: $p_t=\sigma(z_t)$, and its derivative $p_t(1-p_t)$ enters the FL gradient.
• Softmax: in the multi-class case $p_t$ is the softmax component of the correct class.
• Huber is the opposite philosophy: Huber weakens the contribution of the largest errors, treating them as outliers, while FL weakens the smallest ones so that the large ones stand out.
• Triplet loss solves a related problem in metric learning: easy triplets with zero loss give no gradient, and training focuses on the hard ones (hard negative mining).

Summary

Check yourself