Categorical cross-entropy

Why we need it

A model has output class probabilities — say $(0.665,\,0.245,\,0.090)$ for “cat”, “dog”, “bird” — and the picture shows a cat. How good is that? We need one number that gets smaller as the probability of the correct answer grows, and that gradient descent can minimize. The standard answer is categorical cross-entropy (CE): here $\mathrm{CE}=-\ln0.665\approx0.408$.

Where it appears:

• training multi-class classifiers — from logistic regression to ResNet and ViT: the last layer outputs logits, softmax turns them into probabilities, CE scores the answer;
• language models — pre-training GPT, LLaMA, Qwen means minimizing the CE of the next token: the classes are vocabulary tokens, and $e^{\mathrm{CE}}$ is the famous perplexity;
• knowledge distillation — the student minimizes CE against the teacher’s “soft” probabilities;
• statistics — minimizing CE is equivalent to maximum likelihood for the categorical distribution;
• information theory — CE is the mean code length when data come from a distribution $p$ but are encoded according to a model $q$.

Intuition: a penalty for surprise

Suppose the model announces its probabilities in advance and then learns the correct answer. If it gave the correct class probability $p$, its “surprise” is $-\ln p$: at $p=1$ there is no surprise, at $p=0.5$ it is $\ln2\approx0.693$, and as $p\to0$ it grows to infinity. Cross-entropy is the model’s mean surprise on the data.

Why a logarithm? First, it turns the product of probabilities of independent examples into a sum. Second, it rewards honesty: among penalties that depend only on the probability the model assigned to the outcome that happened, with three or more outcomes only the logarithmic one (up to a positive factor and a constant) is minimized on average when the model reports its true probabilities.

The formula, bone by bone

$y=(y_1,\dots,y_K)$

the true distribution; usually one-hot: $y_c=1$ for the correct class $c$ and $0$ for the rest;

$\hat y_k$

the predicted probability of class $k$, usually $\hat y=s=\operatorname{softmax}(z)$;

$\ln$

the natural logarithm: CE is measured in nats; with $\log_2$, in bits, $1$ nat $\approx1.443$ bits;

$-$

the log of a probability is negative; the minus makes the loss non-negative.

For a one-hot label only one term of the sum survives. Substituting $s=\operatorname{softmax}(z)$ gives a formula directly in the logits:

The last form is the one that matters in practice: the loss is computed without an explicit softmax. For a batch of $N$ examples the losses are averaged:

The cheat sheet’s example — logits $(2,\,1,\,0)$, the correct class is the first:

Result: $\mathrm{CE}\approx0.408$, gradient $s-y\approx(-0.335,\,0.245,\,0.090)$. Had the third class been correct, the loss would be $-\ln0.090\approx2.408$ — a confident mistake.

Entropy, cross-entropy and KL divergence

The name comes from information theory. Let events come from a distribution $p$ while we encode them according to a model $q$. Then:

$H(p)$ is the irreducible uncertainty of the data themselves (the entropy), and $\mathrm{KL}(p\,\|\,q)\ge0$ is the penalty for the model $q$ differing from the truth (Gibbs’ inequality; equality only when $q=p$). So the minimum of cross-entropy over $q$ is reached at $q=p$ and equals $H(p)$. For a one-hot label $H(y)=0$, and CE coincides with the KL divergence.

The derivative: why softmax + CE give s − y

Let us compute the gradient of the loss with respect to the logits. One can go through the softmax Jacobian, but the form $\mathrm{CE}=\operatorname{LSE}(z)-z_c$ is shorter: the derivative of LSE is softmax (see the softmax page), and the derivative of $z_c$ with respect to $z_k$ is $y_k$.

This formula has three important consequences:

• the gradient is bounded: every component $s_k-y_k$ lies in $(-1,\,1)$, and they sum to 0 — the correct logit is pulled up exactly as much as the others together are pulled down;
• no saturation on mistakes: if the model is confidently wrong ($s_c\to0$), the gradient on the correct logit tends to $-1$, not to zero. Compare with MSE on top of softmax, where the Jacobian factors kill the gradient precisely when the error is large (see the section on CE and MSE);
• fading on success: as $s\to y$ the gradient tends to 0 — well-learned examples barely affect training.

The loss landscape in 3D

Fix the third logit $z_3=0$ and draw the loss as a function of the other two. For correct class 1:

Where $z_1$ is much larger than both $z_2$ and zero, both exponentials under the logarithm are small and the loss is close to 0. Where the first class loses, the loss grows almost linearly — roughly as $\max(z_2,0)-z_1$. The result is a convex “valley” that slopes gently down but never reaches zero.

Label smoothing

Label smoothing replaces the one-hot target with a mixture with the uniform distribution:

The correct class gets $1-\varepsilon+\varepsilon/K$, each of the others $\varepsilon/K$. Now the loss has a genuine minimum: the gradient $s-y^{\mathrm{LS}}$ vanishes when $s=y^{\mathrm{LS}}$. If all wrong logits are equal and the correct one is ahead of them by $m$, the optimal margin is finite:

Label smoothing (Szegedy et al., 2016, Inception-v3) limits the growth of the logits and usually improves calibration; it is used, for example, in the original Transformer with $\varepsilon=0.1$. The price: the model is never 100% sure any more, and the information about class similarity in its logits is partly erased, which can hurt distillation (Müller et al., 2019).

CE versus MSE: why classifiers are trained on cross-entropy

Why not train a classifier with the squared error $\|s-y\|^2$ between the probabilities and the one-hot label? Compare the gradients with respect to the logits:

The MSE gradient carries a factor $s$ from the Jacobian. If the model is confidently wrong — almost all the mass sits on one wrong class — the gradient nearly vanishes: for small $s_k$ the factor $s_k$ is small, and for the favourite wrong class the bracket is almost zero. The model does not “hear” its own gross mistake. The CE gradient has no such factor.

Language models and perplexity

At every position $t$ a language model predicts the distribution of the next token over a vocabulary of $V$ tokens (tens or hundreds of thousands). This is classification into $V$ classes, and the loss is the mean cross-entropy over positions:

Perplexity is the exponential of CE, i.e. the geometric mean of the values $1/p_t$. It reads as an “effective number of choices”: a model that always spreads its probability uniformly over $k$ tokens, the correct one always among them, has perplexity exactly $k$. A model that knows nothing about the language (uniform over the vocabulary) has $\mathcal L=\ln V$ and $\mathrm{PPL}=V$.

In code

The main rule is to compute CE from the logits through log-softmax: $\mathrm{CE}=\operatorname{LSE}(z)-z_c$. The route “softmax, then logarithm” gives $\ln0=-\infty$ on confident mistakes: for logits $(0,\,800,\,0)$ and the first class correct, naive code returns inf, while the stable one returns the exact answer $800$.

pythonimport numpy as np
np.set_printoptions(precision=3)

def log_softmax(z):
    z = z - z.max(axis=-1, keepdims=True)          # stability: the max trick
    return z - np.log(np.exp(z).sum(axis=-1, keepdims=True))

def cross_entropy(logits, y):
    """Mean CE over a batch. logits: (N, K), y: (N,) — class indices."""
    logp = log_softmax(logits)
    return -logp[np.arange(len(y)), y].mean()

def cross_entropy_grad(logits, y):
    # dCE/dz = (softmax(z) − onehot(y)) / N
    s = np.exp(log_softmax(logits))
    s[np.arange(len(y)), y] -= 1
    return s / len(y)

z = np.array([[2.0, 1.0, 0.0]])
print(cross_entropy(z, np.array([0])))        # ≈ 0.4076
print(cross_entropy(z, np.array([2])))        # ≈ 2.4076 — a confident mistake
print(cross_entropy_grad(z, np.array([0])))   # [[-0.335  0.245  0.09 ]]
print(cross_entropy(z + 1000, np.array([0]))) # ≈ 0.4076 — no overflow

pythonimport torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.0]], requires_grad=True)
target = torch.tensor([0])                  # a class index, not one-hot
loss = F.cross_entropy(logits, target)      # tensor(0.4076): log_softmax + NLL
loss.backward()
print(logits.grad)                          # tensor([[-0.3348,  0.2447,  0.0900]])

F.cross_entropy(logits, target, label_smoothing=0.1)   # tensor(0.5076)

# A language model: logits (B, T, V), tokens (B, T)
B, T, V = 2, 5, 50000
lm_logits = torch.randn(B, T, V)
tokens = torch.randint(0, V, (B, T))
loss = F.cross_entropy(lm_logits.reshape(-1, V), tokens.reshape(-1))
print(torch.exp(loss))                      # perplexity: of order V for random logits

Common mistakes

1. A softmax before CrossEntropyLoss — a double softmax: the probabilities are squeezed, the loss cannot reach zero, the gradients are distorted. Pass logits.
2. np.log(softmax(z)) instead of log-softmax — on a confident mistake the probability rounds to 0, the loss becomes inf and the gradient NaN. The “patch” clip(p, 1e-7, 1) saves you from infinity but cuts off the gradient; LSE is the right way.
3. Mixing up one-hot and class indices. In Keras these are different functions: categorical_crossentropy expects one-hot, sparse_categorical_crossentropy expects indices. In PyTorch indices must be long, and probability targets must have the same shape as the logits.
4. Sum instead of mean. With reduction='sum' the loss and gradient grow with the batch size, and a tuned learning rate stops fitting.
5. Ignoring class imbalance. CE averages over examples, so a rare class barely affects the gradient. Class weights (weight) or Focal Loss help.
6. Incomparable numbers: losses in nats and in bits, perplexities of models with different tokenizers.

Links to other formulas

• Softmax: gives the probabilities $s$; together with CE, the gradient $s-y$, and $\mathrm{CE}=\operatorname{LSE}(z)-z_c$.
• BCE: the case $K=2$ with one output: $-\bigl[y\ln\hat y+(1-y)\ln(1-\hat y)\bigr]$.
• Sigmoid: for two classes softmax reduces to the sigmoid, and CE turns into BCE.
• Focal Loss: $-(1-p_t)^\gamma\ln p_t$ — cross-entropy damped on easy examples.
• MSE: on top of softmax it saturates on confident mistakes (see the section on the comparison).
• Adam: the optimizer usually used to minimize CE in deep networks.

Summary

Check yourself