BCE: binary cross-entropy

Why we need it

A binary classifier outputs a number $\hat y\in(0,\,1)$ — its probability that an object belongs to class 1: the email is spam, the transaction is fraud, the scan shows a tumour. To learn, the model needs a loss function: a single number that grows as the prediction gets worse and can be differentiated. Binary cross-entropy (BCE; in statistics, log loss or negative log-likelihood) is the standard choice for this task.

Where it is used:

• logistic regression and the output of any binary classifier — the pair sigmoid + BCE;
• multi-label classification — every label (photo tags, toxicity categories) gets its own sigmoid and its own BCE, the labels are independent;
• gradient boosting — binary:logistic in XGBoost, binary in LightGBM, Logloss in CatBoost are the same BCE on the raw score of the trees;
• the GAN discriminator is trained with BCE: “real” is 1, “generated” is 0;
• reward models in RLHF and DPO — the loss $-\log\sigma(r_{\text{better}}-r_{\text{worse}})$ is BCE with label 1 on the difference of scores (the Bradley–Terry model);
• a metric — log loss in competitions and reports: it checks not only whether the model guessed right but also how honest its probabilities are.

Intuition: a penalty for surprise

Think of the model as a weather forecaster who names the probability of rain every day. If they said “90%” and it rained, they are hardly surprised — a small penalty. If they said “10%” and it rained, they were wrong and sure of it; the penalty should be large. And if they refuse to commit and always say “50%”, the penalty is the same every day — $\ln2\approx0.693$.

BCE is built exactly like that: the loss is $-\ln$ of the probability of what actually happened. For the correct answer $y=1$ it is $-\ln\hat y$ (the red curve), for $y=0$ it is $-\ln(1-\hat y)$ (the blue one). Drag the point.

The formula, bone by bone

Take one term — the loss on a single object — and look at every part.

$y\in\{0,1\}$

the correct answer (label). It acts as a switch: for $y=1$ the second term is multiplied by 0, for $y=0$ the first one;

$\hat y\in(0,\,1)$

the predicted probability of class 1, usually $\hat y=\sigma(z)$, where $z$ is the logit (the raw output of the model);

$1-\hat y$

the predicted probability of class 0 — the two probabilities add up to 1;

$\log$

the natural logarithm ($\ln$): the loss is measured in nats; with $\log_2$ it would be in bits;

$-$

the minus is needed because the logarithm of a probability is negative: $\log\hat y\le0$, while a loss must be $\ge0$;

$\frac1n\sum_{i=1}^{n}$

averaging over objects, so that the loss does not depend on the batch size.

The switch gives a compact form: let $p_t$ be the probability the model gave to the correct class ($p_t=\hat y$ for $y=1$ and $p_t=1-\hat y$ for $y=0$). Then

The second form, “BCE on logits”, will be needed to derive the gradient and to write stable code. If the labels are written as $s=\pm1$, the same loss becomes the logistic loss $\log(1+e^{-s z})$, where $s z$ is the margin: a positive margin means a correct answer.

The table shows the asymmetry: on the right side the loss quickly settles towards zero, on the wrong side it grows like a logarithm, and the slope $-1/\hat y$ reaches $-1000$ at $\hat y=0.001$.

An example with four objects

Labels $y=(1,\,0,\,1,\,1)$, predictions $\hat y=(0.9,\,0.2,\,0.6,\,0.95)$. The probabilities of the correct answers: $0.9$, $0.8$, $0.6$, $0.95$. The losses: $0.105$, $0.223$, $0.511$, $0.051$. The mean is $\mathrm{BCE}\approx0.223$. The third object is the main “culprit”: the model gave the correct answer only $0.6$.

Where it comes from: maximum likelihood

BCE is not made up by eye — it is derived. Suppose the model says: “the object belongs to class 1 with probability $\hat y$”. That is a Bernoulli distribution, and its formula fits on one line — again with the label as a switch, but now in the exponent:

If the objects are independent, the probability of seeing all the labels of the training set — the likelihood — is a product. Maximum likelihood chooses the model parameters that make this product as large as possible:

The logarithm turns the product into a sum and does not move the maximum (it is strictly increasing), the minus turns the maximum into a minimum, and $\tfrac1n$ turns the sum into a mean. So minimizing BCE is the same as maximizing the likelihood of a Bernoulli model.

In the example above: $\mathcal L=0.9\cdot0.8\cdot0.6\cdot0.95=0.4104$, and $-\ln0.4104\approx0.891$ is exactly the sum of the four losses; divided by 4 it is the same $\mathrm{BCE}\approx0.223$.

The information-theory view: surprise, entropy, KL

The same result comes from another direction. In Claude Shannon’s information theory an event with probability $p$ carries the self-information $I(p)=-\log p$, also called its surprisal. A certain event ($p=1$) is no surprise ($I=0$); a rare one is a big surprise; the surprises of independent events add up. In bits ($\log_2$) an event with probability $\tfrac18$ carries 3 bits; in nats ($\ln$) one bit equals $\ln2\approx0.693$.

Let the real world be a coin with probability $y$ of “1”, while the model believes in the probability $\hat y$. Then:

The cross-entropy is exactly BCE (hence the name), and the formula also works for “soft” labels $y\in[0,\,1]$. The difference between the two is the Kullback–Leibler divergence (KL):

The entropy $H(y)$ does not depend on the model, so minimizing BCE means minimizing KL, the “distance” from the model’s beliefs to the truth. KL is zero only when $\hat y=y$. For hard labels $y\in\{0,1\}$ the entropy is zero, and BCE equals KL.

Properties

• Non-negativity. $\ell\ge0$; for hard labels $\ell=0$ only in the limit $\hat y\to y$. With a sigmoid at the output $\hat y$ is never exactly 0 or 1, so zero is never reached.
• Unbounded above. For $y=1$, $\hat y\to0$ the loss tends to $+\infty$. But in logits the growth is only linear: $\operatorname{softplus}(-z)\approx-z$ for $z\ll0$, so a confident mistake with logit $-10$ costs about 10, not infinity.
• Bounded gradient in the logit. $|\partial\ell/\partial z|=|\hat y-y|<1$: each object pulls the model with a force of at most 1, however wrong the model is.
• Convex in $\hat y$: $\dfrac{\partial^2\ell}{\partial\hat y^2}=\dfrac{y}{\hat y^2}+\dfrac{1-y}{(1-\hat y)^2}>0$. Convex in the logit: $\dfrac{\partial^2\ell}{\partial z^2}=\hat y(1-\hat y)>0$. Hence convexity in the weights of logistic regression (see below).
• Class symmetry: $\ell(z,y)=\ell(-z,1-y)$ — renaming the classes and flipping the sign of the logit changes nothing.
• A strictly proper scoring rule: if the truth is a coin with probability $p$, the mean loss $H(p)+\mathrm{KL}(p\,\|\,q)$ is minimal only for the forecast $q=p$. BCE rewards honesty (see the section on calibration).
• A baseline. A constant model that always predicts the share of positives $\pi$ gets $\mathrm{BCE}=H(\pi)$. For balanced classes this is $\ln2\approx0.693$, for $\pi=0.1$ it is $0.325$. A model worse than this threshold has learned nothing.

The derivative: why σ′ cancels

This is the central formula of the page. First the derivative with respect to the probability $\hat y$, then with respect to the logit $z$, where $\hat y=\sigma(z)$.

This derivative is awkward: as $\hat y\to0$ or $\hat y\to1$ the denominator tends to zero and the gradient explodes (the table above shows $-1000$ at $\hat y=0.001$). But the model does not output $\hat y$ directly — it outputs a logit, and the probability comes from a sigmoid. The chain rule:

The denominator of the first fraction is exactly the derivative of the sigmoid, $\sigma'(z)=\sigma(z)\bigl(1-\sigma(z)\bigr)$ (derived on the sigmoid page). They cancel, and what remains is “prediction minus truth”.

For the weights of logistic regression, $\hat y_i=\sigma(w^\top x_i+b)$, this gives the short formulas used in training:

Why not MSE: saturation and non-convexity

Why not train a classifier with the squared error $(\hat y-y)^2$, as in regression? Let us compute the gradient with respect to the logit when $\hat y=\sigma(z)$:

Here the factor $\sigma'(z)$ no longer cancels with anything. If the model is confidently wrong ($y=1$, $z\ll0$), then $\sigma'(z)\approx0$ and the gradient is almost zero: the model “does not hear” its worst mistake. At $z=-4$ the BCE gradient is $-0.982$, the MSE gradient $-0.0347$, 28 times smaller; at $z=-6$ it is already 203 times smaller. The largest MSE gradient in magnitude, $-0.296$, is reached at $z=-\ln2\approx-0.693$ — MSE pushes hardest on a model that is only slightly wrong.

The second problem is non-convexity. For $y=1$ the second derivative of MSE in the logit is $-2\sigma(1-\sigma)^2(1-3\sigma)$, negative for $\sigma<\tfrac13$, i.e. for $z<-\ln2$. The loss there is concave and almost flat: summing such terms over many objects produces plateaus and saddle-like regions (Figure 5). The third problem is boundedness: MSE never exceeds 1, so “very wrong” costs almost as much as “hopelessly wrong”.

The loss landscape in 3D

Let us look at the loss as a whole, as a function of the parameters. Take logistic regression with one feature, $\hat y=\sigma(wx+b)$, and 24 points (12 of each class). The mean loss is a function of two variables $L(w,b)$, so it can be drawn as a surface.

What the figure shows:

• BCE is a convex bowl with a single minimum at $(1.441,\,0.149)$, where $L=0.451$. Far from the minimum its walls are almost flat inclined planes, not a parabola: the loss grows linearly and the gradient is bounded.
• MSE has plateaus and folds. Its minimum $(1.374,\,0.320)$, $L=0.150$, lies in a narrow valley, and at the edges the surface is almost horizontal.
• Speed of descent. At the start the gradient norm is $0.969$ for BCE and $0.0174$ for MSE, 56 times smaller. The BCE loss is within $0.01$ of its minimal value already at step 15, and the descent stops (gradient norm $<10^{-4}$) at step 63. MSE gets within the same $0.01$ of its minimal value only by step 600 and stops at step 1150; between steps 100 and 400 it crawls across a plateau where the loss barely changes ($0.500\to0.495$).
• Different minima. The optima of the two losses do not coincide: they are different definitions of the “best” model. BCE looks for honest probabilities, MSE for the least squares of the deviations of probabilities from 0 and 1.

BCE in training

Now the two-dimensional case: 60 points of two classes, the model $\hat y=\sigma(w_1x_1+w_2x_2+b)$, full-batch gradient descent with step $\eta=1$ from zero weights:

Look at the sizes of the points: the largest ones are those on the “wrong” side of the boundary. Their gradient $|\hat y-y|$ is close to 1, while for confidently and correctly classified points it is close to zero. BCE itself decides where to spend the model’s “attention”: mostly on mistakes and on points near the boundary.

Linearly separable data: the weights run off to infinity

If a line separates the classes, BCE can be made arbitrarily small just by increasing $\|w\|$: the boundary stays in place while the sigmoid gets steeper and steeper. There is no minimum — only an infimum of 0. The counter in the figure shows it: at step 100 the loss is $0.0205$ and $\|w\|=3.94$, at step 200 — $0.0133$ and $4.77$, at step 400 — $0.0082$ and $5.76$. The norm grows roughly like the logarithm of the number of steps and never stops.

Numerical stability: BCEWithLogits

The formula $-[y\log\hat y+(1-y)\log(1-\hat y)]$ is flawless on paper but breaks in a computer. If you first compute $\hat y=\sigma(z)$ and then the logarithm, three problems arise:

1. log(0). In float32 the value $\sigma(z)$ rounds to exactly 1 already for $z>16.64$: then $1-\hat y=0$, the loss is $\infty$ and the gradient is NaN. In float64 the same starts at $z>36.74$.
2. Loss of precision before that. The difference $1-\sigma(z)$ of two close numbers loses significant digits (catastrophic cancellation): at $z=16$ naive float32 gives $15.942$ instead of $16$, and the plot turns into steps.
3. Clipping is a patch, not a cure. If $\hat y$ is clipped to $[\varepsilon,\,1-\varepsilon]$ with $\varepsilon=10^{-7}$ (as Keras does when given probabilities), the loss hits a ceiling of $-\ln10^{-7}\approx16.118$, and in float32 even $15.942$, because $1-10^{-7}$ rounds to $1-2^{-23}$. Worse, beyond the ceiling the gradient of the clip is zero: confidently wrong examples stop teaching the model. PyTorch BCELoss does something similar — it clamps the logarithm at $-100$.

The cure is never to compute $\hat y$ separately but to compute the loss directly from the logit, through softplus and the log-sum-exp trick:

pythonimport numpy as np

def sigmoid(z):
    # σ(z) = (1 + tanh(z/2)) / 2 — no overflow for any z
    return 0.5 * (1.0 + np.tanh(0.5 * np.asarray(z, dtype=float)))

def bce_with_logits(z, y, pos_weight=1.0):
    """Stable BCE on logits: y·softplus(−z) + (1 − y)·softplus(z)."""
    z, y = np.asarray(z, dtype=float), np.asarray(y, dtype=float)
    sp_pos = np.logaddexp(0.0, z)      # softplus(z)  = −log(1 − σ(z))
    sp_neg = sp_pos - z                # softplus(−z) = −log σ(z)
    return np.mean(pos_weight * y * sp_neg + (1 - y) * sp_pos)

def bce_grad(z, y):
    """Gradient of the mean BCE with respect to the logits: (σ(z) − y) / n."""
    return (sigmoid(z) - y) / len(z)

z = np.array([3.0, -2.0, 0.5, 20.0])
y = np.array([1.0, 0.0, 1.0, 0.0])
print(bce_with_logits(z, y))         # 5.1624 — almost all of it comes from the last example
print(bce_grad(z, y))                # ≈ [-0.0119  0.0298 -0.0944  0.25]
print(bce_with_logits(z, y, 10.0))   # 6.3384 — positive examples weigh 10 times more

# Naive in float32: σ(20) rounds to 1, and log(1 − 1) = −inf
p = 1 / (1 + np.exp(-np.float32(20.0)))
print(-np.log(1 - p))                # inf (and a RuntimeWarning)

pythonimport torch
import torch.nn as nn

z = torch.tensor([3.0, -2.0, 0.5, 20.0], requires_grad=True)   # logits
y = torch.tensor([1.0, 0.0, 1.0, 0.0])

loss = nn.BCEWithLogitsLoss()(z, y)    # σ and log inside one stable formula
loss.backward()
print(loss.item())                     # 5.1624
print(z.grad)                          # (σ(z) − y) / 4 ≈ [-0.0119, 0.0298, -0.0944, 0.2500]

# Bad: sigmoid first, then BCELoss. σ(20) in float32 equals 1,
# BCELoss clamps log at −100, and the last example costs 100 instead of 20
p = torch.sigmoid(z.detach())
print(nn.BCELoss()(p, y).item())       # 25.1624

# Class imbalance: a mistake on a positive example costs 10 times more
loss_w = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([10.0]))

# Label smoothing: BCEWithLogitsLoss has no such argument, so we smooth by hand
eps = 0.1
y_smooth = y * (1 - eps) + eps / 2     # 1 → 0.95, 0 → 0.05

Class imbalance: weights and pos_weight

In real tasks positive examples are often rare: 1% of transactions are fraud, 0.1% of impressions are clicks. Plain BCE honestly learns probabilities, but the probability of the rare class is small almost everywhere, and with a $0.5$ threshold the model finds hardly anyone. A common remedy is a weight $w_+$ for the positive class (pos_weight in PyTorch):

The weight $w_+$ is equivalent to repeating every positive example $w_+$ times. For a well-specified model the optimal odds are multiplied by $w_+$, i.e. the logit shifts by $\ln w_+$. A constant model for a share of positives $\pi$ chooses

That is why the popular choice $w_+=n_{\text{neg}}/n_{\text{pos}}$ “balances” the classes: the constant becomes $0.5$.

Three conclusions from the figure:

• Without weights the mean prediction equals the class share: $0.091=15/165$. This is no coincidence: for logistic regression with a bias, at the optimum $\partial\mathrm{BCE}/\partial b=\frac1n\sum(\hat y_i-y_i)=0$, i.e. $\sum\hat y_i=\sum y_i$.
• The weight moves the threshold; it does not make the model smarter. With $w_+=10$ the bias $b$ shifts by $1.91$ — close to $\ln10\approx2.30$ (on a finite sample $w$ changes too). Recall goes up, precision goes down: this is the same trade-off curve you can also move along with the threshold.
• Weights spoil calibration. The mean $\hat y=0.244$ no longer equals the share $0.091$. If you need real probabilities, train without weights and choose the threshold (it is $1/(1+w_+)$ for the same decision), or convert the odds back: $\text{odds}=\text{odds}_w/w_+$.

Label smoothing

With hard labels the optimal logit is infinite: for the loss on a positive example to become zero, we need $z\to+\infty$. Gradient descent faithfully goes there. On a single positive example with step $\eta=1$ from $z=0$ the logit after 10, 100 and 1000 steps is $2.26$, $4.59$, $6.91$ — it grows roughly like $\ln t$ and never stops. The model learns to be overconfident.

Label smoothing replaces hard labels with soft ones:

Now the gradient $\sigma(z)-y_s$ vanishes at a finite point $z^*$ where $\sigma(z^*)=y_s$, and the minimal loss is not 0 but the entropy $H(y_s)$ (Figure 2). With $\varepsilon=0.1$: $y_s=0.95$, $z^*=\ln19\approx2.944$, and the same descent gives $1.97$, $2.936$, $2.944$ — the logit stops. Go back to Figure 3 and move the $\varepsilon$ slider: the curve acquires a minimum and the orange gradient crosses zero.

Smoothing acts as a regularizer of confidence: the weights do not run off to infinity, the probabilities do not stick to 0 and 1, and calibration often improves. The price is that the optimal forecast is now at most $1-\varepsilon/2$ (the model is no longer drawn above it), and part of the “dark knowledge” about class similarity is lost (Müller, Kornblith, Hinton, 2019). In Keras it is the label_smoothing argument of BinaryCrossentropy; in PyTorch the labels for BCE are smoothed by hand (see the code above).

Calibration: BCE rewards honesty

A model is calibrated if its probabilities can be taken literally: among all objects it gave about $0.8$, class 1 occurs in about 80% of cases. This is checked with a reliability diagram: the predictions are split into bins, and in each bin the mean prediction is compared with the share of ones. The summary number is the expected calibration error:

Why does BCE teach calibration? If among objects with identical features class 1 occurs with frequency $p$, the mean loss of a forecast $q$ is $H(p)+\mathrm{KL}(p\,\|\,q)$, and it is minimal only at $q=p$. Any embellishment — towards confidence or towards caution — costs more. Accuracy cannot do this: it does not care whether the model said $0.51$ or $0.99$.

This is the basis of temperature scaling (Guo et al., 2017): for a trained network one parameter $T$ in $\hat y=\sigma(z/T)$ is fitted by minimizing BCE on a validation set. Here $k=1/T$. The order of predictions and the accuracy stay the same — only the honesty of the probabilities changes.

Common mistakes

1. A double sigmoid. The model has already applied sigmoid, and the loss, BCEWithLogitsLoss, applies it again. Then $\sigma(\sigma(z))\in(0.5,\,0.731)$: the model simply cannot predict class 0 with a probability above one half, and the loss on such objects never drops below $\ln2\approx0.693$.
2. Sigmoid + BCELoss instead of BCEWithLogitsLoss. It works until the first confident example, then come $\log0$, clamping at $-100$ and a zero gradient (see the section on stability).
3. Labels $\pm1$ instead of $\{0,1\}$. With $y=-1$ the BCE formula gives $\log\hat y-2\log(1-\hat y)$, which goes to $-\infty$ as $\hat y\to0$: the loss becomes negative and training diverges. Labels $\pm1$ have their own form, $\log(1+e^{-s z})$.
4. BCE for mutually exclusive classes. If exactly one of $K>2$ classes is correct, you need softmax and categorical cross-entropy. Several sigmoids with BCE mean multi-label, where the classes are independent.
5. Comparing BCE without a baseline. “A loss of 0.05” sounds great, but with 1% positives the constant gives $H(0.01)\approx0.056$ — the model has learned almost nothing. Compare with $H(\pi)$.
6. Class weights and “probabilities”. After pos_weight the sigmoid output is no longer a probability (Figure 8). For decisions with error costs, tune the threshold; for honest probabilities, convert or calibrate.
7. reduction='sum' instead of the mean. The sum grows with the batch size, and so does the effective step size. Change the batch size and the optimal learning rate changes too.

Links to other formulas

• Sigmoid: turns the logit into $\hat y$; its derivative $\sigma(1-\sigma)$ cancels in the BCE gradient.
• Categorical cross-entropy: the generalization to $K$ classes; for $K=2$ softmax + CE coincides with sigmoid + BCE of the logit difference $z_1-z_2$.
• Softmax: for two classes $\operatorname{softmax}(z_1,z_2)_1=\sigma(z_1-z_2)$.
• Focal Loss: $-(1-p_t)^\gamma\log p_t$ — BCE that damps easy examples; with $\gamma=0$ it equals BCE.
• MSE: the right loss for Gaussian noise; with a sigmoid it brings saturation and non-convexity (Figures 4 and 5).
• SGD and Adam: optimizers that use the gradient $\frac1nX^\top(\hat y-y)$.

Summary

Check yourself