The sigmoid: the logistic function

Why we need it

A neural network, logistic regression, gradient boosting — inside all of them you end up with a plain number: $3.7$, $-12$, $0.2$. But the question is often different: how likely is the email spam, will the client repay the loan, is there a person in the frame. A probability must lie between 0 and 1. The sigmoid is the most natural bridge from “any number” to “a probability”.

Where it appears:

• the output of a binary classifier — logistic regression, the last neuron of a network, $p=\sigma(z)$;
• gates in LSTM and GRU — a number between 0 and 1 decides what share of the information to let through and what to forget;
• gradient boosting for classification — XGBoost, LightGBM and CatBoost add trees into a raw score, and the sigmoid turns it into a probability;
• modern activations — SiLU (Swish) is $x\cdot\sigma(x)$; the SwiGLU block of LLaMA and Qwen is built on it;
• statistics — the cumulative distribution function of the logistic law and the link between probability and log-odds (logit).

Intuition: squeezing infinity

Picture a road between a floor at $y=0$ and a ceiling at $y=1$. On the left the road almost lies on the floor, on the right it almost touches the ceiling, and in the middle it rises smoothly. However far you walk left or right, the floor and the ceiling stay out of reach — they are the asymptotes.

Drag the point on the plot. The orange line is the tangent: its slope is the derivative $\sigma'(x)$. The climb is steepest at the centre, while at the edges the road is almost flat — the derivative there is close to zero.

The formula, bone by bone

Let us assemble the sigmoid from three simple steps and follow what each one does to the number $x$.

$e^{-x}$

the exponential is always positive: as $x\to+\infty$ it tends to 0, as $x\to-\infty$ to infinity;

$1+e^{-x}$

the denominator is always greater than 1 — so the fraction is always less than 1;

$\frac{1}{1+e^{-x}}$

the reciprocal of a positive number greater than 1 lies strictly between 0 and 1.

It pays to recognise the same function in other forms:

The second form comes from multiplying the numerator and the denominator by $e^x$; it is convenient for negative $x$ (see the section on code). The third links the sigmoid to the hyperbolic tangent: $\tanh x=2\sigma(2x)-1$ is the same S-curve, squeezed horizontally by 2, stretched twice as tall and shifted down by 1.

The table shows two symmetries: the sigmoid values at $\pm x$ add up to one, and the derivative is an even function.

Properties and shape

• Domain: all real numbers; range: the open interval $(0,\,1)$.
• Monotonicity: strictly increasing, since $\sigma'(x)>0$ for every $x$.
• Asymptotes: $\lim_{x\to-\infty}\sigma(x)=0$ and $\lim_{x\to+\infty}\sigma(x)=1$.
• Point symmetry about $(0,\,0.5)$: $\sigma(-x)=1-\sigma(x)$.
• Inflection point: a single one, at $x=0$; the function is convex on the left and concave on the right.
• Lipschitz continuity: $|\sigma(a)-\sigma(b)|\le\tfrac14|a-b|$ — the output changes no faster than a quarter of the change in the input.

Weight and bias: how a neuron reshapes the sigmoid

In a neuron the sigmoid receives not $x$ but $z=wx+b$. The weight $w$ stretches the curve horizontally, the bias $b$ shifts it. The centre, where the output equals $0.5$, lands at $x_0=-b/w$, and the slope at the centre is $w/4$.

The derivative: σ′ = σ(1 − σ)

This is the key formula of the page. Let us derive it in two ways — head-on and elegantly.

Why this matters in practice: on the forward pass the network has already computed $s=\sigma(z)$. On the backward pass the derivative costs a single multiplication $s(1-s)$ — no new exponential.

The maximum of the derivative and the second derivative

The product $s(1-s)$ for $s\in(0,1)$ is an upside-down parabola with its vertex at $s=\tfrac12$. Hence $\max\sigma'=\tfrac12\cdot\tfrac12=\tfrac14$, reached at $x=0$. Differentiating once more:

The second derivative vanishes at $\sigma=\tfrac12$, i.e. at the inflection point $x=0$. Its extrema are at $x=\pm\ln(2+\sqrt3)\approx\pm1.317$, where the slope of the sigmoid changes fastest; the value there is $\mp\sqrt3/18\approx\mp0.096$.

The Taylor series around zero

From the link with tanh we get the expansion:

Near zero the sigmoid is almost linear: $\sigma(x)\approx\tfrac12+\tfrac{x}{4}$. That is why a network of sigmoids with small weights behaves almost like a linear model. The series converges only for $|x|<\pi$: the function has singular points in the complex plane at $x=\pm i\pi$.

The probabilistic meaning: odds and the logit

Why is it reasonable to read the sigmoid’s output as a probability? Let $p$ be the probability of an event. The ratio $p/(1-p)$ is called the odds: at $p=0.8$ the odds are $4:1$. The logarithm of the odds is the logit:

The logit and the sigmoid are inverses of each other. The logit maps a probability from $(0,1)$ onto the whole real line, where contributions of features are easy to add up. Logistic regression assumes exactly that: the log-odds depend linearly on the features, $\ln\frac{p}{1-p}=w^\top x+b$. Hence the sigmoid at the output.

The sigmoid in many dimensions

A neuron has many inputs: $z=w_1x_1+w_2x_2+\dots+b$. The sigmoid of such a linear function is a “fold” that rises from 0 to 1 across the boundary $w^\top x+b=0$ and stays constant along it. The weight vector $w$ points in the direction of the steepest ascent, and its length sets how steep the fold is.

Training uses the same derivative. For the BCE loss the gradient with respect to the logit is $\sigma(z)-y$ (derived on the BCE page), so a gradient-descent step on the weights is:

The sigmoid in training: saturation and vanishing gradients

If a neuron’s inputs $z$ are large in magnitude, the sigmoid works on its flat tails, where $\sigma'(z)\approx0$. Hardly any gradient flows through such a neuron — it is saturated. Saturation, $\sigma'<0.01$, starts at $|z|>4.6$.

This gives the initialization rule: weights are chosen so that the variance of $z$ is of order one (LeCun initialization, $\operatorname{Var}w=1/n_{\text{in}}$, or Glorot (Xavier) initialization, $\operatorname{Var}w=2/(n_{\text{in}}+n_{\text{out}})$), and the data are normalized.

Vanishing gradients in a deep network

Let $h_k=\sigma(z_k)$, $z_k=W_kh_{k-1}$. By the chain rule the gradient of the loss $\mathcal L$ with respect to the output of layer $l$ is a product of the Jacobians of all layers above it, and each of them contains $\operatorname{diag}\bigl(\sigma'(z_k)\bigr)$:

For the sigmoid every $\sigma'\le\tfrac14$, so after 10 layers the factor from the derivatives alone is at most $0.25^{10}\approx9.5\cdot10^{-7}$. With Glorot initialization the weights $W_k$ do not compensate for it: they keep the norm of the signal at roughly one.

In code: stable and without surprises

The naive formula $1/(1+e^{-x})$ at $x=-1000$ computes $e^{1000}$ — an overflow. The result still comes out as 0, but with a warning, and $\log\sigma(x)$ already yields $-\infty$. The second problem is loss of precision: $1-\sigma(40)$ in float64 is exactly 0, while the true value $\sigma(-40)\approx4.25\cdot10^{-18}$.

pythonimport numpy as np

def sigmoid(x):
    """Stable sigmoid: exp never gets a large positive argument."""
    x = np.asarray(x, dtype=float)
    out = np.empty_like(x)
    pos = x >= 0
    out[pos] = 1.0 / (1.0 + np.exp(-x[pos]))
    ex = np.exp(x[~pos])
    out[~pos] = ex / (1.0 + ex)
    return out

def sigmoid_grad(x):
    s = sigmoid(x)
    return s * (1.0 - s)

def log_sigmoid(x):
    # log σ(x) = −softplus(−x) = −log(1 + e^(−x)), without log(0)
    return -np.logaddexp(0.0, -x)

print(sigmoid([-40.0, 0.0, 40.0]))        # [4.24835426e-18 5.00000000e-01 1.00000000e+00]
print(1 - sigmoid(40.0), sigmoid(-40.0))  # 0.0 4.248354255291589e-18 — the subtraction ate everything
print(log_sigmoid(-1000.0))               # -1000.0 (a naive np.log(sigmoid) would give -inf)

pythonimport torch
import torch.nn as nn

z = torch.tensor([-2.0, 0.0, 2.0], requires_grad=True)
p = torch.sigmoid(z)            # tensor([0.1192, 0.5000, 0.8808])
p.sum().backward()
print(z.grad)                   # tensor([0.1050, 0.2500, 0.1050]) = σ(1 − σ)

# For classification pass the logits to the loss, not σ(z):
loss_fn = nn.BCEWithLogitsLoss()  # a stable log σ(z) inside
loss = loss_fn(z, torch.tensor([0.0, 1.0, 1.0]))

Common mistakes

1. Sigmoids in the hidden layers of a deep network — vanishing gradients and shifted (non-centred) activations. Use ReLU, GELU, SiLU; keep the sigmoid for the output or for gates.
2. Sigmoids instead of softmax for mutually exclusive classes. For $K>2$ mutually exclusive classes you need softmax; a sigmoid on every output means multi-label, where the classes are independent.
3. A double sigmoid: the model already returns a probability and the loss applies σ again (for example, BCEWithLogitsLoss on top of sigmoid). Training runs, but the “probabilities” are squeezed into $(0.5,\,0.73)$, the loss cannot get small, and the gradients are distorted.
4. The “default” 0.5 threshold. For calibrated probabilities it minimizes the error rate, but only when both kinds of error cost the same. With unequal costs, or for F1 or recall on imbalanced classes, the best threshold is different — tune it for your metric on validation data.
5. Reading “σ(z) = 0.9 means the model is 90% sure” without calibration. The sigmoid gives a number in $(0,1)$, but calibration is a property of training (BCE encourages it, overfitting spoils it).

Links to other formulas

• tanh: $\tanh x=2\sigma(2x)-1$ — a centred sigmoid with maximum slope 1.
• Softmax: for two classes $\operatorname{softmax}(z_1,z_2)_1=\sigma(z_1-z_2)$.
• BCE: paired with the sigmoid it gives the gradient $\hat y-y$ with respect to the logit — without the factor $\sigma'$, so no saturation while the model is wrong.
• SiLU / Swish: $x\,\sigma(x)$ — the sigmoid as a “soft gate” for the input itself.
• Focal Loss: usually computed from the probability of the correct class $p_t$: $\sigma(z)$ for $y=1$ and $1-\sigma(z)=\sigma(-z)$ for $y=0$.

Summary

Check yourself