ReLU: the rectified linear unit

Why we need it

Without a nonlinearity a neural network is useless: a composition of linear layers is again a linear function, however many layers you stack. We need a function that “breaks” linearity while being cheap to compute and gentle on the gradient. ReLU (rectified linear unit) does the minimum possible: it passes positive values unchanged and replaces negative ones with zero.

Where it appears:

• convolutional networks — AlexNet, VGG, ResNet: ReLU is the main nonlinearity after the convolutions;
• fully connected networks and tabular models — the default choice for hidden layers;
• the original Transformer (2017) — the FFN block: $\max(0,\,xW_1+b_1)W_2+b_2$; modern LLMs moved to its smooth relatives GELU and SiLU (often inside the gated SwiGLU block);
• mobile networks — ReLU6 $=\min(\max(0,x),\,6)$ in MobileNet suits low-precision arithmetic;
• inside other formulas — $\max(0,\cdot)$ is also the SVM hinge loss, the triplet loss and the positive part of a number $x^+$.

Intuition: a rectifier

The name comes from electronics: a rectifier diode lets current flow one way and blocks it the other way. A ReLU neuron is a detector: while a feature is below the threshold it stays silent; once the threshold is crossed, it responds in proportion to how far.

Drag the point. To the right of zero the tangent coincides with the function itself (slope 1); to the left it lies on the axis (slope 0). At zero there is no single tangent: every line with a slope between 0 and 1 touches the graph from below — these are supporting lines.

The formula, bone by bone

It pays to recognise the same function in different forms — each one highlights its own property:

$\max(0,x)$

a comparison with zero — one operation, no exponentials or divisions;

$\frac{x+|x|}{2}$

for $x>0$ the terms are equal and give $x$; for $x<0$ they cancel out;

$x\cdot\mathbb 1[x>0]$

a gate: the input is multiplied by 0 or 1 depending on its sign. For each example a ReLU network is a linear network with some neurons switched off.

Kinks make any broken line

A neuron $v\cdot\operatorname{ReLU}(x-c)$ is a “hinge”: zero up to the point $c$, then a line with slope $v$. Adding hinges with different $c$ gives a broken line: at each point $c_k$ the slope changes by exactly $v_k$. For example, $\operatorname{ReLU}(x+1)-2\operatorname{ReLU}(x)+\operatorname{ReLU}(x-1)$ is a “hat” of height 1.

This is the main secret of ReLU networks: any continuous piecewise linear function on the line with $m$ kinks is $a+bx$ plus $m$ hinges, i.e. a network with one hidden layer of $m$ neurons (plus the output bias $a$ and the term $bx$, which two neurons can also express: $x=\operatorname{ReLU}(x)-\operatorname{ReLU}(-x)$). And broken lines approximate any continuous function on an interval as closely as we like.

Properties

• Range: $[0,\,+\infty)$ — unbounded above, unlike the sigmoid and tanh.
• Piecewise linearity: two linear pieces, one kink at zero.
• Continuity and the Lipschitz property with constant 1: $|\operatorname{ReLU}(a)-\operatorname{ReLU}(b)|\le|a-b|$.
• Convexity: the graph lies on or below every chord — the maximum of two linear functions is always convex.
• Positive homogeneity: $\operatorname{ReLU}(\alpha x)=\alpha\operatorname{ReLU}(x)$ for $\alpha\ge0$. Scale can be moved between layers: multiplying a layer’s weights by $\alpha>0$ and the next layer’s by $1/\alpha$ gives the same network.
• Idempotence: $\operatorname{ReLU}(\operatorname{ReLU}(x))=\operatorname{ReLU}(x)$.
• Decomposing a number: $x=\operatorname{ReLU}(x)-\operatorname{ReLU}(-x)$ and $|x|=\operatorname{ReLU}(x)+\operatorname{ReLU}(-x)$.
• Sparsity: if the neurons’ inputs are symmetric about zero, about half of the outputs are exact zeros.

The derivative, the subgradient and smooth relatives

Why this hardly hurts training: for continuous data a neuron’s input is exactly zero with probability 0. In floating-point arithmetic exact zeros do occur, but so rarely that the choice of a value in $[0,1]$ hardly matters (in 16-bit arithmetic zeros are noticeably more frequent, and the choice of $\operatorname{ReLU}'(0)$ can sometimes affect training). By convention libraries take 0.

The second derivative is zero everywhere except at zero (in the sense of distributions, $\operatorname{ReLU}''=\delta(x)$). Hence inside each linear region a ReLU network is an affine function of its input, and its gradient with respect to the input is piecewise constant.

Softplus: a smoothed ReLU

If the kink gets in the way (for example, when second-order optimization or a physical model needs a smooth function), it can be rounded off:

The difference $\operatorname{softplus}_\beta(x)-\operatorname{ReLU}(x)=\frac1\beta\ln\bigl(1+e^{-\beta|x|}\bigr)$ is largest at zero, where it equals $\frac{\ln2}{\beta}$; as $\beta\to\infty$ softplus tends to ReLU, and its derivative, the sigmoid, tends to the step. At zero it always equals $\sigma(0)=0.5$ — the middle of the subdifferential $[0,1]$.

A ReLU network in space: linear regions

A first-layer neuron with two inputs is off on one side of the line $w^\top x+b=0$ and linear on the other. Each such line is a fold line. $H$ neurons cut the plane into regions, and within each region the set of active neurons (the activation pattern) is constant, so the network is an ordinary affine function there. The output surface is a polyhedron glued together from flat facets.

$H$ lines in general position divide the plane into at most $1+H+\binom H2$ parts — the number of regions of one layer grows polynomially. Depth works differently: each next layer “folds” the already cut space, and the number of regions can grow exponentially with the number of layers (Montúfar et al., 2014). This is one explanation of why deep networks are more expressive than wide ones with the same number of neurons.

ReLU in training: the network draws a broken line

Let us train one hidden layer $\hat y(x)=\sum_{k=1}^{H}v_k\operatorname{ReLU}(w_kx+b_k)+c$ on points of a sine wave. Each neuron is a hinge with its kink at $x=-b_k/w_k$; gradient descent moves the kinks to where the curve bends and tunes the slopes.

The gradient with respect to a neuron’s parameters comes only from the points where the neuron is on. If there are no such points at all, the neuron stops learning — that is what “dying” means.

The dying ReLU

A neuron is “dead” if $w^\top x+b\le0$ on all training examples. Then its output is always 0, the gradients with respect to $w$ and $b$ are always 0 (see the formula above), and a gradient step by itself will not revive it. For a first-layer neuron, whose inputs are the fixed data, this is permanent (with plain SGD). In deeper layers a neuron’s input changes as the earlier layers learn, and the momentum of SGD with momentum or Adam keeps moving the parameters for a while, so neurons occasionally come back — but you cannot count on it.

How it happens: one large gradient step sharply decreases the bias $b$ or turns $w$ so that the neuron’s hyperplane moves outside the cloud of data. The larger the learning rate, the more often this happens.

How to treat it:

• a moderate learning rate, warm-up and gradient clipping;
• normalization and adaptive optimizers: BatchNorm before the activation centres each neuron’s input over the batch, and Adam limits each parameter’s step to roughly $\eta$, so a single huge gradient cannot throw the bias far (this lowers the risk but does not remove it);
• activations with a non-zero slope on the left — Leaky ReLU, PReLU, ELU, GELU, SiLU: the gradient on the left is not zero, so the neuron can come back;
• careful initialization: zero biases (sometimes slightly positive), He weights. Strongly negative biases kill neurons before training even starts.

He initialization: where the 2 comes from

Let $z=\sum_{j=1}^{n}w_jh_j$ with independent weights, $\mathbb Ew_j=0$, $\operatorname{Var}w_j=c/n$. Then $\operatorname{Var}z=c\cdot q$, where $q=\mathbb E[h^2]$ is the mean square of the inputs. If $z$ is symmetric about zero, ReLU zeroes out exactly half of its distribution, and

With $c=1$ (as for tanh) the signal halves at every layer; with $c=3$ it grows by half. Only $c=2$ preserves the scale: this is He initialization (He et al., 2015), $\operatorname{Var}w=2/n_\text{in}$. The backward pass works the same way (with $n_\text{out}$ instead of $n_\text{in}$, which is the same thing for layers of equal width), so the gradients neither vanish nor explode either.

In code

ReLU never overflows and never loses precision: a comparison with zero is exact in any arithmetic. Attention belongs elsewhere: the convention at zero, dead neurons and initialization.

pythonimport numpy as np

def relu(x):
    return np.maximum(0.0, x)

def relu_grad(x):
    # subgradient: take 0 at the point 0, like PyTorch and TensorFlow
    return (x > 0).astype(float)

x = np.array([-3.0, 0.0, 2.5])
print(relu(x), relu_grad(x))     # [0.  0.  2.5] [0. 0. 1.]

# “Dead” neurons: not a single positive input on the whole dataset
rng = np.random.default_rng(0)
X = rng.normal(size=(1000, 20))
W = rng.normal(size=(20, 64)) * np.sqrt(2 / 20)   # He initialization
for b0 in (0.0, -5.0):
    Z = X @ W + b0
    dead = np.mean((Z > 0).sum(axis=0) == 0)
    print(b0, f"{dead:.0%}")     # 0.0 0%  and  -5.0 66%

pythonimport torch
import torch.nn as nn

x = torch.tensor([-1.0, 0.0, 2.0], requires_grad=True)
torch.relu(x).sum().backward()
print(x.grad)                    # tensor([0., 0., 1.]) — 0 at zero

# He initialization: Var w = 2 / fan_in
layer = nn.Linear(512, 512)
nn.init.kaiming_normal_(layer.weight, mode="fan_in", nonlinearity="relu")
nn.init.zeros_(layer.bias)

# The FFN block of the original Transformer: Linear → ReLU → Linear
ffn = nn.Sequential(nn.Linear(512, 2048), nn.ReLU(), nn.Linear(2048, 512))
# nn.ReLU(inplace=True) saves memory but overwrites the input if it is still needed

Common mistakes

1. ReLU at the output of a regression whose target can be negative. The network physically cannot output $y<0$. A regression output needs a linear layer; ReLU at the output fits only quantities that are non-negative by nature (and even then softplus is often better: a ReLU output has zero gradient for negative inputs).
2. Initialization meant for tanh. With $\operatorname{Var}w=1/n$ the signal in a deep ReLU network decays as $2^{-L}$: after 30 layers by a factor of $2^{30}\approx10^9$ (Figure 7). Use He initialization.
3. Too large a learning rate. It kills neurons en masse, and they cannot be restored (Figure 6). The symptom: the loss is frozen and the share of zero activations is close to 100%.
4. “The derivative of ReLU at zero is 1/2” or “it is undefined, so it is a bug”. Mathematically, at zero there is the subdifferential $[0,1]$, and any value from it is admissible; libraries take 0. In practice it hardly affects training.
5. inplace=True where the input is needed later. The operation overwrites its input; if that input is used in a residual connection or by another operation’s backward pass, PyTorch raises an error or the result is wrong.

Links to other formulas

• Leaky ReLU: $\max(\alpha x,x)$ — a non-zero slope $\alpha$ on the left cures dying.
• GELU and SiLU: $x\Phi(x)$ and $x\sigma(x)$ — smooth “gates” instead of the hard $x\cdot\mathbb 1[x>0]$; far from zero they behave like ReLU.
• Sigmoid: the derivative of softplus is $\sigma(\beta x)$; ReLU displaced the sigmoid from hidden layers because of saturation.
• tanh: saturates on both sides; tanh gets Xavier initialization, ReLU gets He.
• Huber and Triplet loss: the cut-off $\max(0,\cdot)$ appears in loss functions too.
• Adam and SGD: the optimizer and the learning rate decide how many neurons die.

Summary

Check yourself