Leaky ReLU: the “leaky” ReLU

Why we need it

ReLU is the most popular activation in deep learning: $\max(0,x)$ is instant to compute, and to the right of zero the gradient does not vanish. But it has a weak spot. To the left of zero ReLU does not dampen the signal, it zeroes it — and the gradient with it. A neuron whose input is negative on every example outputs 0, receives a zero gradient and never learns again. This is the “dying ReLU”.

Leaky ReLU is the minimal fix: to the left of zero the graph is not horizontal but slightly tilted, $\alpha x$ with a small $\alpha$ (usually from $0.01$ to $0.3$). The signal “leaks” through the closed door — hence the name.

Where it appears:

• GAN discriminators — in DCGAN (Radford et al., 2015) every discriminator layer uses Leaky ReLU with $\alpha=0.2$;
• YOLO detectors — the Darknet network of YOLOv2 and YOLOv3 is built on Leaky ReLU with $\alpha=0.1$;
• graph networks — the attention coefficients in GAT are computed through Leaky ReLU with $\alpha=0.2$;
• convolutional networks with PReLU — the variant with a learnable $\alpha$ (He et al., 2015) helped push the ImageNet top-5 error below the estimated human error for the first time;
• any network where ReLU “dies” — the swap is one line and costs almost nothing.

Intuition: a door that never closes completely

ReLU is like a door: a positive signal passes through in full, a negative one hits the closed leaf. Leaky ReLU leaves a gap — the negative signal gets through, but weakened $1/\alpha$ times. With $\alpha=0.1$ the input $-2$ becomes $-0.2$.

Drag the point along the graph. The orange “slope triangle” shows the derivative: one step of 1 along $x$ changes the value by $\alpha$ to the left of zero and by 1 to the right. Below is the derivative itself: two horizontal steps with a jump at zero. Move the $\alpha$ slider: from ReLU ($\alpha=0$) through the leaky variant to the line $y=x$ ($\alpha=1$).

The formula, bone by bone

It pays to be able to write the same function in three ways — each one is useful in its own place.

1. By cases

$x$

the neuron’s input — usually the pre-activation $z=w^\top h+b$;

$\alpha$

the slope of the negative part, a hyperparameter; usually $0<\alpha<1$;

$\alpha x$

the “leak”: a negative input is not zeroed but shrunk $1/\alpha$ times.

At $x=0$ both branches give zero, so it does not matter which branch zero belongs to: the function is continuous.

2. Through the maximum

For $x>0$ and $\alpha\le1$ we have $x\ge\alpha x$, and for $x<0$ the opposite, $\alpha x\ge x$. So the maximum picks the right branch by itself. But only when $\alpha\le1$: with $\alpha=2$ the term $2x$ wins on the right, and you get a different function. This form is handy in code (np.maximum(alpha * x, x)), but the condition is easy to forget.

3. A linear part plus the absolute value

Check: for $x>0$ we get $\frac{1+\alpha}{2}x+\frac{1-\alpha}{2}x=x$, for $x<0$ — $\frac{1+\alpha}{2}x-\frac{1-\alpha}{2}x=\alpha x$. This form holds for any $\alpha$ and explains the main point: Leaky ReLU is a linear function plus “a bit of absolute value”. All the nonlinearity lives in the second term, and its share $\frac{1-\alpha}{2}$ shrinks as $\alpha$ grows. At $\alpha=1$ the nonlinearity disappears completely.

To the right of zero all variants coincide with ReLU; they differ only in how much negative signal “leaks” through.

Properties

• Domain and range: the whole real line $\mathbb R$ (for $\alpha>0$). ReLU only takes values in $[0,\,+\infty)$.
• Continuity and piecewise linearity: two lines glued at zero. The function is differentiable everywhere except $x=0$.
• Strict monotonicity for $\alpha>0$: a larger input gives a larger output. Hence the function is invertible: $\operatorname{LReLU}_\alpha^{-1}(y)=y$ for $y>0$ and $y/\alpha$ for $y\le0$, i.e. $\operatorname{LReLU}_{1/\alpha}$. ReLU has no inverse: all negative inputs are glued into zero.
• Positive homogeneity: $\operatorname{LReLU}(cx)=c\,\operatorname{LReLU}(x)$ for $c\ge0$. The scale of the input carries over to the output — hence the simple initialization formulas.
• Lipschitz continuity with constant $\max(1,|\alpha|)$, i.e. 1 for $\alpha\le1$: $|\operatorname{LReLU}(a)-\operatorname{LReLU}(b)|\le|a-b|$.
• Unboundedness above and below, though downwards the function grows $1/\alpha$ times more slowly.
• Special cases: $\alpha=0$ is ReLU, $\alpha=1$ the identity, $\alpha=-1$ the absolute value $|x|$ (no longer monotonic).

Mean and spread of the output

Let the neuron’s input be $z\sim N(0,1)$. Then

At $\alpha=0.01$ the mean is $0.395$ (ReLU: $0.399$), at $\alpha=0.2$ it is $0.319$. The larger $\alpha$, the closer the outputs are to zero mean, which makes the next layer slightly easier to train. The second moment $\frac{1+\alpha^2}{2}$ will be needed in the section on initialization.

The derivative: a step instead of a cliff

On each branch the function is linear, so the derivative is simply the slope of the branch:

What to do about zero in practice? At the kink there is a subgradient: for $0\le\alpha\le1$ any number $g\in[\alpha,\,1]$ defines a line $y=gx$ that lies nowhere above the graph. Libraries pick one value by convention — PyTorch takes $\alpha$ at $x=0$. An input rarely lands exactly on zero, so the choice hardly matters.

Why a step is better than a cliff

By the chain rule the gradient with respect to the weight $w_j$ of a neuron $a=\operatorname{LReLU}(z)$, $z=\sum_j w_jx_j+b$, is

For ReLU the middle factor is 0 when $z<0$; for Leaky ReLU it is $\alpha$. On the left the gradient is $1/\alpha$ times weaker, but not zero: a neuron “stuck” in the negative zone still receives an error signal.

The derivative with respect to α: a step towards PReLU

If $\alpha$ becomes a learnable parameter, we need the derivative with respect to it. It is simple too:

The slope $\alpha$ learns only from examples that land on the left branch — this is how PReLU works (section 9). The second derivative in $x$ is zero everywhere except at zero: all the curvature is concentrated in the kink, and in the sense of generalized functions $\operatorname{LReLU}''(x)=(1-\alpha)\,\delta(x)$.

The dying ReLU and how Leaky cures it

How a neuron dies. Suppose that after an unlucky large step (too high a learning rate, an outlier in the data, a bad initialization) the neuron’s bias $b$ has become strongly negative. Then $z=w^\top x+b<0$ on every example, ReLU outputs 0, the derivative $\operatorname{ReLU}'(z)=0$ — and the gradients with respect to $w$, $b$ and the output weight $v$ vanish all at once. Gradient descent will not move this neuron any more: it is dead (strictly for good for a first-layer neuron under plain SGD; in deeper layers the neuron’s input changes with the earlier layers, and neurons occasionally come back). The CS231n course notes mention that with too high a learning rate as much as 40% of a network can end up dead.

Below is the cleanest possible experiment. A single neuron $f(x)=v\cdot\operatorname{act}(wx+b)$ has to learn the “hinge” $y=1.5\max(0,\,x-1)$ from 41 points on $[-3,\,3]$. Start: $w=1$, $b=-4$, $v=1$ — the kink is at $x=4$, outside the data, so $z<0$ on every example. Training is gradient descent on MSE with step $0.1$.

Note the trade-off. A large $\alpha$ brings the neuron back to life quickly, but the left branch stays tilted: with $\alpha=0.1$ the final model has slope $\alpha v w\approx0.116$ on the left instead of zero, hence the residual error. A small $\alpha$ barely distorts the shape of the function, but revives more slowly.

A network of Leaky ReLUs: a piecewise-linear landscape

In a network with two inputs every neuron $\operatorname{LReLU}(w_k^\top x+b_k)$ “folds” the plane along the line $w_k^\top x+b_k=0$: on one side of it the neuron’s contribution is $1/\alpha$ times flatter than on the other. The sum of neurons $f(x)=\sum_k v_k\operatorname{LReLU}(w_k^\top x+b_k)$ is a polyhedral surface: inside every region where the set of active neurons does not change it is flat, and on the kink lines it changes slope.

The number of regions does not depend on $\alpha$ (as long as $\alpha\ne1$): only the kink lines set it. What does depend on $\alpha$ is what happens where all neurons are off. With ReLU there is a plateau with zero gradient, and examples that land on it tell the network nothing. With Leaky ReLU the plateau is tilted, and the gradient there is generally non-zero.

Deep networks: He initialization for Leaky ReLU

In a deep network the signal passes through dozens of layers $z_{l+1}=W_{l+1}\operatorname{LReLU}(z_l)$. If at every layer the variance is multiplied by a number even slightly different from 1, after 50 layers this becomes an explosion or a disappearance of the signal. Let us compute that factor.

With $\alpha=0$ we get the famous $g=\sqrt2$ of He (Kaiming) initialization for ReLU. For Leaky ReLU the coefficient is slightly smaller: $1.414$ at $\alpha=0.01$, $1.387$ at $\alpha=0.2$, $1.355$ at $\alpha=0.3$.

PReLU, RReLU and the whole family

How to choose $\alpha$? There are three answers.

• A fixed $\alpha$ — the classic Leaky ReLU (Maas, Hannun, Ng, 2013, $\alpha=0.01$). Today $0.1$–$0.3$ is common: in the comparison by Xu et al. (2015) a “very leaky” variant with $\alpha=1/5.5\approx0.18$ worked better than $0.01$.
• A learnable $\alpha$ — PReLU (Parametric ReLU, He et al., 2015). $\alpha$ becomes a parameter — one per layer or one per channel — and is learned by gradient descent together with the weights. The initial value in PyTorch is $0.25$.
• A random $\alpha$ — RReLU (Randomized ReLU): during training $\alpha$ is drawn anew for every element of the input from the uniform distribution $U(1/8,\,1/3)$ (PyTorch’s default bounds); at inference the mean $\frac{1/8+1/3}{2}=\frac{11}{48}\approx0.229$ is used. The noise acts as regularization.

For PReLU the gradient with respect to $\alpha$ is $\min(0,\,z)$ from the section on the derivative: the slope learns only from negative inputs. Let us watch gradient descent find both the weight and the slope.

PReLU adds only one parameter per channel, yet in the work of He et al., together with a new initialization, it made it possible to train very deep convolutional networks and reach a top-5 error of $4.94\%$ on ImageNet — below the estimated human error of $5.1\%$. The authors note that weight decay should not be applied to $\alpha$: it pulls $\alpha$ towards zero, that is, back to ReLU.

Smooth relatives

The kink at zero can be smoothed. ELU replaces the left line with the exponential $\alpha(e^x-1)$, which saturates at $-\alpha$ (here $\alpha$ is ELU’s own parameter, usually 1). SELU adds a scale: $\lambda\approx1.0507$ and $\alpha\approx1.6733$ are chosen so that in a fully connected network with LeCun initialization the activations keep zero mean and unit variance by themselves. GELU and SiLU are smooth and even non-monotonic functions used in today’s transformers.

In code

In NumPy Leaky ReLU is one line. It is safer to write np.where than np.maximum(alpha * x, x): the latter silently breaks for $\alpha>1$.

pythonimport numpy as np

def leaky_relu(x, alpha=0.01):
    # where, not maximum(alpha * x, x): this form is correct for any alpha
    return np.where(x > 0, x, alpha * x)

def leaky_relu_grad(x, alpha=0.01):
    return np.where(x > 0, 1.0, alpha)      # alpha at zero, as in PyTorch

def he_std(n_in, alpha=0.01):
    # He initialization for Leaky ReLU: Var w = 2 / ((1 + alpha²) · n_in)
    return np.sqrt(2.0 / ((1.0 + alpha**2) * n_in))

x = np.array([-3.0, -0.5, 0.0, 2.0])
print(leaky_relu(x))        # [-0.03  -0.005  0.     2.   ]
print(leaky_relu_grad(x))   # [0.01 0.01 0.01 1.  ]
print(he_std(256, 0.2))     # 0.08667190566019205

pythonimport torch
import torch.nn as nn

act = nn.LeakyReLU(negative_slope=0.01)   # 0.01 is the default
x = torch.tensor([-3.0, 0.0, 2.0], requires_grad=True)
act(x).sum().backward()
print(x.grad)                    # tensor([0.0100, 0.0100, 1.0000]) — α at zero

prelu = nn.PReLU(num_parameters=1, init=0.25)   # α is a learnable parameter
print(prelu(torch.tensor([-2.0, 3.0])).detach())  # tensor([-0.5000,  3.0000])

lin = nn.Linear(256, 256)
# std = √(2 / ((1 + a²) · 256)); the default a=0 means “as for ReLU”
nn.init.kaiming_normal_(lin.weight, a=0.2, nonlinearity="leaky_relu")
print(nn.init.calculate_gain("leaky_relu", 0.2))  # 1.3867504905630728

Numerically Leaky ReLU is safe: no exponentials, no division, no overflow. The only subtlety is inplace=True in PyTorch: it saves memory but overwrites the input, which other operations of the graph may still need.

Common mistakes

1. Writing max(alpha * x, x) with $\alpha>1$. The formula holds only for $\alpha\le1$; for larger $\alpha$ you get a different function. The safe form is where(x > 0, x, alpha * x).
2. Too large an $\alpha$. As $\alpha\to1$ the share of nonlinearity $\frac{1-\alpha}{2}$ goes to zero, the function becomes almost linear, and a deep network loses expressiveness. A sensible range is from $0.01$ to $0.3$.
3. Initializing “as for ReLU” without correcting for $\alpha$. The variance factor is $1+\alpha^2$ per layer: at $\alpha=0.3$ the RMS of the signal grows about $1.09^{24.5}\approx8.3$ times from layer 1 to layer 50 (figure 5). Pass $\alpha$ to kaiming_normal_(..., a=alpha).
4. Hoping that Leaky ReLU alone will cure unstable training. It removes the zero gradient, but not the cause: too high a learning rate is still dangerous.
5. Weight decay on the $\alpha$ of PReLU. It pulls $\alpha$ towards zero, and PReLU turns into ReLU. Exclude $\alpha$ from the parameter group with weight decay.
6. Different defaults in PyTorch ($0.01$), TensorFlow ($0.2$) and Keras ($0.3$): when a model moves between frameworks, the result silently changes.

Links to other formulas

• ReLU: the special case $\alpha=0$; more on the dying ReLU there.
• SiLU / Swish and GELU: smooth activations that also let a small negative signal through on the left, but then fade it out.
• Sigmoid and tanh: saturating activations; Leaky ReLU has no saturation at all, and hence no vanishing gradient caused by saturation.
• MSE: the loss function in the experiments of figures 3 and 6.
• SGD and Adam: too large an optimizer step is the main cause of death for ReLU neurons.

Summary

Check yourself