SiLU / Swish: an input that gates itself

Why we need it

ReLU has two drawbacks: it has a kink at zero, and on the left it throws away the whole signal together with the gradient. SiLU (Sigmoid Linear Unit) works differently: the input $x$ is multiplied by the sigmoid of itself. A large positive input passes almost in full, a large negative one is almost completely suppressed, and near zero the input passes “partially” — smoothly, without kinks.

The same function with a parameter, $x\,\sigma(\beta x)$, is called Swish. With $\beta=1$ it is exactly SiLU; PyTorch and other libraries call it just that: nn.SiLU.

Where it appears:

• large language models — LLaMA, Qwen, Mistral and PaLM use SiLU inside the SwiGLU block (section 7); in model configs it is the line hidden_act = "silu";
• computer vision — EfficientNet (2019) is built on Swish, and the YOLOv5 and YOLOv8 detectors use SiLU;
• mobile networks — MobileNetV3 uses the “hard” Swish, a cheap piecewise-polynomial approximation $x\cdot\operatorname{ReLU6}(x+3)/6$;
• reinforcement learning — Elfwing, Uchibe and Doya (2017) used it to approximate value functions under the name SiLU.

Intuition: the input decides how much of itself to let through

Picture a tap whose handle is turned by the flow itself. The sigmoid $\sigma(x)$ is how far the tap is open, from 0 to 1. If the input is large and positive, the tap is almost fully open ($\sigma\approx1$) and the output almost equals the input. If the input is strongly negative, the tap is almost closed and the output is close to zero. This mechanism is called self-gating: the input is both the signal and the decision about how much of the signal to pass.

Drag the blue point. The teal curve is the gate $\sigma(x)$, the blue one is the product $x\cdot\sigma(x)$. Look at the area around $x=-1$: the gate is already partly closed, but not fully, while the input is negative — and the output dips below zero.

The formula, bone by bone

Let us build SiLU from two familiar parts and look at it in several forms.

$x$

the neuron’s input (pre-activation); it is also the signal to be let through;

$\sigma(x)=\frac{1}{1+e^{-x}}$

the sigmoid, a number in $(0,\,1)$ — the share of the signal that will pass;

$x\cdot\sigma(x)$

the result: with $\sigma\approx1$ the output is $\approx x$, with $\sigma\approx0$ it is $\approx0$.

Through the hyperbolic tangent ($\sigma(x)=\tfrac12(1+\tanh\tfrac x2)$) we get a form that explains a lot:

Compare with ReLU: $\operatorname{ReLU}(x)=\frac{x}{2}+\frac{|x|}{2}$. Their odd parts are identical, and the even part of SiLU, $\frac x2\tanh\frac x2$, is a smoothed absolute value $\frac{|x|}{2}$: for large $|x|$ the tangent is $\pm1$ and they coincide, while near zero the corner is replaced by the parabola $\approx\frac{x^2}{4}$.

The table shows three features: negative values on the left (the deepest is $-0.2785$), a derivative of $0.5$ at zero, and a derivative above 1 at $x=2$ and $x=3$.

Properties

• Smoothness: SiLU is infinitely differentiable — there is not a single kink.
• Asymptotes: $\operatorname{SiLU}(x)-x\to0$ as $x\to+\infty$ and $\operatorname{SiLU}(x)\to0$ as $x\to-\infty$, exponentially fast in both cases.
• Non-monotonicity: the function decreases on $(-\infty,\,x^*)$ and increases on $(x^*,\,+\infty)$, where $x^*\approx-1.2785$.
• Bounded below: the range is $[-0.2785,\,+\infty)$. It is unbounded above.
• Link with ReLU: $\operatorname{ReLU}(x)-\operatorname{SiLU}(x)=|x|\,\sigma(-|x|)$ — an even function between 0 and $0.2785$. SiLU always lies below ReLU, but never more than $0.2785$ below.
• Antisymmetry of the difference: $\operatorname{SiLU}(x)-\operatorname{SiLU}(-x)=x$, because $\sigma(x)+\sigma(-x)=1$.
• Convexity: the function is convex on the interval $(-2.3994,\,2.3994)$ and concave outside it; the endpoints $\pm2.3994$ are the inflection points.

The minimum: where and why −0.278

The minimum is where the derivative is zero. The result is neatly written through the Lambert W function — the solution of $We^{W}=c$:

The Swish-β family

Adding a steepness $\beta$ to the gate gives the Swish family:

With $\beta=0$ the gate is always half open, and the function is linear. As $\beta\to\infty$ the sigmoid turns into a step, and we get ReLU. Swish-β is a bridge between a linear function and ReLU, and SiLU is the point $\beta=1$ on that bridge. The substitution $u=\beta x$ shows that all members of the family are one and the same graph at a different scale: $\operatorname{Swish}_\beta(x)=\frac{1}{\beta}\operatorname{SiLU}(\beta x)$.

The derivative: above one and below zero

By the product rule the derivative of SiLU is made of two parts: the gate and how fast the gate opens.

Properties of the derivative:

• $\operatorname{SiLU}'(0)=\sigma(0)=\tfrac12$ — at zero the slope is exactly one half;
• $\operatorname{SiLU}'(x)+\operatorname{SiLU}'(-x)=1$ — the derivative of the identity $\operatorname{SiLU}(x)-\operatorname{SiLU}(-x)=x$;
• the derivative is negative for $x<-1.2785$ (left of the minimum), with the smallest value $\approx-0.0998$ at $x\approx-2.3994$;
• the derivative is above 1 for $x>1.2785$ — by the symmetry of the previous item — with a maximum of $\approx1.0998$ at $x\approx2.3994$; then it tends to 1 from above.

The points $\pm2.3994$ are the roots of $x\tanh\frac{x}{2}=2$: that is where the second derivative vanishes

Smoothness and relatives: ReLU and GELU

Three functions on one plot: ReLU, SiLU and GELU $=x\,\Phi(x)$, where $\Phi$ is the cumulative distribution function of the standard normal. Look not only at the functions but also at the first and second derivatives: that is where the difference in smoothness shows.

• ReLU: the first derivative is a step, the second a delta function. For the optimizer the loss surface is made of flat pieces with sharp edges.
• SiLU and GELU: the derivatives are continuous, the second derivatives are smooth “humps” peaking at zero ($0.5$ for SiLU, $2\varphi(0)\approx0.798$ for GELU). The loss landscape becomes smooth, and the gradient changes gradually.
• GELU ≈ Swish with $\beta=1.702$: $x\,\Phi(x)\approx x\,\sigma(1.702x)$ with a largest error of about $0.02$. So GELU looks like a copy of SiLU shrunk $1.702$ times: its dip is shallower and closer to zero ($-0.170$ at $x\approx-0.752$ versus $-0.278$ at $x\approx-1.278$; Swish-1.702 itself would give $-0.164$). Its maximum derivative, however, is higher ($1.129$ versus $1.100$), and scaling cannot explain that: it is the same $1.0998$ for every Swish-β. This is the difference between the normal and the logistic law (see below).

The probabilistic meaning: a random gate

SiLU and GELU share an explanation. Let a neuron pass the input in full if the input is larger than a random noise $\varepsilon$, and block it completely otherwise: $y=x\cdot[\varepsilon\le x]$. Then the mean output is

The sigmoid is the distribution function of the logistic law, $\Phi$ that of the normal one. So SiLU is an “averaged ReLU with a random threshold”: instead of cutting exactly at zero, the neuron cuts at a random point near zero, and on average the result is a smooth curve.

SwiGLU: SiLU in large language models

In a transformer the attention block is followed by a fully connected block (feed-forward network, FFN). The classic version is $W_2\,\operatorname{ReLU}(W_1x)$ (or GELU instead of ReLU). Shazeer (2020) proposed replacing it with a gated block (gated linear unit), where one branch decides how much of the other to let through:

$x\in\mathbb R^{d}$

the token vector of dimension $d$ (for LLaMA-7B $d=4096$);

$W_1x$

the gate branch: through SiLU it decides which components to pass and how much;

$W_3x$

the value branch: linear, with no activation — what gets passed;

$\odot$

the element-wise product of two vectors of dimension $h$ (the hidden size);

$W_2$

the projection back to dimension $d$.

For a single component this is a function of two numbers: $g(a,b)=\operatorname{SiLU}(a)\cdot b$, where $a$ is a component of the gate branch and $b$ of the value branch. It is worth seeing in 3D.

The gradients here are simple and instructive: $\frac{\partial g}{\partial b}=\operatorname{SiLU}(a)$ — the value branch learns in proportion to how far the gate is open; $\frac{\partial g}{\partial a}=\operatorname{SiLU}'(a)\cdot b$ — the gate learns only if there is something to pass through it.

Why the hidden size is 8/3·d

SwiGLU has three matrices instead of two. To keep the parameter count of an ordinary FFN with $h=4d$ ($2\cdot d\cdot4d=8d^2$), the hidden size is reduced to $h=\frac{2}{3}\cdot4d=\frac83 d$: then $3\cdot d\cdot\frac83d=8d^2$. For LLaMA-7B $\frac83\cdot4096\approx10923$ is rounded up to a multiple of 256: $h=11008$. That gives $3\cdot4096\cdot11008=135{,}266{,}304$ parameters per block versus $134{,}217{,}728$ for an FFN with $h=4d$ — only $0.8\%$ more.

SiLU in training

How does a smooth activation help in practice? A network of ReLU neurons assembles a function from line segments, a network of SiLUs from smooth “bends”. If the target dependence is smooth, SiLU neurons find it easier to reproduce. Besides, SiLU has no “dead” zone: its derivative vanishes only at the single point $x^*\approx-1.278$, and for other negative inputs it is small but not zero, so the neuron does not “die” the way a ReLU with a negative input does.

The experiment: a $1\to6\to1$ network approximates the curve $y=\sin1.5x+0.3x$ from 40 points. Both networks start from the same weights and differ only in the activation; training is Adam with step $0.03$ for 2000 steps.

To avoid conclusions from a single run, we repeated the experiment for eight different initializations. SiLU was more accurate in all eight, by a factor of $1.4$ to $5.2$. But this is a property of a smooth target: on a piecewise-linear dependence ReLU may do no worse. So the choice of activation is an empirical question, and in large models it is settled by experiment.

In code

The naive form x / (1 + np.exp(-x)) at $x=-1000$ computes $e^{1000}$ and emits an overflow warning (the result is still $-0$). It is cleaner to compute the sigmoid through $\tanh$: $\sigma(x)=\tfrac12(1+\tanh\tfrac x2)$ — the tangent never overflows.

pythonimport numpy as np

def sigmoid(x):
    # σ(x) = (1 + tanh(x/2)) / 2 — no overflow for any x
    return 0.5 * (1.0 + np.tanh(0.5 * x))

def silu(x):
    return x * sigmoid(x)

def silu_grad(x):
    s = sigmoid(x)
    return s * (1.0 + x * (1.0 - s))      # = SiLU(x) + σ(x)·(1 − SiLU(x))

def swish(x, beta=1.0):
    return x * sigmoid(beta * x)

np.set_printoptions(precision=4, suppress=True)
x = np.array([-1000.0, -1.0, 0.0, 1.0, 2.3994])
print(silu(x))        # [-0.     -0.2689  0.      0.7311  2.1997]
print(silu_grad(x))   # [-0.      0.0723  0.5     0.9277  1.0998]

pythonimport torch
import torch.nn as nn
import torch.nn.functional as F

z = torch.tensor([-2.0, 0.0, 2.0], requires_grad=True)
y = F.silu(z)                    # tensor([-0.2384, 0.0000, 1.7616])
y.sum().backward()
print(z.grad)                    # tensor([-0.0908, 0.5000, 1.0908])

class SwiGLU(nn.Module):
    """The FFN block of LLaMA and Qwen: W2 · (SiLU(W1 x) ⊙ W3 x)."""
    def __init__(self, d, hidden):
        super().__init__()
        self.w1 = nn.Linear(d, hidden, bias=False)   # gate branch
        self.w3 = nn.Linear(d, hidden, bias=False)   # value branch
        self.w2 = nn.Linear(hidden, d, bias=False)   # back to size d
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

ffn = SwiGLU(4096, 11008)                          # the sizes of LLaMA-7B
print(sum(p.numel() for p in ffn.parameters()))   # 135266304

Common mistakes

1. “SiLU is non-negative, like ReLU.” No: for $x<0$ it is negative, with a minimum of $-0.278$. If something that needs a non-negative input follows (a logarithm or a square root, say), this matters.
2. “SiLU is monotonic.” No: to the left of $x\approx-1.278$ it decreases, and its derivative is negative there. It has no inverse on the whole line.
3. Confusing SiLU and GELU. They are different functions: $x\sigma(x)$ and $x\Phi(x)$. You cannot swap one for the other in a pretrained model — the weights are tuned to a specific activation.
4. An activation on both branches of SwiGLU. SiLU sits only on the gate branch $W_1x$; the value branch $W_3x$ is linear. An extra activation changes the model.
5. A hidden size of $4d$ in SwiGLU. With three matrices this gives $1.5$ times more parameters than an ordinary FFN. For a fair comparison take $h\approx\frac83d$.
6. The naive formula with exp(-x) in hand-written kernels: overflow for large negative $x$ and loss of precision in half precision. Use F.silu or the tanh form.

Links to other formulas

• Sigmoid: the gate of SiLU; the derivative $\sigma'=\sigma(1-\sigma)$ enters $\operatorname{SiLU}'$.
• ReLU: the limit of $x\sigma(\beta x)$ as $\beta\to\infty$; $\operatorname{ReLU}-\operatorname{SiLU}=|x|\sigma(-|x|)\le0.2785$.
• GELU: $x\Phi(x)\approx x\sigma(1.702x)$ — the same self-gate with normal noise instead of logistic.
• Leaky ReLU: another way not to zero the negative part — the line $\alpha x$ instead of a smooth dip.
• tanh: $\operatorname{SiLU}(x)=\frac x2\bigl(1+\tanh\frac x2\bigr)$.
• Softmax and Adam: SwiGLU’s neighbours in a transformer — the attention mechanism and the optimizer such models are trained with.

Summary

Check yourself