GELU: the input times a probability

Why we need it

An activation function is the nonlinearity placed after every linear layer. Without it, any number of layers collapses into a single matrix multiplication. For a long time the standard was ReLU: $\max(0,x)$ — let the positive through, zero out the negative. GELU (Gaussian Error Linear Unit, Hendrycks and Gimpel, 2016) solves the same task more gently: the input is not simply passed or cut, it is multiplied by the probability that it should be passed.

Where it appears:

• BERT, RoBERTa, GPT, GPT-2, GPT-3 — in every feed-forward (FFN) block of the transformer: $\operatorname{FFN}(h)=W_2\,\operatorname{GELU}(W_1h+b_1)+b_2$;
• Vision Transformer (ViT) and many computer-vision models after it;
• GeGLU — the gated variant $\operatorname{GELU}(xW)\odot(xV)$, used for example in Gemma; its relative SwiGLU with SiLU powers LLaMA and Qwen;
• wherever smoothness matters: GELU has derivatives of every order, no kink at zero and no “dead” zone with zero gradient like ReLU.

Intuition: the input times a probability

Imagine a gate that decides what share of the signal $x$ to let through. ReLU’s gate is hard: for $x>0$ it is fully open, for $x<0$ it is shut. GELU’s gate is open by $\Phi(x)$ — a share between 0 and 1 that grows smoothly with $x$. A large positive signal passes almost entirely, a large negative one is almost extinguished, and near zero half of the signal gets through: $\Phi(0)=0.5$.

Drag the point. The grey bar is the whole input $x$, the blue one is the part that got through: its length is $x\cdot\Phi(x)$. The teal dashed line is the share $\Phi(x)$ itself.

The formula, bone by bone

All the complexity of GELU is hidden in $\Phi$. Let us take it apart.

$\varphi(t)$

the density of the standard normal distribution $N(0,1)$ — the Gaussian bell with its maximum $1/\sqrt{2\pi}\approx0.399$ at zero;

$\Phi(x)$

the cumulative distribution function: the area under the bell to the left of $x$, i.e. the probability $P(\varepsilon\le x)$ for $\varepsilon\sim N(0,1)$; it grows from 0 to 1, $\Phi(0)=\tfrac12$;

$x\cdot\Phi(x)$

the input multiplied by that probability.

The integral of a Gaussian cannot be expressed through elementary functions, so it has a special name — the error function, hence “Gaussian Error Linear Unit”:

The factor $1/\sqrt2$ converts the standard normal variable with density $e^{-t^2/2}$ to the kernel $e^{-t^2}$ for which erf is defined. The erf itself is odd and runs from $-1$ to $1$, so $\tfrac12(1+\operatorname{erf})$ runs from 0 to 1.

The table shows three things. On the right GELU quickly catches up with ReLU: at $x=3$ the difference is only $0.004$. On the left the output is negative but small: the deepest value in the table is $-0.159$ at $x=-1$. The derivative at zero is $0.5$, and for $x>0.752$ (in the table, from $x=1$ on) it exceeds one: the function grows slightly faster than the line $y=x$ as it catches up with it.

Where it came from: a noisy mask on average

The authors of GELU combined two ideas. ReLU multiplies the input by a deterministic mask: $x\cdot\mathbb 1[x>0]$. Dropout multiplies the input by a random mask that does not depend on the input. What if the mask is random but does depend on the input: the larger $x$, the more likely it is to be passed?

Such a mask is easy to build: take noise $\varepsilon\sim N(0,1)$ and set $m=1$ if $\varepsilon\le x$. Then $P(m=1)=\Phi(x)$ by the definition of the distribution function. Why the normal law? The authors justify it this way: neuron inputs, especially after normalization (BatchNorm, LayerNorm), are distributed roughly normally, and $\Phi(x)$ answers a natural question: what share of typical inputs is smaller than this $x$.

Properties and shape

• Range: $[-0.170,\,+\infty)$ — bounded below, unbounded above.
• Smoothness: GELU is infinitely differentiable; $\operatorname{GELU}(0)=0$, $\operatorname{GELU}'(0)=\tfrac12$.
• Non-monotonicity: the function decreases on $(-\infty,\,-0.752)$ and increases to the right; the minimum is $\approx-0.170$.
• Asymptotes: $\operatorname{GELU}(x)-x\to0$ as $x\to+\infty$ and $\operatorname{GELU}(x)\to0$ as $x\to-\infty$, with the left tail dying out like a Gaussian: $x\Phi(x)\approx-\varphi(x)$, already $\operatorname{GELU}(-4)\approx-1.3\cdot10^{-4}$.
• Identity: $\operatorname{GELU}(x)-\operatorname{GELU}(-x)=x$, because $\Phi(x)+\Phi(-x)=1$.
• Convexity: only on the interval $(-\sqrt2,\,\sqrt2)$; outside it the function is concave (the sign of the second derivative is in the next section).

GELU = ReLU minus an even “dip”

The identity gives a handy form. For $x>0$: $x\Phi(x)=x-x\Phi(-x)$; for $x<0$: $x\Phi(x)=-|x|\Phi(-|x|)$. In both cases

The correction $|x|\Phi(-|x|)$ is even: it is the same on both sides of zero, peaks at $0.170$ at $|x|=0.752$ and fades quickly. SiLU $x\,\sigma(x)$ has the same property, only its dip is wider and deeper: $0.278$ at $|x|=1.278$.

The derivative: Φ(x) + x·φ(x)

The derivative is needed for backpropagation. Let us derive it and find the minimum of the function.

The derivative has two terms. The first is the share $\Phi(x)$ itself: that is how the derivative would look if the gate did not depend on $x$. The second is a correction for the gate opening as $x$ grows; it is positive to the right of zero and negative to the left. Compare with ReLU, whose derivative is a 0/1 step.

Where the minimum is

The minimum is where $\operatorname{GELU}'(x)=0$, i.e. $\Phi(x)=-x\,\varphi(x)$. This equation has no solution in elementary functions; Newton’s method $x_{k+1}=x_k-\operatorname{GELU}'(x_k)/\operatorname{GELU}''(x_k)$ from $x_0=-1$ gives $-0.6557$, $-0.7448$, $-0.75175$, $-0.751792$ — every iteration roughly doubles the number of correct digits.

The second equality follows from the minimum condition $\Phi(x^*)=-x^*\varphi(x^*)$. At the minimum the share $\Phi(x^*)\approx0.226$ gets through.

Fast approximations: tanh and the sigmoid

Computing erf costs more than elementary functions, so the authors proposed two approximations for when speed matters more than exactness. The first one uses tanh:

This is the formula in the original BERT and GPT-2 code (Hugging Face calls it gelu_new, PyTorch — approximate="tanh"). Its largest error is $4.7\cdot10^{-4}$ at $|x|\approx2.70$. The second approximation uses the sigmoid:

It is coarser: the error reaches $0.020$ at $|x|\approx2.27$. Both errors are even functions, because all three formulas satisfy the identity $f(x)-f(-x)=x$.

The x·Φ(x/s) family in 3D: from ReLU to a line

The noise in the mask need not have unit scale. If $\varepsilon\sim N(0,s^2)$, then $P(\varepsilon\le x)=\Phi(x/s)$, and we get a whole family of functions with a “temperature” $s$:

• $s\to0$: no noise, $\Phi(x/s)$ becomes a step, and $f\to\operatorname{ReLU}(x)$;
• $s=1$: the usual GELU;
• $s\to\infty$: the noise drowns the input, $\Phi(x/s)\to\tfrac12$, and $f\to x/2$ — a straight line, the nonlinearity disappears.

The second equality shows that all members of the family are similar: the minimum is at $x=-0.752\,s$ and equals $-0.170\,s$. On the surface this is the straight teal line along the bottom of the “ravine”.

In a network GELU is applied to a vector element-wise: $h=\operatorname{GELU}(W_1x+b_1)$. Each coordinate is its own “fold” along the hyperplane $w_j^\top x+b_j=0$, but unlike ReLU the fold is smooth, with a small negative dip on the side where ReLU gives zero.

GELU in training and in the transformer

A network with one hidden layer builds its answer as a sum of “bricks” $w_{2j}\,f(w_{1j}x+b_{1j})$. With ReLU every brick is a broken line with a single bend, and the sum is a broken line too. With GELU the bricks are smooth, so a smooth dependence can be described with fewer of them.

Training uses the same derivative: the gradient arriving at a neuron is multiplied by $\operatorname{GELU}'(z)=\Phi(z)+z\varphi(z)$. Unlike ReLU, a neuron with negative $z$ keeps receiving a signal, so GELU networks suffer less from “dead” neurons.

The transformer’s feed-forward block

In BERT and GPT every attention layer is followed by a two-layer perceptron that processes each token on its own:

In BERT-base $d=768$, so each token in each of the 12 layers needs $4d=3072$ GELU evaluations — almost 19 million evaluations per pass for a sequence of 512 tokens. That is why the speed of the activation itself matters, and the tanh approximation is often used instead of the exact erf. The FFN is preceded by LayerNorm, so at the start of training the inputs $W_1h+b_1$ are of order one — exactly the region where GELU has both the dip and the smooth transition, i.e. where it differs noticeably from ReLU.

In code: exact, fast and stable

NumPy has no erf function, so it is taken from SciPy. To avoid losing precision at large negative $x$, compute the distribution function through the complementary error function: $\Phi(x)=\tfrac12\operatorname{erfc}(-x/\sqrt2)$. The formula $\tfrac12(1+\operatorname{erf})$ at $x=-10$ subtracts almost equal numbers and gives exactly 0, although the true value is $\operatorname{GELU}(-10)\approx-7.6\cdot10^{-23}$; this does not matter for the activation itself, but the same $\Phi$ is often needed in other formulas.

pythonimport numpy as np
from scipy.special import erfc        # NumPy itself has no erf

SQRT2 = np.sqrt(2.0)

def gelu(x):
    """Exact GELU: x·Φ(x) with Φ(x) = ½·erfc(−x/√2) — no subtraction of close numbers."""
    x = np.asarray(x, dtype=float)
    return x * 0.5 * erfc(-x / SQRT2)

def gelu_grad(x):
    """The derivative: Φ(x) + x·φ(x)."""
    x = np.asarray(x, dtype=float)
    cdf = 0.5 * erfc(-x / SQRT2)
    pdf = np.exp(-0.5 * x * x) / np.sqrt(2.0 * np.pi)
    return cdf + x * pdf

def gelu_tanh(x):
    # the approximation from the original BERT and GPT-2 code
    return 0.5 * x * (1.0 + np.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * x**3)))

np.set_printoptions(precision=5, suppress=True)
x = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
print(gelu(x))        # [-0.00405 -0.15866  0.       0.84134  2.99595]
print(gelu_grad(x))   # [-0.01195 -0.08332  0.5      1.08332  1.01195]
print(np.abs(gelu_tanh(x) - gelu(x)).max())   # ≈ 4.1e-04 (at x = ±3)

pythonimport torch
import torch.nn as nn
import torch.nn.functional as F

x = torch.tensor([-1.0, 0.0, 1.0], requires_grad=True)
y = F.gelu(x)                          # −0.1587, 0.0000, 0.8413
y.sum().backward()
print(x.grad)                          # −0.0833, 0.5000, 1.0833 = Φ(x) + x·φ(x)
print(F.gelu(x, approximate="tanh"))   # −0.1588, 0.0000, 0.8412

# A transformer FFN block (as in BERT and GPT): d → 4d → d
class FeedForward(nn.Module):
    def __init__(self, d_model=768, mult=4, p_drop=0.1):
        super().__init__()
        self.fc1 = nn.Linear(d_model, mult * d_model)
        self.act = nn.GELU()           # for GPT-2 weights: nn.GELU(approximate="tanh")
        self.fc2 = nn.Linear(mult * d_model, d_model)
        self.drop = nn.Dropout(p_drop)

    def forward(self, h):
        return self.drop(self.fc2(self.act(self.fc1(h))))

Common mistakes

1. Assuming GELU’s output is non-negative, as with ReLU. It can go down to $-0.170$; if a logarithm, a square root or a division follows, this has to be taken into account.
2. Mixing the exact GELU and the tanh approximation between training and inference, or when converting a model (ONNX, TensorRT, another framework). The error is small, but it accumulates across layers.
3. Confusing $x\,\sigma(1.702x)$ with GELU. It is a fast approximation with an error of up to $0.02$, essentially SiLU with $\beta=1.702$.
4. Believing GELU fully cures vanishing gradients. As $x\to-\infty$ the derivative dies out like a Gaussian: already $\operatorname{GELU}'(-4)\approx-0.0005$. Normalizing the inputs is still necessary.
5. Writing $\Phi$ as $\tfrac12(1+\operatorname{erf}(x))$ and forgetting the division by $\sqrt2$. That is the distribution function of $N(0,\tfrac12)$, i.e. $x\Phi(\sqrt2x)$ — a different curve.

Links to other formulas

• ReLU: the limit of $x\,\Phi(x/s)$ as $s\to0$ and the “skeleton” of GELU: $\operatorname{GELU}=\operatorname{ReLU}-|x|\Phi(-|x|)$.
• SiLU / Swish: the same “input × soft gate” scheme with the sigmoid instead of $\Phi$; $x\,\sigma(1.702x)\approx\operatorname{GELU}(x)$.
• Sigmoid: $\sigma(1.702x)$ approximates $\Phi(x)$ with an error of at most $0.0095$ — the link between logit and probit models.
• tanh: the basis of the most popular GELU approximation.
• Leaky ReLU: another way to let a signal through for $x<0$ — a constant small slope instead of a smooth dip.
• Adam and MSE: the optimizer and the loss of the simulation in Figure 7.

Summary

Check yourself