Tanh: the hyperbolic tangent

Why we need it

The sigmoid turns a number into a share between 0 and 1. Inside a network, though, we often need not a share but a signed signal: “above normal” or “below normal”, “amplify” or “damp”. The hyperbolic tangent is the same smooth S-curve, but with outputs in $(-1,\,1)$ and its centre at zero.

Where it appears:

• recurrent networks — in a plain RNN $h_t=\tanh(Wh_{t-1}+Ux_t+b)$; in LSTM and GRU, tanh produces the candidate for the memory, and in LSTM it also bounds the cell’s output;
• hidden layers of classic networks — before the ReLU era tanh was the standard, and it is still good in small networks and physics-informed models (PINN) where smoothness matters;
• a generator’s output — if images are normalized to $[-1,\,1]$, the last layer of a GAN generator (DCGAN, for example) uses tanh;
• bounding actions and logits — in reinforcement learning (SAC) actions are squashed into $(-1,\,1)$ with tanh, and Gemma 2 “soft-caps” its logits: $c\cdot\tanh(z/c)$;
• inside other activations — the popular approximation of GELU is written through tanh.

Intuition: a sigmoid standing on zero

Picture a volume knob with a sign: zero is silence, turning right makes it louder, turning left makes it louder in antiphase. As long as you stay near zero, the output faithfully follows the input: $\tanh x\approx x$. Further out the knob gets stiffer: the output runs into $\pm1$ and stops growing. This is saturation.

Drag the point. The orange line is the tangent; its slope is the derivative. The grey dashed curve is the sigmoid: the same shape, but shifted up, half as tall and four times flatter.

The formula, bone by bone

The formula holds two exponentials: the growing $e^{x}$ and the decaying $e^{-x}$. The numerator is their difference, the denominator their sum. Halves of these expressions have names of their own — the hyperbolic sine and cosine:

$e^{x}-e^{-x}$

the difference is odd: it changes sign under $x\to-x$; for $x>0$ it is positive;

$e^{x}+e^{-x}$

the sum is always positive and at least 2 — the denominator never vanishes;

$|\sinh x|<\cosh x$

the difference is always smaller in absolute value than the sum, hence $|\tanh x|<1$;

$x\to\pm\infty$

one of the exponentials becomes negligible, and the fraction tends to $\pm1$.

Multiplying the numerator and the denominator by $e^{x}$ or by $e^{-x}$ gives forms convenient for computation, and the last form links tanh to the sigmoid:

The table shows the oddness, $\tanh(-x)=-\tanh x$, while the derivative is an even function. The sigmoid is listed for comparison: its values are symmetric not about zero but about $0.5$.

Properties and shape

• Domain: all real numbers; range: the open interval $(-1,\,1)$ — the values $\pm1$ are never reached.
• Oddness: $\tanh(-x)=-\tanh x$; the graph is symmetric about the origin, $\tanh 0=0$.
• Monotonicity: strictly increasing, since $\tanh' x=1-\tanh^2x>0$ for every $x$.
• Asymptotes: $\lim_{x\to\pm\infty}\tanh x=\pm1$.
• Inflection point: a single one, at $x=0$; the function is convex for $x<0$ and concave for $x>0$.
• Lipschitz with constant 1: $|\tanh a-\tanh b|\le|a-b|$; in particular $|\tanh x|\le|x|$, with equality only at $x=0$.
• Almost the identity near zero: $\tanh x=x-\tfrac{x^3}{3}+\dots$ — with small weights a tanh network behaves almost linearly.

A soft limiter: c · tanh(x / c)

Near zero the function $c\tanh(x/c)$ barely changes its input ($\approx x$), while large values are smoothly pressed towards $\pm c$. Unlike the hard cut $\operatorname{clip}(x,-c,c)$, it has no kinks, and its derivative never drops to zero abruptly. Gemma 2 bounds its attention logits ($c=50$) and final logits ($c=30$) this way, so that they do not blow up during training.

The derivative: tanh′ = 1 − tanh²

The key formula of the page. Let us derive it in three ways — each one is useful somewhere else.

In practice: on the forward pass the network has already computed $t=\tanh z$. On the backward pass the derivative costs a single multiplication $1-t^2$ — no new exponentials. That is why frameworks keep the output of tanh, not its input, for the backward pass.

The second derivative and the sharpest bends

Differentiating once more: $\tanh'' x=-2\tanh x\,(1-\tanh^2x)$. It vanishes only at $x=0$ (the inflection point), and its extrema are at $\tanh^2x=\tfrac13$, i.e. $x=\pm\operatorname{artanh}\tfrac{1}{\sqrt3}\approx\pm0.658$, with values $\mp\tfrac{4}{3\sqrt3}\approx\mp0.770$. At these points the slope is already $1-\tfrac13=\tfrac23$ and falls fastest: here the line $y\approx x$ gives way to the bend towards saturation.

The Taylor series around zero

The series converges only for $|x|<\pi/2\approx1.571$: the denominator $\cosh x$ vanishes at the complex points $x=\pm i\pi/2$, and the radius of convergence equals the distance to the nearest one. That is half the sigmoid’s radius ($\pi$), since tanh is squeezed horizontally by a factor of 2.

Zero-centring: why tanh trains better than the sigmoid

Consider a neuron of the next layer: $z=\sum_i w_ia_i+b$, where $a_i$ are the outputs of the previous layer. The gradient of the loss with respect to a weight is

If all $a_i>0$ (sigmoid outputs), then on a single example all components of the gradient share one sign — the sign of $\delta$. An SGD step moves all the neuron’s weights either up or down. If reaching the optimum requires raising some weights and lowering others, the only way is to zigzag.

A second view is through the shape of the loss surface. For a linear neuron with the squared loss $\tfrac12(w^\top a-y)^2$ the Hessian is $\mathbb E[aa^\top]=\operatorname{Cov}(a)+\mu\mu^\top$, where $\mu$ is the mean input. A non-zero mean adds one large eigenvalue along $\mu$, and the “valley” stretches. For the inputs $\sigma(z)$ below the eigenvalues are $0.034$ and $0.530$ (condition number 15.5); for $\tanh(z)$ they are $0.340$ and $0.415$ (1.22).

Tanh on the plane: folds, ridges and hills

A neuron with two inputs computes $\tanh(w_1x_1+w_2x_2+b)$. On the plane this is a fold: the surface goes from $-1$ to $+1$ across the line $w^\top x+b=0$ and stays constant along it. The sum of two opposite folds gives a ridge, and four folds plus one more tanh at the output give a hill that differs from $-1$ only inside a square.

This is how the universal approximation theorem works (Cybenko, 1989; Hornik, 1991): a network with one hidden layer of sufficiently many tanh neurons approximates any continuous function on a closed bounded set to any accuracy. Hills are made of folds, and any landscape can be made of hills. Our hill uses a second tanh at the output for clarity, but the theorem needs only one hidden layer with a linear output. It says such weights exist, but not how many neurons are needed or whether training will find them.

Tanh in training: activations through layers and curve fitting

What happens to the signal in a deep network

Let the weights of each layer be random with variance $g^2/n$, where $n$ is the number of inputs of a neuron. If $g$ is small, the signal fades from layer to layer and tanh acts almost linearly. If $g$ is large, the neurons’ inputs are large and tanh goes into saturation: outputs stick to $\pm1$ and the derivatives are almost zero.

Hence tanh layers use Glorot (Xavier) initialization, $\operatorname{Var}w=\frac{2}{n_\text{in}+n_\text{out}}$, and PyTorch recommends multiplying it by the gain $g=5/3$ (calculate_gain('tanh')): the factor compensates for tanh squeezing the signal. For a square layer this is exactly the case $g=5/3$ in Figure 6.

A tanh network fits a curve

A hidden layer of $H$ neurons and a linear output: $\hat y(x)=\sum_{k=1}^{H}v_k\tanh(w_kx+b_k)+c$. Each term is a “step” with smooth edges; training moves, stretches and scales the steps until their sum lies on the data.

Tanh in recurrent networks and LSTM

A plain recurrent network updates its state with the same formula at every step: $h_t=\tanh(Wh_{t-1}+Ux_t+b)$. Here tanh works as a fuse: however long the text, the state stays in $(-1,\,1)$ and does not “explode”. But boundedness is paid for with the gradient (the factors are multiplied from $s=T$ on the left to $s=t+1$ on the right):

Every factor contains $1-h_s^2\le1$. If the weights are small, the product decays exponentially. If the weights are large, the states are pressed against $\pm1$, $1-h_s^2\to0$ — and the gradient fades too. It survives only in a narrow window between these regimes. In a multidimensional RNN near the edge of this window the gradient can also explode (Pascanu et al., 2013), which is why tanh RNNs are trained with gradient-norm clipping.

LSTM solves this with a separate “highway” — the cell state $c_t$, which is updated by addition rather than through tanh:

Here $f_t,i_t,o_t\in(0,\,1)$ are gates built on sigmoids, and tanh plays two roles: the candidate $g_t\in(-1,\,1)$ can both increase and decrease the memory (a sigmoid could only add), and $\tanh(c_t)$ bounds the output. Along the chain $c_{t-1}\to c_t$ the gradient is multiplied by $f_t$ (ignoring the dependence of the gates on $h_{t-1}$), with no $\tanh'$ factor, so with $f_t\approx1$ it reaches distant steps.

In code: stable and without surprises

The naive formula through exponentials computes $e^{1000}=\infty$ at $x=1000$, and the result is $\infty/\infty=\text{nan}$. The built-in np.tanh and torch.tanh are stable: for $x\gtrsim19$ they return exactly $1.0$ in float64. But then $1-\tanh^2 x$ gives exactly 0, although the true value at $x=20$ is about $1.7\cdot10^{-17}$. If you need the derivative on the far tails, compute it through $\operatorname{sech}^2$ in a stable form.

pythonimport numpy as np

def tanh_grad_from_output(t):
    """The derivative from the already computed output t = tanh(x): one multiplication."""
    return 1.0 - t * t

def tanh_grad(x):
    """sech²x = 4e^(−2|x|) / (1 + e^(−2|x|))²:
    no overflow and no loss of precision."""
    e = np.exp(-2.0 * np.abs(x))
    return 4.0 * e / (1.0 + e) ** 2

def naive_tanh(x):
    return (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))

print(naive_tanh(1000.0))       # nan: inf / inf (plus an overflow warning)
print(np.tanh(1000.0))          # 1.0 — the built-in function is stable
print(1 - np.tanh(20.0) ** 2)   # 0.0: tanh(20) rounds to 1 in float64
print(tanh_grad(20.0))          # 1.6993417021166355e-17 — the correct value

pythonimport torch
import torch.nn as nn

x = torch.tensor([-2.0, 0.0, 2.0], requires_grad=True)
y = torch.tanh(x)               # tensor([-0.9640,  0.0000,  0.9640])
y.sum().backward()
print(x.grad)                   # tensor([0.0707, 1.0000, 0.0707]) = 1 − tanh²

# Initialization for tanh: gain 5/3
layer = nn.Linear(256, 256)
nn.init.xavier_uniform_(layer.weight, gain=nn.init.calculate_gain("tanh"))

# A recurrent network with tanh and gradient clipping against explosions
rnn = nn.RNN(input_size=32, hidden_size=64, nonlinearity="tanh", batch_first=True)
out, h = rnn(torch.randn(8, 100, 32))   # 8 sequences of length 100
out.pow(2).mean().backward()
nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0)

Common mistakes

1. tanh at the output instead of the sigmoid for a probability. The output of tanh lies in $(-1,\,1)$ and is not a probability. Binary classification needs the sigmoid and BCE; tanh at the output is appropriate when the targets are scaled to $[-1,\,1]$.
2. The derivative of the input instead of the output. The formula $1-t^2$ applies to the output $t=\tanh x$. A common slip is $1-x^2$ of the input: at $x=2$ it gives $-3$ instead of $0.071$.
3. Unnormalized inputs and large weights. If the features are in the hundreds or the weights are too large, tanh is saturated from the first steps and hardly learns. Standardize the data and use an initialization meant for tanh.
4. “tanh solves vanishing gradients”. It is better than the sigmoid (a derivative up to 1 and centring), but it saturates too. For deep networks use the ReLU family, normalization and residual connections; for sequences, LSTM/GRU.
5. A home-made tanh through exp. It returns nan for large $|x|$. Use np.tanh or torch.tanh.

Links to other formulas

• Sigmoid: $\tanh x=2\sigma(2x)-1$ and $\sigma(x)=\tfrac12(1+\tanh\tfrac x2)$; the derivatives are related by $\tanh' x=4\sigma'(2x)$.
• ReLU: does not saturate for $x>0$ and displaced tanh from deep convolutional networks.
• GELU: the popular approximation $0.5x\bigl(1+\tanh\bigl(\sqrt{2/\pi}\,(x+0.044715x^3)\bigr)\bigr)$.
• SiLU: $x\sigma(x)=\tfrac x2\bigl(1+\tanh\tfrac x2\bigr)$ — the same S-curve acting as a “soft gate”.
• MSE: the natural partner for a tanh output when the targets lie in $[-1,\,1]$.
• Adam: the optimizer of Figure 7; its adaptive step helps it get through plateaus caused by saturation.

Summary

Check yourself