Neural network formulas: a cheat sheet

Squashes any number into the interval $(0,\,1)$, so the output reads as a probability. The curve is S-shaped and symmetric about the point $(0,\,0.5)$; its asymptotes are $y=0$ and $y=1$.

$\sigma(0)=0.5$, $\sigma(2)\approx0.881$, $\sigma(-2)\approx0.119$. The slope peaks at zero: $\sigma'(0)=0.25$, while $\sigma'(2)\approx0.105$.

def sigmoid(x):
    return 0.5 * (1 + np.tanh(0.5 * x))  # = 1 / (1 + e^-x), no overflow

def sigmoid_grad(x):
    s = sigmoid(x)
    return s * (1 - s)

The same S shape, but centred at zero with range $(-1,\,1)$. The function is odd: $\tanh(-x)=-\tanh(x)$.

$\tanh(1)\approx0.762$, $\tanh'(0)=1$ — four times steeper than the sigmoid at zero, $\tanh'(1)\approx0.420$.

def tanh_grad(x):
    t = np.tanh(x)
    return 1 - t ** 2

Passes positive values unchanged and zeroes out negative ones. It is cheap, and to the right of zero the gradient does not vanish.

$\operatorname{ReLU}(-3)=0$, $\operatorname{ReLU}(2.5)=2.5$. At $x=0$ there is no derivative; libraries take $0$ by convention.

def relu(x):
    return np.maximum(0, x)

def relu_grad(x):
    return (x > 0).astype(float)

Like ReLU, but the negative part is not zeroed: it is multiplied by a small $\alpha$ (with $0<\alpha<1$). It cures the dying ReLU.

With $\alpha=0.01$: $\operatorname{LReLU}(-3)=-0.03$, $\operatorname{LReLU}(2)=2$.

def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

Turns a vector of logits (the network’s raw outputs) into a probability distribution: every $s_i\in(0,\,1)$ and they sum to 1. It sits at the end of a multi-class classifier.

In practice the maximum is subtracted from every logit: $\dfrac{e^{x_i-\max x}}{\sum_j e^{x_j-\max x}}$. The result is the same, with no exponential overflow.

Logits $(2,\,1,\,0)$ give probabilities $\approx(0.665,\,0.245,\,0.090)$.

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

The average absolute error. It is in the same units as the target and is the easiest to interpret. It is more robust to outliers than MSE.

Running example: $y=(3,\,5,\,8)$, $\hat{y}=(2.5,\,5,\,10)$, errors $y-\hat{y}=(0.5,\,0,\,-2)$. $\mathrm{MAE}=(0.5+0+2)/3\approx0.833$.

mae = np.mean(np.abs(y - y_hat))

The average squared error. Squaring punishes large misses hard. Its units are the target’s units squared.

Running example: $y=(3,\,5,\,8)$, $\hat{y}=(2.5,\,5,\,10)$, errors $y-\hat{y}=(0.5,\,0,\,-2)$. $\mathrm{MSE}=(0.25+0+4)/3\approx1.417$: the single error of $-2$ makes up 94% of the sum.

mse = np.mean((y - y_hat) ** 2)

The square root of MSE brings the error back to the target’s units. It is as readable as MAE but still punishes large errors more.

Running example: $y=(3,\,5,\,8)$, $\hat{y}=(2.5,\,5,\,10)$, errors $y-\hat{y}=(0.5,\,0,\,-2)$. $\mathrm{RMSE}=\sqrt{1.417}\approx1.190$ versus $\mathrm{MAE}\approx0.833$.

rmse = np.sqrt(np.mean((y - y_hat) ** 2))

The error as a percentage of the actual value: “on average the model is off by MAPE %”. Useful when the relative error matters more than the absolute one.

Running example: $y=(3,\,5,\,8)$, $\hat{y}=(2.5,\,5,\,10)$, errors $y-\hat{y}=(0.5,\,0,\,-2)$. $\mathrm{MAPE}=\frac{100\%}{3}\bigl(\tfrac{0.5}{3}+0+\tfrac{2}{8}\bigr)\approx13.9\%$.

mape = 100 * np.mean(np.abs((y - y_hat) / y))

Quadratic near zero like MSE (fine tuning), linear for large errors like MAE (not afraid of outliers). Both branches and their derivatives meet at $|r|=\delta$.

Running example: $y=(3,\,5,\,8)$, $\hat{y}=(2.5,\,5,\,10)$, errors $y-\hat{y}=(0.5,\,0,\,-2)$. With $\delta=1$: losses $0.125$, $0$ and $1\cdot(2-0.5)=1.5$; the mean is $\approx0.542$.

def huber(y, y_hat, delta=1.0):
    r = np.abs(y - y_hat)
    return np.mean(np.where(r <= delta, 0.5 * r ** 2,
                            delta * (r - 0.5 * delta)))

For two classes. The label $y\in\{0,1\}$ acts as a switch: of the two terms only one survives, $-\log$ of the probability of the correct answer.

The correct answer is $y=1$. Confident and right, $\hat{y}=0.9$: the loss is $-\ln0.9\approx0.105$. Confident and wrong, $\hat{y}=0.1$: $-\ln0.1\approx2.303$ — 22 times more.

p = np.clip(y_hat, 1e-7, 1 - 1e-7)
bce = -np.mean(y * np.log(p)
               + (1 - y) * np.log(1 - p))

For $K$ classes; $y$ is a one-hot vector (a one in the position of the correct class). Only one term survives — $-\log$ of the probability of the correct class.

Logits $(2,\,1,\,0)$, the correct class is the first. $s\approx(0.665,\,0.245,\,0.090)$, $\mathrm{CE}=-\ln0.665\approx0.408$, gradient $s-y\approx(-0.335,\,0.245,\,0.090)$: the correct logit is pulled up and the others down.

def cross_entropy(logits, k):
    z = logits - logits.max()
    return np.log(np.exp(z).sum()) - z[k]

$\eta=0.001$ (learning rate), $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$. All operations are element-wise: every weight has its own $m$ and $v$.

First step, $g_1=0.5$, $m_0=v_0=0$: $m_1=0.05$, $v_1=0.00025$. Without correction the step would be $3.16$ times too large: $0.05/\sqrt{0.00025}\approx3.16$, because $v$ is understated more than $m$; with it $\hat{m}_1=0.5$, $\hat{v}_1=0.25$, and $\hat{m}_1/\sqrt{\hat{v}_1}=1$. On the first step the weight moves by exactly $\eta$, whatever the scale of the gradient.

m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g ** 2
m_hat = m / (1 - b1 ** t)
v_hat = v / (1 - b2 ** t)
theta -= lr * m_hat / (np.sqrt(v_hat) + eps)

$\operatorname{SiLU}(1)\approx0.731$, $\operatorname{SiLU}(-1)\approx-0.269$: unlike ReLU, small negative values get through.

$\operatorname{GELU}(1)\approx0.841$, $\operatorname{GELU}(-1)\approx-0.159$.

Margin $m=0.2$. $d(a,p)=0.4$, $d(a,n)=1.1$: $L=0$, the triplet is already separated. $d(a,p)=0.9$, $d(a,n)=1.0$: $L=0.1$ — the “other” is too close.

$p_t$ is the probability of the correct class. With $\gamma=2$ an easy example with $p_t=0.9$ weighs 100 times less than in CE ($0.001$ versus $0.105$), while a hard one with $p_t=0.3$ only about 2 times less ($0.590$ versus $1.204$).