Abdulatipov J.
Materials · Practice · ML / AI

MAE and MSE by example

20 problems solved step by step, from the example on the board to outliers and choosing a model. Every step is written out as in class: the pairs $y_i$ and $\hat y_i$, the errors, their absolute values and squares, the sum, the division by $n$. The numbers are simple enough for mental arithmetic or pen and paper. And for backpropagation: the derivatives of both metrics with respect to every prediction, a gradient descent step and the chain rule down to the weights of a neuron.

Javharbek Abdulatipov · 50 min read
20 examples8 ★☆☆9 ★★☆3 ★★★
$$\begin{gathered}\mathrm{MAE}=\frac{1}{n}\sum_{i=1}^{n}\bigl|y_i-\hat{y}_i\bigr|\\[6pt]\mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}\bigl(y_i-\hat{y}_i\bigr)^2\end{gathered}$$

How to read the formulas

Both metrics answer the question “how wrong is the model?”. There are $n$ objects; for each we know the correct answer $y_i$ and the model's prediction $\hat y_i$ (read “y hat”). The difference $e_i = y_i - \hat y_i$ is the error on object $i$.

$y_i$
the correct answer (ground truth) for object $i$
$\hat y_i$
the model's prediction for object $i$
$e_i = y_i - \hat y_i$
the error: plus means the model undershot, minus means it overshot
$n$
the number of objects
$\sum_{i=1}^{n}$
the sum over all objects from $1$ to $n$

A four-step recipe

  1. Write out the pairs $(y_i,\ \hat y_i)$ and compute the errors $e_i = y_i - \hat y_i$.
  2. For MAE take the absolute value of each error $|e_i|$; for MSE take the square $e_i^2$.
  3. Add up all the absolute values (or all the squares).
  4. Divide the sum by $n$: that is the MAE (or the MSE).
How to read the answer

MAE is the average size of a miss in the same units as $y$: “on average we are off by 0.83”. MSE is the average squared miss, in “squared” units; it reacts more strongly to large errors. For both metrics, smaller is better and zero is perfect.

Derivatives for backpropagation

A neural network learns like this: it computes the loss (say, MSE), then walks backwards, from the loss to the weights, multiplying derivatives layer by layer with the chain rule. This pass is called backpropagation. The very first link of the chain is the derivative of the loss with respect to the prediction $\hat y_i$: it says in which direction and how strongly to move each prediction. If you are just getting to know MAE and MSE, you can skip this section and come back to it later.

$g_i$
$g_i$ is the derivative of the loss with respect to the prediction $\hat y_i$, a component of the gradient $\nabla_{\hat y} L = (g_1, \dots, g_n)$

The derivative of MSE

$$\begin{aligned}\frac{\partial\,\mathrm{MSE}}{\partial \hat y_i} &= \frac{1}{n}\cdot 2\,(y_i-\hat y_i)\cdot(-1)\\ &= \frac{2}{n}\,(\hat y_i - y_i)\end{aligned}$$

Of the whole sum, only one term depends on $\hat y_i$: $(y_i - \hat y_i)^2$. The derivative of a square is “two times the expression”, and the derivative of $y_i - \hat y_i$ with respect to $\hat y_i$ is $-1$ (the chain rule). Hence the sign: $(\hat y_i - y_i)$, not $(y_i - \hat y_i)$.

The derivative of MAE

$$\frac{\partial\,\mathrm{MAE}}{\partial \hat y_i} = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$$
$$\operatorname{sign}(x)=\begin{cases}\phantom{-}1, & x>0\\\phantom{-}0, & x=0\\-1, & x<0\end{cases}$$

The derivative of $|x|$ is $1$ to the right of zero and $-1$ to the left. At $x = 0$ itself the absolute value has a kink and no derivative; in practice we take $0$, one of the admissible values (a subgradient), as PyTorch and NumPy do.

How to read the sign

Plus means the prediction is too high and descent will lower it; minus means it is too low and descent will raise it; zero means the object stays put. A gradient descent step: $\hat y_i \leftarrow \hat y_i - \eta\, g_i$, where $\eta$ is the learning rate. The key difference between the metrics: with MSE the strength of the signal is proportional to the error, with MAE it is always $\tfrac{1}{n}$.

The board example: the gradient and one descent step

For $y = (3,\ 5,\ 8)$ and $\hat y = (2.5,\ 5,\ 10)$ the derivatives with respect to each prediction are:

$i$$y_i$$\hat y_i$$g_i$ for MAE$g_i$ for MSE
132.5≈ −0.33≈ −0.33
25500
3810≈ 0.33≈ 1.33

Let us take one descent step directly on the predictions with $\eta = 0.3$, first along the MSE gradient, then along the MAE gradient:

$$\text{MSE:}\quad \begin{aligned}\hat y_{1} &\leftarrow 2.5 - 0.3\cdot (-\tfrac{1}{3}) = 2.6\\\hat y_{2} &\leftarrow 5 - 0.3\cdot 0 = 5\\\hat y_{3} &\leftarrow 10 - 0.3\cdot (\tfrac{4}{3}) = 9.6\end{aligned}$$
$$\text{MAE:}\quad \begin{aligned}\hat y_{1} &\leftarrow 2.5 - 0.3\cdot (-\tfrac{1}{3}) = 2.6\\\hat y_{2} &\leftarrow 5 - 0.3\cdot 0 = 5\\\hat y_{3} &\leftarrow 10 - 0.3\cdot (\tfrac{1}{3}) = 9.9\end{aligned}$$

Both metrics went down: MSE ≈ 1.42 → ≈ 0.91, MAE ≈ 0.83 → ≈ 0.77. MSE moved the third prediction the most, the one with the largest error.

In a real network the predictions are not moved directly; the weights are. The gradient with respect to the predictions is the signal that travels back to the weights; how exactly is shown in the section “From the prediction to the weights”. In every example below the derivatives are written out in the “Derivatives for backpropagation” block.

The example from the board

Let us start with the example solved in class: three objects and three predictions. Each step below is exactly what was written on the board, only in more detail.

Example 1★☆☆

The example from the board

$y = (3,\ 5,\ 8)$,   $\hat y = (2.5,\ 5,\ 10)$. Find MAE and MSE.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 3-2.5\rvert = \lvert 0.5\rvert = 0.5\\\lvert e_{2}\rvert &= \lvert 5-5\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 8-10\rvert = \lvert -2\rvert = 2\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 0.5 + 0 + 2 = 2.5$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 2.5 = \frac{2.5}{3} \approx 0.83$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (3-2.5)^2 = 0.5^2 = 0.25\\e_{2}^2 &= (5-5)^2 = 0^2 = 0\\e_{3}^2 &= (8-10)^2 = (-2)^2 = 4\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 0.25 + 0 + 4 = 4.25$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 4.25 = \frac{4.25}{3} \approx 1.42$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
132.50.50.50.25
255000
3810−224
Sum2.54.25
Mean (÷ 3)≈ 0.83≈ 1.42
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(2.5-3)}{3} \approx -0.33\\g_{2} &= \frac{\operatorname{sign}(5-5)}{3} = 0\\g_{3} &= \frac{\operatorname{sign}(10-8)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(2.5-3)}{3} \approx -0.33\\g_{2} &= \frac{2\cdot(5-5)}{3} = 0\\g_{3} &= \frac{2\cdot(10-8)}{3} \approx 1.33\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (-0.33,\ 0,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (-0.33,\ 0,\ 1.33)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 1.33 in absolute value because it has the largest error (2).

What the derivatives show

Object 3 (error 2) gets a signal of 1.33 from MSE, four times stronger than object 1 (0.33) with an error of 0.5. With MAE both signals have the same absolute value, 0.33: the absolute value does not tell “a little” from “a lot”.

Answer: $\mathrm{MAE} \approx 0.83$,   $\mathrm{MSE} \approx 1.42$

What the example shows

Two of the three errors are small and one equals 2. MAE ≈ 0.83: “on average we are off by less than one”. MSE ≈ 1.42 is larger than MAE: the square of the error 2 is 4, and it pulls the average up.

Warm-up

Simple sets of numbers to make the recipe a habit. The first problem is open; try the rest yourself.

Example 2★☆☆

A perfect prediction

$y = (2,\ 4,\ 6)$,   $\hat y = (2,\ 4,\ 6)$. Find MAE and MSE.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 2-2\rvert = \lvert 0\rvert = 0\\\lvert e_{2}\rvert &= \lvert 4-4\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 6-6\rvert = \lvert 0\rvert = 0\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 0 + 0 + 0 = 0$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 0 = \frac{0}{3} = 0$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (2-2)^2 = 0^2 = 0\\e_{2}^2 &= (4-4)^2 = 0^2 = 0\\e_{3}^2 &= (6-6)^2 = 0^2 = 0\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 0 + 0 + 0 = 0$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 0 = \frac{0}{3} = 0$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
122000
244000
366000
Sum00
Mean (÷ 3)00
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(2-2)}{3} = 0\\g_{2} &= \frac{\operatorname{sign}(4-4)}{3} = 0\\g_{3} &= \frac{\operatorname{sign}(6-6)}{3} = 0\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(2-2)}{3} = 0\\g_{2} &= \frac{2\cdot(4-4)}{3} = 0\\g_{3} &= \frac{2\cdot(6-6)}{3} = 0\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = (0,\ 0,\ 0)\\[4pt]\nabla\mathrm{MSE} = (0,\ 0,\ 0)\end{gathered}$$

All derivatives are zero: the prediction is already perfect, and backpropagation has nothing to fix.

Answer: $\mathrm{MAE} = 0$,   $\mathrm{MSE} = 0$

What the example shows

All errors are zero, so MAE = 0 and MSE = 0. Neither metric can go below zero: absolute values and squares are never negative.

Example 3★☆☆

An error at a single point

$y = (1,\ 2,\ 3,\ 4)$,   $\hat y = (1,\ 2,\ 3,\ 6)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 1-1\rvert = \lvert 0\rvert = 0\\\lvert e_{2}\rvert &= \lvert 2-2\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 3-3\rvert = \lvert 0\rvert = 0\\\lvert e_{4}\rvert &= \lvert 4-6\rvert = \lvert -2\rvert = 2\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{4}\lvert e_i\rvert = 0 + 0 + 0 + 2 = 2$$

3. MAE: divide the sum by $n = 4$:

$$\mathrm{MAE} = \frac{1}{4}\cdot 2 = \frac{2}{4} = 0.5$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (1-1)^2 = 0^2 = 0\\e_{2}^2 &= (2-2)^2 = 0^2 = 0\\e_{3}^2 &= (3-3)^2 = 0^2 = 0\\e_{4}^2 &= (4-6)^2 = (-2)^2 = 4\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{4}e_i^2 = 0 + 0 + 0 + 4 = 4$$

6. MSE: divide the sum by $n = 4$:

$$\mathrm{MSE} = \frac{1}{4}\cdot 4 = \frac{4}{4} = 1$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
111000
222000
333000
446−224
Sum24
Mean (÷ 4)0.51
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(1-1)}{4} = 0\\g_{2} &= \frac{\operatorname{sign}(2-2)}{4} = 0\\g_{3} &= \frac{\operatorname{sign}(3-3)}{4} = 0\\g_{4} &= \frac{\operatorname{sign}(6-4)}{4} = 0.25\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(1-1)}{4} = 0\\g_{2} &= \frac{2\cdot(2-2)}{4} = 0\\g_{3} &= \frac{2\cdot(3-3)}{4} = 0\\g_{4} &= \frac{2\cdot(6-4)}{4} = 1\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0\\0\\0\\0.25\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}0\\0\\0\\1\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.25: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 0.5$,   $\mathrm{MSE} = 1$

What the example shows

There is one error of 2 among four objects: MAE = 2 / 4 = 0.5. In MSE it enters as 2² = 4, so MSE = 1, twice the MAE.

Example 4★☆☆

The same error everywhere

$y = (5,\ 7,\ 9)$,   $\hat y = (6,\ 8,\ 10)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 5-6\rvert = \lvert -1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 7-8\rvert = \lvert -1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 9-10\rvert = \lvert -1\rvert = 1\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 1 + 1 + 1 = 3$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 3 = \frac{3}{3} = 1$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (5-6)^2 = (-1)^2 = 1\\e_{2}^2 &= (7-8)^2 = (-1)^2 = 1\\e_{3}^2 &= (9-10)^2 = (-1)^2 = 1\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 1 + 1 + 1 = 3$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 3 = \frac{3}{3} = 1$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
156−111
278−111
3910−111
Sum33
Mean (÷ 3)11
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(6-5)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(8-7)}{3} \approx 0.33\\g_{3} &= \frac{\operatorname{sign}(10-9)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(6-5)}{3} \approx 0.67\\g_{2} &= \frac{2\cdot(8-7)}{3} \approx 0.67\\g_{3} &= \frac{2\cdot(10-9)}{3} \approx 0.67\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0.33,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (0.67,\ 0.67,\ 0.67)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 1$,   $\mathrm{MSE} = 1$

What the example shows

The model overshoots by 1 everywhere. If all |eᵢ| equal the same a, then MAE = a and MSE = a². Here a = 1, so both metrics equal 1.

Example 5★☆☆

The sign of the error does not matter

$y = (10,\ 20)$,   $\hat y = (12,\ 18)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 10-12\rvert = \lvert -2\rvert = 2\\\lvert e_{2}\rvert &= \lvert 20-18\rvert = \lvert 2\rvert = 2\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{2}\lvert e_i\rvert = 2 + 2 = 4$$

3. MAE: divide the sum by $n = 2$:

$$\mathrm{MAE} = \frac{1}{2}\cdot 4 = \frac{4}{2} = 2$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (10-12)^2 = (-2)^2 = 4\\e_{2}^2 &= (20-18)^2 = 2^2 = 4\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{2}e_i^2 = 4 + 4 = 8$$

6. MSE: divide the sum by $n = 2$:

$$\mathrm{MSE} = \frac{1}{2}\cdot 8 = \frac{8}{2} = 4$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
11012−224
22018224
Sum48
Mean (÷ 2)24
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(12-10)}{2} = 0.5\\g_{2} &= \frac{\operatorname{sign}(18-20)}{2} = -0.5\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(12-10)}{2} = 2\\g_{2} &= \frac{2\cdot(18-20)}{2} = -2\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = (0.5,\ -0.5)\\[4pt]\nabla\mathrm{MSE} = (2,\ -2)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.5: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 2$,   $\mathrm{MSE} = 4$

What the example shows

The first error is −2 (the model overshot), the second +2 (it undershot). The absolute value and the square remove the sign: both errors give 2 in MAE and 4 in MSE. Misses up and down do not cancel out.

Example 6★☆☆

Errors smaller than one

$y = (1,\ 1)$,   $\hat y = (1.5,\ 0.5)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 1-1.5\rvert = \lvert -0.5\rvert = 0.5\\\lvert e_{2}\rvert &= \lvert 1-0.5\rvert = \lvert 0.5\rvert = 0.5\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{2}\lvert e_i\rvert = 0.5 + 0.5 = 1$$

3. MAE: divide the sum by $n = 2$:

$$\mathrm{MAE} = \frac{1}{2}\cdot 1 = \frac{1}{2} = 0.5$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (1-1.5)^2 = (-0.5)^2 = 0.25\\e_{2}^2 &= (1-0.5)^2 = 0.5^2 = 0.25\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{2}e_i^2 = 0.25 + 0.25 = 0.5$$

6. MSE: divide the sum by $n = 2$:

$$\mathrm{MSE} = \frac{1}{2}\cdot 0.5 = \frac{0.5}{2} = 0.25$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
111.5−0.50.50.25
210.50.50.50.25
Sum10.5
Mean (÷ 2)0.50.25
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(1.5-1)}{2} = 0.5\\g_{2} &= \frac{\operatorname{sign}(0.5-1)}{2} = -0.5\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(1.5-1)}{2} = 0.5\\g_{2} &= \frac{2\cdot(0.5-1)}{2} = -0.5\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = (0.5,\ -0.5)\\[4pt]\nabla\mathrm{MSE} = (0.5,\ -0.5)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.5: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 0.5$,   $\mathrm{MSE} = 0.25$

What the example shows

Here MSE = 0.25 is smaller than MAE = 0.5: squaring a number below 1 makes it smaller still (0.5² = 0.25). That is why comparing MAE and MSE by size makes no sense: they have different units and behave differently.

Example 7★☆☆

Five objects: divide by 5

$y = (2,\ 4,\ 6,\ 8,\ 10)$,   $\hat y = (3,\ 3,\ 7,\ 7,\ 10)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 2-3\rvert = \lvert -1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 4-3\rvert = \lvert 1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 6-7\rvert = \lvert -1\rvert = 1\\\lvert e_{4}\rvert &= \lvert 8-7\rvert = \lvert 1\rvert = 1\\\lvert e_{5}\rvert &= \lvert 10-10\rvert = \lvert 0\rvert = 0\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{5}\lvert e_i\rvert = 1 + 1 + 1 + 1 + 0 = 4$$

3. MAE: divide the sum by $n = 5$:

$$\mathrm{MAE} = \frac{1}{5}\cdot 4 = \frac{4}{5} = 0.8$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (2-3)^2 = (-1)^2 = 1\\e_{2}^2 &= (4-3)^2 = 1^2 = 1\\e_{3}^2 &= (6-7)^2 = (-1)^2 = 1\\e_{4}^2 &= (8-7)^2 = 1^2 = 1\\e_{5}^2 &= (10-10)^2 = 0^2 = 0\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{5}e_i^2 = 1 + 1 + 1 + 1 + 0 = 4$$

6. MSE: divide the sum by $n = 5$:

$$\mathrm{MSE} = \frac{1}{5}\cdot 4 = \frac{4}{5} = 0.8$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
123−111
243111
367−111
487111
51010000
Sum44
Mean (÷ 5)0.80.8
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(3-2)}{5} = 0.2\\g_{2} &= \frac{\operatorname{sign}(3-4)}{5} = -0.2\\g_{3} &= \frac{\operatorname{sign}(7-6)}{5} = 0.2\\g_{4} &= \frac{\operatorname{sign}(7-8)}{5} = -0.2\\g_{5} &= \frac{\operatorname{sign}(10-10)}{5} = 0\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(3-2)}{5} = 0.4\\g_{2} &= \frac{2\cdot(3-4)}{5} = -0.4\\g_{3} &= \frac{2\cdot(7-6)}{5} = 0.4\\g_{4} &= \frac{2\cdot(7-8)}{5} = -0.4\\g_{5} &= \frac{2\cdot(10-10)}{5} = 0\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0.2\\-0.2\\0.2\\-0.2\\0\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}0.4\\-0.4\\0.4\\-0.4\\0\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.2: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 0.8$,   $\mathrm{MSE} = 0.8$

What the example shows

Always divide by the number of objects n, not by the number of nonzero errors: there are four errors but five objects, so MAE = 4 / 5 = 0.8.

Negative numbers, decimals and a single object

The same recipe with trickier numbers: minus signs, decimals and the case of a single object.

Example 8★★☆

Negative numbers: temperature

$y = (-2,\ 0,\ 3)$,   $\hat y = (1,\ -1,\ 3)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert -2-1\rvert = \lvert -3\rvert = 3\\\lvert e_{2}\rvert &= \lvert 0-(-1)\rvert = \lvert 1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 3-3\rvert = \lvert 0\rvert = 0\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 3 + 1 + 0 = 4$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 4 = \frac{4}{3} \approx 1.33$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (-2-1)^2 = (-3)^2 = 9\\e_{2}^2 &= (0-(-1))^2 = 1^2 = 1\\e_{3}^2 &= (3-3)^2 = 0^2 = 0\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 9 + 1 + 0 = 10$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 10 = \frac{10}{3} \approx 3.33$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
1−21−339
20−1111
333000
Sum410
Mean (÷ 3)≈ 1.33≈ 3.33
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(1-(-2))}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(-1-0)}{3} \approx -0.33\\g_{3} &= \frac{\operatorname{sign}(3-3)}{3} = 0\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(1-(-2))}{3} = 2\\g_{2} &= \frac{2\cdot(-1-0)}{3} \approx -0.67\\g_{3} &= \frac{2\cdot(3-3)}{3} = 0\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ -0.33,\ 0)\\[4pt]\nabla\mathrm{MSE} \approx (2,\ -0.67,\ 0)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 1 the hardest: its signal is 2 in absolute value because it has the largest error (3).

Answer: $\mathrm{MAE} \approx 1.33$,   $\mathrm{MSE} \approx 3.33$

What the example shows

The key is careful subtraction: 0 − (−1) = 0 + 1 = 1, and (−3)² = 9, not −9. Absolute values and squares are never negative.

Example 9★★☆

Decimals

$y = (1.2,\ 2.5,\ 3.1)$,   $\hat y = (1,\ 2.9,\ 3.1)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 1.2-1\rvert = \lvert 0.2\rvert = 0.2\\\lvert e_{2}\rvert &= \lvert 2.5-2.9\rvert = \lvert -0.4\rvert = 0.4\\\lvert e_{3}\rvert &= \lvert 3.1-3.1\rvert = \lvert 0\rvert = 0\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 0.2 + 0.4 + 0 = 0.6$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 0.6 = \frac{0.6}{3} = 0.2$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (1.2-1)^2 = 0.2^2 = 0.04\\e_{2}^2 &= (2.5-2.9)^2 = (-0.4)^2 = 0.16\\e_{3}^2 &= (3.1-3.1)^2 = 0^2 = 0\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 0.04 + 0.16 + 0 = 0.2$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 0.2 = \frac{0.2}{3} \approx 0.0667$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
11.210.20.20.04
22.52.9−0.40.40.16
33.13.1000
Sum0.60.2
Mean (÷ 3)0.2≈ 0.0667
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(1-1.2)}{3} \approx -0.33\\g_{2} &= \frac{\operatorname{sign}(2.9-2.5)}{3} \approx 0.33\\g_{3} &= \frac{\operatorname{sign}(3.1-3.1)}{3} = 0\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(1-1.2)}{3} \approx -0.13\\g_{2} &= \frac{2\cdot(2.9-2.5)}{3} \approx 0.27\\g_{3} &= \frac{2\cdot(3.1-3.1)}{3} = 0\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (-0.33,\ 0.33,\ 0)\\[4pt]\nabla\mathrm{MSE} \approx (-0.13,\ 0.27,\ 0)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 2 the hardest: its signal is 0.27 in absolute value because it has the largest error (0.4).

Answer: $\mathrm{MAE} = 0.2$,   $\mathrm{MSE} \approx 0.0667$

What the example shows

Decimals work the same way, just be careful with squares: 0.2² = 0.04 and 0.4² = 0.16. MSE ≈ 0.0667 is much smaller than MAE = 0.2 because all errors are below 1.

Example 10★☆☆

One object, n = 1

$y = (7)$,   $\hat y = (4)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 7-4\rvert = \lvert 3\rvert = 3\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{1}\lvert e_i\rvert = 3$$

3. MAE: divide the sum by $n = 1$:

$$\mathrm{MAE} = \frac{1}{1}\cdot 3 = 3$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (7-4)^2 = 3^2 = 9\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{1}e_i^2 = 9$$

6. MSE: divide the sum by $n = 1$:

$$\mathrm{MSE} = \frac{1}{1}\cdot 9 = 9$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
174339
Sum39
Mean (÷ 1)39
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(4-7)}{1} = -1\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(4-7)}{1} = -6\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = (-1)\\[4pt]\nabla\mathrm{MSE} = (-6)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±1: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 3$,   $\mathrm{MSE} = 9$

What the example shows

With n = 1 there is nothing to average: MAE is just the absolute error (3) and MSE is its square (9).

How MAE differs from MSE

Here the examples are chosen so that the metrics “argue”: the same MAE with different MSE, an outlier, doubled errors, the mean versus the median.

Example 11★★☆

Same MAE, different MSE

$y = (3,\ 5,\ 7,\ 9)$. Model A: $\hat y = (4,\ 6,\ 6,\ 8)$; Model B: $\hat y = (3,\ 5,\ 7,\ 5)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

Model A

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 3-4\rvert = \lvert -1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 5-6\rvert = \lvert -1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 7-6\rvert = \lvert 1\rvert = 1\\\lvert e_{4}\rvert &= \lvert 9-8\rvert = \lvert 1\rvert = 1\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{4}\lvert e_i\rvert = 1 + 1 + 1 + 1 = 4$$

3. MAE: divide the sum by $n = 4$:

$$\mathrm{MAE} = \frac{1}{4}\cdot 4 = \frac{4}{4} = 1$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (3-4)^2 = (-1)^2 = 1\\e_{2}^2 &= (5-6)^2 = (-1)^2 = 1\\e_{3}^2 &= (7-6)^2 = 1^2 = 1\\e_{4}^2 &= (9-8)^2 = 1^2 = 1\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{4}e_i^2 = 1 + 1 + 1 + 1 = 4$$

6. MSE: divide the sum by $n = 4$:

$$\mathrm{MSE} = \frac{1}{4}\cdot 4 = \frac{4}{4} = 1$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
134−111
256−111
376111
498111
Sum44
Mean (÷ 4)11
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(4-3)}{4} = 0.25\\g_{2} &= \frac{\operatorname{sign}(6-5)}{4} = 0.25\\g_{3} &= \frac{\operatorname{sign}(6-7)}{4} = -0.25\\g_{4} &= \frac{\operatorname{sign}(8-9)}{4} = -0.25\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(4-3)}{4} = 0.5\\g_{2} &= \frac{2\cdot(6-5)}{4} = 0.5\\g_{3} &= \frac{2\cdot(6-7)}{4} = -0.5\\g_{4} &= \frac{2\cdot(8-9)}{4} = -0.5\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0.25\\0.25\\-0.25\\-0.25\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}0.5\\0.5\\-0.5\\-0.5\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.25: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Model B

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 3-3\rvert = \lvert 0\rvert = 0\\\lvert e_{2}\rvert &= \lvert 5-5\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 7-7\rvert = \lvert 0\rvert = 0\\\lvert e_{4}\rvert &= \lvert 9-5\rvert = \lvert 4\rvert = 4\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{4}\lvert e_i\rvert = 0 + 0 + 0 + 4 = 4$$

3. MAE: divide the sum by $n = 4$:

$$\mathrm{MAE} = \frac{1}{4}\cdot 4 = \frac{4}{4} = 1$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (3-3)^2 = 0^2 = 0\\e_{2}^2 &= (5-5)^2 = 0^2 = 0\\e_{3}^2 &= (7-7)^2 = 0^2 = 0\\e_{4}^2 &= (9-5)^2 = 4^2 = 16\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{4}e_i^2 = 0 + 0 + 0 + 16 = 16$$

6. MSE: divide the sum by $n = 4$:

$$\mathrm{MSE} = \frac{1}{4}\cdot 16 = \frac{16}{4} = 4$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
133000
255000
377000
4954416
Sum416
Mean (÷ 4)14
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(3-3)}{4} = 0\\g_{2} &= \frac{\operatorname{sign}(5-5)}{4} = 0\\g_{3} &= \frac{\operatorname{sign}(7-7)}{4} = 0\\g_{4} &= \frac{\operatorname{sign}(5-9)}{4} = -0.25\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(3-3)}{4} = 0\\g_{2} &= \frac{2\cdot(5-5)}{4} = 0\\g_{3} &= \frac{2\cdot(7-7)}{4} = 0\\g_{4} &= \frac{2\cdot(5-9)}{4} = -2\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0\\0\\0\\-0.25\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}0\\0\\0\\-2\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.25: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

What the derivatives show

The MSE gradients show the same difference: model A has four signals of 0.5 each, while model B puts its whole signal of −2 on a single point. With MAE that point's signal is only −0.25, like any other error.

Comparison

MAEMSE
Model A11
Model B14

Answer: Model A: $\mathrm{MAE} = 1$,   $\mathrm{MSE} = 1$; Model B: $\mathrm{MAE} = 1$,   $\mathrm{MSE} = 4$

What the example shows

Both models have MAE = 1. But A is slightly off everywhere, while B is exact at three points and misses one badly. MSE tells them apart: 1 for A versus 4 for B. MSE “punishes” large misses.

Example 12★★☆

An outlier

$y = (2,\ 3,\ 4,\ 20)$,   $\hat y = (3,\ 3,\ 3,\ 4)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 2-3\rvert = \lvert -1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 3-3\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 4-3\rvert = \lvert 1\rvert = 1\\\lvert e_{4}\rvert &= \lvert 20-4\rvert = \lvert 16\rvert = 16\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{4}\lvert e_i\rvert = 1 + 0 + 1 + 16 = 18$$

3. MAE: divide the sum by $n = 4$:

$$\mathrm{MAE} = \frac{1}{4}\cdot 18 = \frac{18}{4} = 4.5$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (2-3)^2 = (-1)^2 = 1\\e_{2}^2 &= (3-3)^2 = 0^2 = 0\\e_{3}^2 &= (4-3)^2 = 1^2 = 1\\e_{4}^2 &= (20-4)^2 = 16^2 = 256\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{4}e_i^2 = 1 + 0 + 1 + 256 = 258$$

6. MSE: divide the sum by $n = 4$:

$$\mathrm{MSE} = \frac{1}{4}\cdot 258 = \frac{258}{4} = 64.5$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
123−111
233000
343111
42041616256
Sum18258
Mean (÷ 4)4.564.5
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(3-2)}{4} = 0.25\\g_{2} &= \frac{\operatorname{sign}(3-3)}{4} = 0\\g_{3} &= \frac{\operatorname{sign}(3-4)}{4} = -0.25\\g_{4} &= \frac{\operatorname{sign}(4-20)}{4} = -0.25\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(3-2)}{4} = 0.5\\g_{2} &= \frac{2\cdot(3-3)}{4} = 0\\g_{3} &= \frac{2\cdot(3-4)}{4} = -0.5\\g_{4} &= \frac{2\cdot(4-20)}{4} = -8\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0.25\\0\\-0.25\\-0.25\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}0.5\\0\\-0.5\\-8\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.25: only the direction of the error matters. MSE pulls object 4 the hardest: its signal is 8 in absolute value because it has the largest error (16).

What the derivatives show

When training with MSE, the outlier gets a signal of −8, 16 times stronger than the other points (±0.5), and drags the model towards itself. With MAE its signal is −0.25, the same as an ordinary error. That is why, with outliers in the data, models are trained with MAE or its smooth relative, Huber.

Answer: $\mathrm{MAE} = 4.5$,   $\mathrm{MSE} = 64.5$

What the example shows

One point (y = 20) is far away. In MAE its error of 16 is 16 out of the sum of 18; in MSE it is 256 out of 258, almost the whole metric from a single point. MSE is very sensitive to outliers; MAE is much calmer.

Example 13★★☆

Errors twice as large

$y = (1,\ 2,\ 3)$. Model A: $\hat y = (2,\ 3,\ 4)$; Model B: $\hat y = (3,\ 4,\ 5)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

Model A

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 1-2\rvert = \lvert -1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 2-3\rvert = \lvert -1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 3-4\rvert = \lvert -1\rvert = 1\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 1 + 1 + 1 = 3$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 3 = \frac{3}{3} = 1$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (1-2)^2 = (-1)^2 = 1\\e_{2}^2 &= (2-3)^2 = (-1)^2 = 1\\e_{3}^2 &= (3-4)^2 = (-1)^2 = 1\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 1 + 1 + 1 = 3$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 3 = \frac{3}{3} = 1$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
112−111
223−111
334−111
Sum33
Mean (÷ 3)11
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(2-1)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(3-2)}{3} \approx 0.33\\g_{3} &= \frac{\operatorname{sign}(4-3)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(2-1)}{3} \approx 0.67\\g_{2} &= \frac{2\cdot(3-2)}{3} \approx 0.67\\g_{3} &= \frac{2\cdot(4-3)}{3} \approx 0.67\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0.33,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (0.67,\ 0.67,\ 0.67)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Model B

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 1-3\rvert = \lvert -2\rvert = 2\\\lvert e_{2}\rvert &= \lvert 2-4\rvert = \lvert -2\rvert = 2\\\lvert e_{3}\rvert &= \lvert 3-5\rvert = \lvert -2\rvert = 2\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 2 + 2 + 2 = 6$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 6 = \frac{6}{3} = 2$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (1-3)^2 = (-2)^2 = 4\\e_{2}^2 &= (2-4)^2 = (-2)^2 = 4\\e_{3}^2 &= (3-5)^2 = (-2)^2 = 4\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 4 + 4 + 4 = 12$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 12 = \frac{12}{3} = 4$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
113−224
224−224
335−224
Sum612
Mean (÷ 3)24
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(3-1)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(4-2)}{3} \approx 0.33\\g_{3} &= \frac{\operatorname{sign}(5-3)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(3-1)}{3} \approx 1.33\\g_{2} &= \frac{2\cdot(4-2)}{3} \approx 1.33\\g_{3} &= \frac{2\cdot(5-3)}{3} \approx 1.33\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0.33,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (1.33,\ 1.33,\ 1.33)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

What the derivatives show

Doubling the errors doubles the MSE gradient as well (0.67 → 1.33), while the MAE gradient stays the same (0.33): it depends only on the sign of the error.

Comparison

MAEMSE
Model A11
Model B24

Answer: Model A: $\mathrm{MAE} = 1$,   $\mathrm{MSE} = 1$; Model B: $\mathrm{MAE} = 2$,   $\mathrm{MSE} = 4$

What the example shows

Model B errs twice as much as A (by 2 instead of 1). MAE doubles as well (1 → 2), while MSE quadruples (1 → 4): the square of a doubled number is four times larger.

Example 14★★★

One number for everyone: mean or median?

$y = (2,\ 4,\ 9)$. Model A: $\hat y = (5,\ 5,\ 5)$; Model B: $\hat y = (4,\ 4,\ 4)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

Model A

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 2-5\rvert = \lvert -3\rvert = 3\\\lvert e_{2}\rvert &= \lvert 4-5\rvert = \lvert -1\rvert = 1\\\lvert e_{3}\rvert &= \lvert 9-5\rvert = \lvert 4\rvert = 4\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 3 + 1 + 4 = 8$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 8 = \frac{8}{3} \approx 2.67$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (2-5)^2 = (-3)^2 = 9\\e_{2}^2 &= (4-5)^2 = (-1)^2 = 1\\e_{3}^2 &= (9-5)^2 = 4^2 = 16\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 9 + 1 + 16 = 26$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 26 = \frac{26}{3} \approx 8.67$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
125−339
245−111
3954416
Sum826
Mean (÷ 3)≈ 2.67≈ 8.67
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(5-2)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(5-4)}{3} \approx 0.33\\g_{3} &= \frac{\operatorname{sign}(5-9)}{3} \approx -0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(5-2)}{3} = 2\\g_{2} &= \frac{2\cdot(5-4)}{3} \approx 0.67\\g_{3} &= \frac{2\cdot(5-9)}{3} \approx -2.67\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0.33,\ -0.33)\\[4pt]\nabla\mathrm{MSE} \approx (2,\ 0.67,\ -2.67)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 2.67 in absolute value because it has the largest error (4).

Model B

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 2-4\rvert = \lvert -2\rvert = 2\\\lvert e_{2}\rvert &= \lvert 4-4\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 9-4\rvert = \lvert 5\rvert = 5\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 2 + 0 + 5 = 7$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 7 = \frac{7}{3} \approx 2.33$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (2-4)^2 = (-2)^2 = 4\\e_{2}^2 &= (4-4)^2 = 0^2 = 0\\e_{3}^2 &= (9-4)^2 = 5^2 = 25\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 4 + 0 + 25 = 29$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 29 = \frac{29}{3} \approx 9.67$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
124−224
244000
3945525
Sum729
Mean (÷ 3)≈ 2.33≈ 9.67
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(4-2)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(4-4)}{3} = 0\\g_{3} &= \frac{\operatorname{sign}(4-9)}{3} \approx -0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(4-2)}{3} \approx 1.33\\g_{2} &= \frac{2\cdot(4-4)}{3} = 0\\g_{3} &= \frac{2\cdot(4-9)}{3} \approx -3.33\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0,\ -0.33)\\[4pt]\nabla\mathrm{MSE} \approx (1.33,\ 0,\ -3.33)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 3.33 in absolute value because it has the largest error (5).

What the derivatives show

If the model is a single number $c$, the derivative with respect to it is the sum of the gradient components. For MSE it is zero at $c = 5$ (the mean): $2 + 0.67 - 2.67 = 0$; for MAE at $c = 4$ (the median): $0.33 + 0 - 0.33 = 0$. A zero derivative is the minimum: that is why the best constants are the mean and the median.

Comparison

MAEMSE
Model A≈ 2.67≈ 8.67
Model B≈ 2.33≈ 9.67

Answer: Model A: $\mathrm{MAE} \approx 2.67$,   $\mathrm{MSE} \approx 8.67$; Model B: $\mathrm{MAE} \approx 2.33$,   $\mathrm{MSE} \approx 9.67$

What the example shows

Model A always answers with the mean (2 + 4 + 9) / 3 = 5, model B with the median 4. By MAE the median wins (≈ 2.33 versus ≈ 2.67), by MSE the mean wins (≈ 8.67 versus ≈ 9.67). This is a general rule: among all constants, the median minimizes MAE and the mean minimizes MSE.

Example 15★★☆

The signed mean error is misleading

$y = (10,\ 10,\ 10,\ 10)$,   $\hat y = (15,\ 5,\ 15,\ 5)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 10-15\rvert = \lvert -5\rvert = 5\\\lvert e_{2}\rvert &= \lvert 10-5\rvert = \lvert 5\rvert = 5\\\lvert e_{3}\rvert &= \lvert 10-15\rvert = \lvert -5\rvert = 5\\\lvert e_{4}\rvert &= \lvert 10-5\rvert = \lvert 5\rvert = 5\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{4}\lvert e_i\rvert = 5 + 5 + 5 + 5 = 20$$

3. MAE: divide the sum by $n = 4$:

$$\mathrm{MAE} = \frac{1}{4}\cdot 20 = \frac{20}{4} = 5$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (10-15)^2 = (-5)^2 = 25\\e_{2}^2 &= (10-5)^2 = 5^2 = 25\\e_{3}^2 &= (10-15)^2 = (-5)^2 = 25\\e_{4}^2 &= (10-5)^2 = 5^2 = 25\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{4}e_i^2 = 25 + 25 + 25 + 25 = 100$$

6. MSE: divide the sum by $n = 4$:

$$\mathrm{MSE} = \frac{1}{4}\cdot 100 = \frac{100}{4} = 25$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
11015−5525
21055525
31015−5525
41055525
Sum20100
Mean (÷ 4)525
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(15-10)}{4} = 0.25\\g_{2} &= \frac{\operatorname{sign}(5-10)}{4} = -0.25\\g_{3} &= \frac{\operatorname{sign}(15-10)}{4} = 0.25\\g_{4} &= \frac{\operatorname{sign}(5-10)}{4} = -0.25\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(15-10)}{4} = 2.5\\g_{2} &= \frac{2\cdot(5-10)}{4} = -2.5\\g_{3} &= \frac{2\cdot(15-10)}{4} = 2.5\\g_{4} &= \frac{2\cdot(5-10)}{4} = -2.5\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} = \begin{pmatrix}0.25\\-0.25\\0.25\\-0.25\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} = \begin{pmatrix}2.5\\-2.5\\2.5\\-2.5\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.25: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Answer: $\mathrm{MAE} = 5$,   $\mathrm{MSE} = 25$

What the example shows

The errors −5, +5, −5, +5 add up to 0. “On average the model is right”? No: every prediction is off by 5. That is why metrics take the absolute value or the square: MAE = 5, MSE = 25.

Real-life cases

Units of measurement, grades, delivery times, a reverse problem and the main practical question: which model to choose.

Example 16★★☆

Prices in thousands: units

$y = (100,\ 150,\ 200)$,   $\hat y = (110,\ 140,\ 230)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 100-110\rvert = \lvert -10\rvert = 10\\\lvert e_{2}\rvert &= \lvert 150-140\rvert = \lvert 10\rvert = 10\\\lvert e_{3}\rvert &= \lvert 200-230\rvert = \lvert -30\rvert = 30\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 10 + 10 + 30 = 50$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 50 = \frac{50}{3} \approx 16.67$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (100-110)^2 = (-10)^2 = 100\\e_{2}^2 &= (150-140)^2 = 10^2 = 100\\e_{3}^2 &= (200-230)^2 = (-30)^2 = 900\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 100 + 100 + 900 = 1100$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 1100 = \frac{1100}{3} \approx 366.67$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
1100110−1010100
21501401010100
3200230−3030900
Sum501100
Mean (÷ 3)≈ 16.67≈ 366.67
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(110-100)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(140-150)}{3} \approx -0.33\\g_{3} &= \frac{\operatorname{sign}(230-200)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(110-100)}{3} \approx 6.67\\g_{2} &= \frac{2\cdot(140-150)}{3} \approx -6.67\\g_{3} &= \frac{2\cdot(230-200)}{3} = 20\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ -0.33,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (6.67,\ -6.67,\ 20)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 20 in absolute value because it has the largest error (30).

Answer: $\mathrm{MAE} \approx 16.67$,   $\mathrm{MSE} \approx 366.67$

What the example shows

The prices are in thousands of dollars. MAE ≈ 16.67 is in thousands too: “on average we are off by 16.7 thousand”. MSE ≈ 366.67 is measured in “thousands squared”, which is hard to explain to a client. That is why RMSE = √MSE ≈ 19.15 thousand is often reported next to it.

Example 17★★☆

The grades of six students

$y = (5,\ 4,\ 3,\ 5,\ 4,\ 2)$,   $\hat y = (4,\ 4,\ 4,\ 4,\ 4,\ 4)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 5-4\rvert = \lvert 1\rvert = 1\\\lvert e_{2}\rvert &= \lvert 4-4\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 3-4\rvert = \lvert -1\rvert = 1\\\lvert e_{4}\rvert &= \lvert 5-4\rvert = \lvert 1\rvert = 1\\\lvert e_{5}\rvert &= \lvert 4-4\rvert = \lvert 0\rvert = 0\\\lvert e_{6}\rvert &= \lvert 2-4\rvert = \lvert -2\rvert = 2\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{6}\lvert e_i\rvert = 1 + 0 + 1 + 1 + 0 + 2 = 5$$

3. MAE: divide the sum by $n = 6$:

$$\mathrm{MAE} = \frac{1}{6}\cdot 5 = \frac{5}{6} \approx 0.83$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (5-4)^2 = 1^2 = 1\\e_{2}^2 &= (4-4)^2 = 0^2 = 0\\e_{3}^2 &= (3-4)^2 = (-1)^2 = 1\\e_{4}^2 &= (5-4)^2 = 1^2 = 1\\e_{5}^2 &= (4-4)^2 = 0^2 = 0\\e_{6}^2 &= (2-4)^2 = (-2)^2 = 4\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{6}e_i^2 = 1 + 0 + 1 + 1 + 0 + 4 = 7$$

6. MSE: divide the sum by $n = 6$:

$$\mathrm{MSE} = \frac{1}{6}\cdot 7 = \frac{7}{6} \approx 1.17$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
154111
244000
334−111
454111
544000
624−224
Sum57
Mean (÷ 6)≈ 0.83≈ 1.17
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(4-5)}{6} \approx -0.17\\g_{2} &= \frac{\operatorname{sign}(4-4)}{6} = 0\\g_{3} &= \frac{\operatorname{sign}(4-3)}{6} \approx 0.17\\g_{4} &= \frac{\operatorname{sign}(4-5)}{6} \approx -0.17\\g_{5} &= \frac{\operatorname{sign}(4-4)}{6} = 0\\g_{6} &= \frac{\operatorname{sign}(4-2)}{6} \approx 0.17\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(4-5)}{6} \approx -0.33\\g_{2} &= \frac{2\cdot(4-4)}{6} = 0\\g_{3} &= \frac{2\cdot(4-3)}{6} \approx 0.33\\g_{4} &= \frac{2\cdot(4-5)}{6} \approx -0.33\\g_{5} &= \frac{2\cdot(4-4)}{6} = 0\\g_{6} &= \frac{2\cdot(4-2)}{6} \approx 0.67\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx \begin{pmatrix}-0.17\\0\\0.17\\-0.17\\0\\0.17\end{pmatrix}\\[8pt]\nabla\mathrm{MSE} \approx \begin{pmatrix}-0.33\\0\\0.33\\-0.33\\0\\0.67\end{pmatrix}\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.17: only the direction of the error matters. MSE pulls object 6 the hardest: its signal is 0.67 in absolute value because it has the largest error (2).

Answer: $\mathrm{MAE} \approx 0.83$,   $\mathrm{MSE} \approx 1.17$

What the example shows

The model gives everyone a 4. It is off by 1 for three students and by 2 for one: MAE = 5 / 6 ≈ 0.83 points. In MSE the error of 2 contributes 4 out of 7, so MSE ≈ 1.17 is noticeably larger than MAE.

Example 18★★☆

Delivery time in minutes

$y = (30,\ 45,\ 60)$,   $\hat y = (35,\ 45,\ 50)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 30-35\rvert = \lvert -5\rvert = 5\\\lvert e_{2}\rvert &= \lvert 45-45\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 60-50\rvert = \lvert 10\rvert = 10\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 5 + 0 + 10 = 15$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 15 = \frac{15}{3} = 5$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (30-35)^2 = (-5)^2 = 25\\e_{2}^2 &= (45-45)^2 = 0^2 = 0\\e_{3}^2 &= (60-50)^2 = 10^2 = 100\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 25 + 0 + 100 = 125$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 125 = \frac{125}{3} \approx 41.67$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
13035−5525
24545000
360501010100
Sum15125
Mean (÷ 3)5≈ 41.67
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(35-30)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(45-45)}{3} = 0\\g_{3} &= \frac{\operatorname{sign}(50-60)}{3} \approx -0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(35-30)}{3} \approx 3.33\\g_{2} &= \frac{2\cdot(45-45)}{3} = 0\\g_{3} &= \frac{2\cdot(50-60)}{3} \approx -6.67\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ 0,\ -0.33)\\[4pt]\nabla\mathrm{MSE} \approx (3.33,\ 0,\ -6.67)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 6.67 in absolute value because it has the largest error (10).

Answer: $\mathrm{MAE} = 5$,   $\mathrm{MSE} \approx 41.67$

What the example shows

MAE = 5 minutes is a clear answer for a customer: “on average the courier is 5 minutes early or late”. MSE ≈ 41.67 “minutes squared” is convenient for training a model but not for explaining it.

Example 19★★★

The reverse problem: find the prediction

Given: $y = (4,\ 6)$, the first prediction $\hat y_1 = 5$ and $\mathrm{MAE} = 1$. Find the second prediction $\hat y_2$ and the MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

1. Write MAE for $n = 2$ and plug in what is known:

$$\frac{1}{2}\bigl(|4-5| + |6-\hat y_2|\bigr) = 1$$

2. Multiply both sides by 2 and move the known absolute value:

$$\begin{aligned}1 + |6-\hat y_2| &= 2\\|6-\hat y_2| &= 1\end{aligned}$$

3. The absolute value equals 1, so the expression inside is $1$ or $-1$:

$$\begin{aligned}6-\hat y_2 = 1 &\Rightarrow \hat y_2 = 5\\6-\hat y_2 = -1 &\Rightarrow \hat y_2 = 7\end{aligned}$$

4. Check the MSE for both answers: it is the same:

$$\begin{aligned}\hat y_2 = 5:\quad \mathrm{MSE} = \frac{1 + 1}{2} = 1\\\hat y_2 = 7:\quad \mathrm{MSE} = \frac{1 + 1}{2} = 1\end{aligned}$$

Answer: $\hat y_2 = 5$ or $\hat y_2 = 7$; MSE = 1.

What the example shows

A reverse problem is solved with the same recipe: write the formula, plug in what is known and find the unknown. The absolute value gives two answers: the prediction could have missed up or down.

Example 20★★★

Which model is better?

$y = (10,\ 20,\ 30)$. Model A: $\hat y = (12,\ 18,\ 33)$; Model B: $\hat y = (10,\ 20,\ 36)$. Find MAE and MSE.

Try to solve it yourself first, then open the solution.

Solution and answer

Model A

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 10-12\rvert = \lvert -2\rvert = 2\\\lvert e_{2}\rvert &= \lvert 20-18\rvert = \lvert 2\rvert = 2\\\lvert e_{3}\rvert &= \lvert 30-33\rvert = \lvert -3\rvert = 3\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 2 + 2 + 3 = 7$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 7 = \frac{7}{3} \approx 2.33$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (10-12)^2 = (-2)^2 = 4\\e_{2}^2 &= (20-18)^2 = 2^2 = 4\\e_{3}^2 &= (30-33)^2 = (-3)^2 = 9\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 4 + 4 + 9 = 17$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 17 = \frac{17}{3} \approx 5.67$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
11012−224
22018224
33033−339
Sum717
Mean (÷ 3)≈ 2.33≈ 5.67
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(12-10)}{3} \approx 0.33\\g_{2} &= \frac{\operatorname{sign}(18-20)}{3} \approx -0.33\\g_{3} &= \frac{\operatorname{sign}(33-30)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(12-10)}{3} \approx 1.33\\g_{2} &= \frac{2\cdot(18-20)}{3} \approx -1.33\\g_{3} &= \frac{2\cdot(33-30)}{3} = 2\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0.33,\ -0.33,\ 0.33)\\[4pt]\nabla\mathrm{MSE} \approx (1.33,\ -1.33,\ 2)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. MSE pulls object 3 the hardest: its signal is 2 in absolute value because it has the largest error (3).

Model B

1. The error of each object $e_i = y_i - \hat y_i$ and its absolute value:

$$\begin{aligned}\lvert e_{1}\rvert &= \lvert 10-10\rvert = \lvert 0\rvert = 0\\\lvert e_{2}\rvert &= \lvert 20-20\rvert = \lvert 0\rvert = 0\\\lvert e_{3}\rvert &= \lvert 30-36\rvert = \lvert -6\rvert = 6\end{aligned}$$

2. Sum of the absolute values:

$$\sum_{i=1}^{3}\lvert e_i\rvert = 0 + 0 + 6 = 6$$

3. MAE: divide the sum by $n = 3$:

$$\mathrm{MAE} = \frac{1}{3}\cdot 6 = \frac{6}{3} = 2$$

4. The square of each error:

$$\begin{aligned}e_{1}^2 &= (10-10)^2 = 0^2 = 0\\e_{2}^2 &= (20-20)^2 = 0^2 = 0\\e_{3}^2 &= (30-36)^2 = (-6)^2 = 36\end{aligned}$$

5. Sum of the squares:

$$\sum_{i=1}^{3}e_i^2 = 0 + 0 + 36 = 36$$

6. MSE: divide the sum by $n = 3$:

$$\mathrm{MSE} = \frac{1}{3}\cdot 36 = \frac{36}{3} = 12$$
$i$$y_i$$\hat y_i$$e_i$$|e_i|$$e_i^2$
11010000
22020000
33036−6636
Sum636
Mean (÷ 3)212
Derivatives for backpropagation

7. The derivative of MAE with respect to each prediction: $g_i = \frac{1}{n}\operatorname{sign}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{\operatorname{sign}(10-10)}{3} = 0\\g_{2} &= \frac{\operatorname{sign}(20-20)}{3} = 0\\g_{3} &= \frac{\operatorname{sign}(36-30)}{3} \approx 0.33\end{aligned}$$

8. The derivative of MSE with respect to each prediction: $g_i = \frac{2}{n}(\hat y_i - y_i)$:

$$\begin{aligned}g_{1} &= \frac{2\cdot(10-10)}{3} = 0\\g_{2} &= \frac{2\cdot(20-20)}{3} = 0\\g_{3} &= \frac{2\cdot(36-30)}{3} = 4\end{aligned}$$

The gradient vectors with respect to the predictions ($\nabla = \nabla_{\hat y}$):

$$\begin{gathered}\nabla\mathrm{MAE} \approx (0,\ 0,\ 0.33)\\[4pt]\nabla\mathrm{MSE} = (0,\ 0,\ 4)\end{gathered}$$

MAE sends every miss a signal of the same strength $\pm\tfrac{1}{n}$ = ±0.33: only the direction of the error matters. With MSE all nonzero signals have the same absolute value: the errors are the same size.

Comparison

MAEMSE
Model A≈ 2.33≈ 5.67
Model B212

Answer: Model A: $\mathrm{MAE} \approx 2.33$,   $\mathrm{MSE} \approx 5.67$; Model B: $\mathrm{MAE} = 2$,   $\mathrm{MSE} = 12$

What the example shows

By MAE, B is better (2 versus ≈ 2.33): it is exact at two points out of three. By MSE, A is better (≈ 5.67 versus 12): it has no large miss. The answer depends on the task: if rare large errors are dangerous (say, forecasting the load on a bridge), choose by MSE; if the typical error matters, choose by MAE.

From the prediction to the weights: the chain rule

Take the simplest neuron with no activation function: $\hat y_i = w\,x_i + b$. The inputs are $x = (1,\ 2,\ 3)$, the correct answers $y = (3,\ 5,\ 8)$, the current weights $w = 2$, $b = 0$, so the predictions are $\hat y = (2,\ 4,\ 6)$. How does the gradient with respect to the predictions turn into a gradient with respect to the weights?

Each prediction depends on $w$ through the factor $x_i$ and on $b$ with the factor $1$: $\frac{\partial \hat y_i}{\partial w} = x_i$, $\frac{\partial \hat y_i}{\partial b} = 1$. By the chain rule:

$$\begin{aligned}\frac{\partial L}{\partial w} &= \sum_{i=1}^{n} \frac{\partial L}{\partial \hat y_i}\cdot\frac{\partial \hat y_i}{\partial w} = \sum_{i=1}^{n} g_i\,x_i\\ \frac{\partial L}{\partial b} &= \sum_{i=1}^{n} g_i\end{aligned}$$

With MSE

The gradient with respect to the predictions:

$$\nabla_{\hat y}\mathrm{MSE} \approx (-0.67,\ -0.67,\ -1.33)$$

The gradient with respect to the weights:

$$\begin{aligned}\frac{\partial\,\mathrm{MSE}}{\partial w} &= (-\tfrac{2}{3})\cdot 1 + (-\tfrac{2}{3})\cdot 2\\ &\quad + (-\tfrac{4}{3})\cdot 3 = -6\\[6pt]\frac{\partial\,\mathrm{MSE}}{\partial b} &= (-\tfrac{2}{3}) + (-\tfrac{2}{3}) + (-\tfrac{4}{3})\\ &= -\frac{8}{3} \approx -2.67\end{aligned}$$

A descent step with $\eta = 0.1$:

$$\begin{aligned}w &\leftarrow 2 - 0.1\cdot (-6) = 2.6\\b &\leftarrow 0 - 0.1\cdot (-\tfrac{8}{3}) = \tfrac{4}{15} \approx 0.27\end{aligned}$$

The loss after the step: MSE 2 → 0.08.

With MAE

The gradient with respect to the predictions:

$$\nabla_{\hat y}\mathrm{MAE} \approx (-0.33,\ -0.33,\ -0.33)$$

The gradient with respect to the weights:

$$\begin{aligned}\frac{\partial\,\mathrm{MAE}}{\partial w} &= (-\tfrac{1}{3})\cdot 1 + (-\tfrac{1}{3})\cdot 2\\ &\quad + (-\tfrac{1}{3})\cdot 3 = -2\\[6pt]\frac{\partial\,\mathrm{MAE}}{\partial b} &= (-\tfrac{1}{3}) + (-\tfrac{1}{3}) + (-\tfrac{1}{3})\\ &= -1\end{aligned}$$

A descent step with $\eta = 0.1$:

$$\begin{aligned}w &\leftarrow 2 - 0.1\cdot (-2) = 2.2\\b &\leftarrow 0 - 0.1\cdot (-1) = 0.1\end{aligned}$$

The loss after the step: MAE ≈ 1.33 → ≈ 0.83.

With MSE the derivative with respect to $w$ is −6, with MAE it is −2: the third object is off by 2, and MSE “shouts” louder, so the step is larger. This is how one and the same error signal travels back from the loss to the weights; in a deep network the chain is simply longer: the signal is multiplied by the derivative of every layer.

Compute your own example

Enter the correct answers $y$ and the predictions $\hat y$, and the calculator will write out the solution in the same steps, including the derivatives for backpropagation. Separate the numbers with commas, spaces or semicolons; use a point for decimals.

Try:

Common mistakes

  • Forgot the absolute value. The errors −2 and +2 “cancel” to zero, yet the model was wrong both times (examples 5 and 15).
  • Squared a negative number incorrectly. $(-3)^2 = 9$, not $-9$: the square of any number is non-negative (example 8).
  • Divided by the wrong $n$. Divide by the number of objects, not by the number of nonzero errors (example 7).
  • Averaged first and squared afterwards. MSE is the mean of the squares, not the square of the mean: for the errors −5 and +5, MSE = 25, while the “square of the mean error” would be 0.
  • Compared MAE and MSE “by size”. They have different units: with errors below 1, MSE is smaller than MAE; with large errors it is larger (examples 6 and 3).
  • Swapped the order of subtraction. That is harmless: $|y - \hat y| = |\hat y - y|$ and $(y - \hat y)^2 = (\hat y - y)^2$. Still, the signed error is conventionally $y_i - \hat y_i$.
  • Got the sign of the derivative wrong. $\frac{\partial}{\partial \hat y_i}(y_i - \hat y_i)^2 = 2(y_i - \hat y_i)\cdot(-1) = 2(\hat y_i - y_i)$: the inner derivative brings a minus. With the wrong sign, descent goes uphill and the loss grows.
  • Forgot the factor $\tfrac{2}{n}$ or $\tfrac{1}{n}$. The direction does not change, but the step becomes $n$ (or $2n$) times larger, and so does the suitable learning rate.
  • Looking for the derivative of MAE at zero. There is none: the absolute value has a kink. A subgradient is used instead, usually $0$, as in PyTorch.

Summary

MAEMSE
Formula$\tfrac{1}{n}\sum |e_i|$$\tfrac{1}{n}\sum e_i^2$
Unitsthe same as $y$the units of $y$ squared
Large errorscounted proportionallyamplified: error ×2 → contribution ×4
Outliersrobustsensitive
Best constantthe medianthe mean
When to choosethe typical error matters, there are outlierslarge misses are especially dangerous
Derivative w.r.t. $\hat y_i$$-\tfrac{1}{n}\operatorname{sign} e_i$$-\tfrac{2}{n}\,e_i$
Signal strengththe same: $\pm\tfrac{1}{n}$proportional to the error
  • One recipe: errors → absolute values or squares → sum → divide by $n$.
  • The absolute value and the square remove the sign, so misses up and down do not cancel out.
  • MSE “punishes” large errors and MAE does not; the same model can be better by one metric and worse by the other.
  • Backpropagation needs the derivative of the loss with respect to the prediction: $\tfrac{2}{n}(\hat y_i - y_i)$ for MSE and $\tfrac{1}{n}\operatorname{sign}(\hat y_i - y_i)$ for MAE. Then it is multiplied by the derivatives of the layers with the chain rule.
  • The full theory is on the MAE, MSE and RMSE pages of the cheat sheet.

Check yourself

1. $y = (2,\ 4)$, $\hat y = (3,\ 1)$. What is the MAE?
2. For the same numbers, what is the MSE?
3. All errors were tripled. How many times larger does the MSE get?
4. Which single number $c$ minimizes the MAE for the data $(1,\ 2,\ 10)$?
5. The model's errors are $0.1,\ 0.2,\ 0.3$. Which is larger?
6. $y = (2,\ 4)$, $\hat y = (3,\ 1)$. What is the MSE gradient with respect to the predictions?
7. For the same numbers, what is the MAE gradient?
8. The error of one object grew from 1 to 10. How does its MAE gradient component change?

← Materials