MAE: the mean absolute error

Why we need it

A model predicts a flat’s price, a delivery time, demand for a product. The client’s first question is: “How far off is it?” The answer “4 minutes on average” or “120 thousand” is exactly MAE, the mean absolute error. Anyone understands it without a background in statistics, because it is measured in the same units as the quantity itself.

Where it appears:

• a regression quality metric — forecasting demand, travel time, prices, server load; reports put it next to RMSE;
• a loss function — L1Loss in PyTorch; it is chosen when the data contain outliers or when the target is the median rather than the mean;
• computer vision — L1 over pixels in pix2pix and in super-resolution (EDSR), L1 over box coordinates in the DETR detector;
• gradient boosting — loss="absolute_error" in scikit-learn, objective="l1" in LightGBM: leaf values are medians of the residuals;
• robust statistics — least absolute deviations (LAD) regression, quantile regression, the median as an estimate of a “typical” value.

Intuition: the average length of a miss

Draw the truth $y_i$ as points, the predictions $\hat y_i$ as blue markers and the error as a vertical segment between them. MAE is the average length of the orange segments. Direction does not matter: undershooting and overshooting by the same distance cost the same.

Below is the cheat sheet’s running example: $y=(3,\,5,\,8)$, $\hat y=(2.5,\,5,\,10)$. Drag the blue markers. On the lower plot every error $e_i=y_i-\hat y_i$ sits on the “V” of $|e|$; MAE is the average height of these points (the teal line).

The formula, bone by bone

Let us split the formula into four steps and see what each one does.

$y_i$

the true value for object $i$ (a price, a time, demand);

$\hat y_i$

the model’s prediction for the same object;

$e_i=y_i-\hat y_i$

the error (residual): positive means the model underestimated, negative means it overestimated;

$|e_i|$

the absolute value drops the sign so that undershoots and overshoots do not cancel out; $|e|=\sqrt{e^2}$ is a distance on the number line;

$\frac1n\sum$

averaging makes the metric independent of the sample size: MAE on 100 and on 10,000 objects can be compared.

Why can we not do without the absolute value? The mean error without it, $\frac1n\sum e_i$, equals $(0.5+0-2)/3=-0.5$ in our example, and it can be zero for an arbitrarily bad model: errors of $+100$ and $-100$ average to zero. This quantity is called the bias — useful, but a different metric.

The $e_i^2$ column is there for comparison with MSE. In vector form MAE is the $L_1$ norm of the error vector divided by $n$:

Hence the second name of the loss, L1 loss. The distance $\|y-\hat y\|_1$ is called the Manhattan distance: it is how a taxi moves on a grid of blocks — the sum of the shifts along each axis.

Properties and the best constant

• Non-negativity: $\mathrm{MAE}\ge0$, and $\mathrm{MAE}=0$ only if every prediction is exact.
• Units are those of $y$. Scale carries over directly: convert roubles to thousands of roubles and MAE shrinks exactly 1000 times.
• Linear contribution of errors: doubling one error doubles its contribution instead of quadrupling it as in MSE.
• Convexity: the absolute value is convex and a sum of convex functions is convex, so MAE has no “false” local minima — neither in the predictions nor in the parameters of a linear model. But the convexity is not strict: the minimum may be non-unique.
• Piecewise linearity: as a function of the predictions MAE consists of linear pieces glued along kinks where some error is zero. There is no derivative at the kinks.
• Link with RMSE: $\mathrm{MAE}\le\mathrm{RMSE}\le\sqrt n\,\mathrm{MAE}$. In the example: $0.833\le1.190\le1.443$. The larger the ratio RMSE/MAE, the “heavier the tails” of the errors.

The best constant is the median

Let the model be as simple as possible: it answers the same number $c$ for every object. Which $c$ is best under MAE? The function $\mathrm{MAE}(c)=\frac1n\sum|y_i-c|$ is a sum of “V”s with their tips at the data points. Its slope anywhere between the data points is

While fewer than half the points lie to the left of $c$, the slope is negative and it pays to move $c$ to the right; once more than half lie to the left, the slope is positive. The minimum is where the points split in half, that is, at the median.

The derivative: the sign of the error and the subgradient

The derivative of the absolute value is the sign: $|e|'=\operatorname{sign}(e)$ for $e\ne0$. At zero the “V” has a kink and there is no ordinary derivative. Convex analysis offers a substitute — the subgradient.

A subgradient of a convex function $f$ at a point $x$ is any number $g$ such that the line $f(x)+g\,(z-x)$ never rises above the graph: $f(z)\ge f(x)+g\,(z-x)$ for all $z$. For $|e|$ at zero these are all $g$ with $|g|\le1$: any line through the origin with slope between $-1$ and $1$ can be slid under the “V”. The set of subgradients is called the subdifferential.

Robustness to outliers

An outlier is an object with an anomalous $y$: a typo, a sensor glitch, a rare event. What matters for a loss is how hard such an object pulls the model. For MSE the pull is proportional to the error ($2e/n$); for MAE it is bounded: $1/n$, however far away the point lies.

Watch the blue line: while the outlier is below it, you can drag it down as far as you like — the line does not move at all. The sign of its error does not change, and MAE reacts only to signs. The MSE line, on the contrary, follows the outlier without limit.

MAE in parameter space: a polyhedron

Let the model be a line $\hat y=wx+b$ and the data five points with $x=(-2,\,-1,\,0,\,1,\,2)$ and $y=(1.3,\,2,\,3.1,\,4.5,\,7)$. Each pair $(w,\,b)$ gives its own line and its own MAE, so we get a surface over the parameter plane.

Each term $|y_i-wx_i-b|$ is a “gable roof” with its ridge along the line $b=y_i-wx_i$, where the $i$-th error is zero. The sum of five roofs is a polyhedron: flat faces, sharp edges, vertices. At least one minimizer always lies at a vertex where edges cross, i.e. an optimal line can always be chosen to pass through at least two data points.

The two surfaces give different answers: the MAE line passes through the points at $x=-1$ and $x=1$ and almost ignores the “sticking-out” last point $y=7$; the MSE line is steeper to reduce that point’s square. Each is better by its own measure: the MSE line has $\mathrm{MAE}=0.456>0.39$, the MAE line has $\mathrm{MSE}=0.378>0.229$.

MAE in training: subgradient descent

Gradient descent steps against the gradient: $\theta\leftarrow\theta-\eta\,g$. MAE has a peculiarity: near the minimum the gradient does not shrink. It jumps between vectors built from $\pm1/n$, and with a constant step $\eta$ the parameters keep leaping over the bottom.

For convex functions with a bounded subgradient (as MAE has) theory gives an exact recipe: with a step $\eta_t\propto1/\sqrt t$ the best value found converges to the minimum at the rate $O(\ln t/\sqrt t)$ — almost $1/\sqrt t$. With a constant step only a neighbourhood of the optimum is guaranteed: the excess $\mathrm{MAE}-\mathrm{MAE}^*$ stays of order $\eta$. That is why in practice MAE is trained with a learning rate schedule — needed by SGD and by Adam alike, since Adam’s step does not shrink near the bottom either — or the loss is replaced by the smooth Huber loss.

Generalization: quantiles and the probabilistic meaning

The median is the quantile of level $0.5$. Tilt the “V” so that the right branch is steeper than the left one, and the minimum moves to another quantile. This is the pinball loss, or quantile loss:

With $\tau=0.9$ underestimation ($e>0$) is penalized nine times harder than overestimation, and the optimal constant is the $90\%$ quantile. This is how prediction intervals are built: a model for $\tau=0.1$ and a model for $\tau=0.9$ give a band that should contain about $80\%$ of the observations.

For the optimal line the share of points below it is at most $\tau$, and the share of points below or on it is at least $\tau$: $53/60\le0.9\le55/60$. This follows directly from the condition $0\in\partial$ — the same one that gave the median.

MAE as maximum likelihood

Let $y_i=\hat y_i+\varepsilon_i$, where the noise $\varepsilon$ follows the Laplace distribution with density $p(\varepsilon)=\frac{1}{2s}e^{-|\varepsilon|/s}$. Then the negative log-likelihood is

For a fixed $s$, maximizing the likelihood is the same as minimizing MAE, and the estimate of the noise scale is MAE itself: $\hat s=\mathrm{MAE}$. By choosing MAE we implicitly assume noise with heavier tails than the Gaussian. For MSE the same reasoning leads to normal noise — see MSE.

In code

MAE takes one line. The subgradient and training are more interesting. In NumPy:

pythonimport numpy as np

y     = np.array([3.0, 5.0, 8.0])
y_hat = np.array([2.5, 5.0, 10.0])

def mae(y, y_hat):
    return np.mean(np.abs(y - y_hat))

def mae_grad(y, y_hat):
    # subgradient w.r.t. the predictions: sign(ŷ − y) / n, 0 at zero
    return np.sign(y_hat - y) / y.size

print(mae(y, y_hat))        # 0.8333333333333334
print(mae_grad(y, y_hat))   # [-0.33333333  0.          0.33333333]

# the best constant under MAE is the median, not the mean
data = np.array([2.0, 3.0, 5.0, 9.0, 16.0])
print(np.median(data), mae(data, np.median(data)))   # 5.0 4.0
print(data.mean(), mae(data, data.mean()))           # 7.0 4.4

In PyTorch the loss is called L1Loss. Look at the gradient: the second object’s error is exactly zero and the framework takes the subgradient $0$. The second part of the listing trains the line of Figures 5–6 with a decaying step:

pythonimport torch
import torch.nn as nn

y     = torch.tensor([3.0, 5.0, 8.0])
y_hat = torch.tensor([2.5, 5.0, 10.0], requires_grad=True)

loss = nn.L1Loss()(y_hat, y)     # same as (y_hat - y).abs().mean()
loss.backward()
print(loss.item())               # 0.8333333134651184
print(y_hat.grad)                # tensor([-0.3333,  0.0000,  0.3333])

# a linear model under MAE: the step decays as 1/√t
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
t = torch.tensor([1.3, 2.0, 3.1, 4.5, 7.0])
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
opt = torch.optim.SGD([w, b], lr=0.5)
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda k: 1 / (k + 1) ** 0.5)
for step in range(3000):
    opt.zero_grad()
    nn.functional.l1_loss(w * x + b, t).backward()
    opt.step(); sched.step()
print(round(w.item(), 2), round(b.item(), 2))   # 1.25 3.25 — a line through 2 points

scikit-learn has the metric, exact linear regression under MAE (via a linear program) and boosting with MAE:

pythonimport numpy as np
from sklearn.metrics import mean_absolute_error
from sklearn.linear_model import QuantileRegressor
from sklearn.ensemble import HistGradientBoostingRegressor

print(mean_absolute_error([3, 5, 8], [2.5, 5, 10]))   # 0.8333333333333334

# linear regression under MAE is quantile regression with τ = 0.5
X = np.array([[-2.0], [-1.0], [0.0], [1.0], [2.0]])
y = np.array([1.3, 2.0, 3.1, 4.5, 7.0])
lad = QuantileRegressor(quantile=0.5, alpha=0.0).fit(X, y)   # alpha=0: no L1 penalty (the default is 1.0)
print(lad.coef_, lad.intercept_)                             # [1.25] 3.25

# boosting on MAE: leaf values are medians of the residuals
gbr = HistGradientBoostingRegressor(loss="absolute_error")

Common mistakes

1. Forgetting the absolute value: np.mean(y - y_hat) is the bias, not MAE. Errors of opposite sign cancel, and a bad model can score zero.
2. Comparing MAE across tasks. MAE $=5$ for prices in thousands and for temperature in degrees are incomparable numbers. For comparisons across scales there is MAPE, or MAE divided by the MAE of a simple baseline.
3. Training on MSE and reporting MAE (or the other way round). MSE drives the model to the conditional mean, MAE to the median. On skewed data (prices, incomes, waiting times) these are different numbers, and the model that wins on one metric usually loses on the other.
4. A constant step with an MAE loss. The parameters will jitter around the minimum with an amplitude of order $\eta$. You need a step schedule or a smoothed loss.
5. Expecting a unique solution. The minimum of MAE can be a whole interval (an even number of points) or an edge of the polyhedron. Different solvers will return different but equally good answers.
6. Reading MAE as the “typical error” without looking at the tail. MAE $=1$ can hide rare misses of 50. Look at the error distribution, its quantiles and RMSE next to MAE.

Links to other formulas

• MSE: a square instead of the absolute value — smoothness and the mean instead of the median, but sensitivity to outliers.
• RMSE: $\mathrm{RMSE}\ge\mathrm{MAE}$. For normal errors $\mathrm{RMSE}/\mathrm{MAE}=\sqrt{\pi/2}\approx1.253$, for Laplace errors $\sqrt2\approx1.414$; in the running example $1.428$.
• Huber: a square near zero and the absolute value far away — a smooth minimum and robustness to outliers at the same time.
• MAPE: the same MAE, but of the relative errors $|e_i|/|y_i|$.
• SGD and Adam: optimizers that need a decaying step with MAE.

Summary

Check yourself