MAPE: the mean absolute percentage error

Why it is needed

The sales team asks: “how accurate is the forecast?”. The answer “MAE is 37 units” says little: for bread that is excellent, for cars a disaster. The answer “on average we are off by 12%” needs no explanation and reads the same for bread and for cars. That is why MAPE (mean absolute percentage error) is one of the most popular metrics in demand forecasting, finance and energy.

Where it shows up:

• demand and sales forecasting — retail, logistics, purchase planning; it is often the main KPI of a forecasting team;
• finance and economics — the accuracy of revenue, price and macro forecasts;
• energy — grid load forecasting, where the percentage error translates directly into the cost of reserves;
• comparing series of different scales — MAPE is dimensionless, so accuracy can be averaged over thousands of products;
• forecasting competitions — in Makridakis’s M3 competition its “symmetric” variant sMAPE was one of the key metrics, and in M4 methods were ranked by sMAPE together with MASE.

Intuition: the error as a share of the actual

Missing by 1 unit when 3 were sold means being off by a third. Missing by the same unit when 100 were sold is almost a bull’s-eye. MAPE measures each error relative to its own actual and averages the resulting percentages.

Drag the blue points — the forecasts. On the right, the percentage error of each object (APE, absolute percentage error) and their mean — MAPE.

The formula bone by bone

Let us take the formula apart from the inside out.

$y_i$

the actual; must be non-zero;

$\hat y_i$

the model’s forecast;

$|y_i-\hat y_i|$

the absolute error, as in MAE;

$\frac{|y_i-\hat y_i|}{|y_i|}$

the relative error APE — the share of the actual by which we missed; independent of the units;

$\frac{100\%}{n}\sum_{i=1}^n$

the mean over the objects, converted to per cent.

The running example step by step:

Two equivalent forms reveal how the metric works:

The first: MAPE is a weighted MAE with weights $w_i=1/|y_i|$; objects with a small actual get a huge weight. The second: for positive actuals APE depends only on the ratio $q_i=\hat y_i/y_i$, $\mathrm{APE}_i=|1-q_i|$. That is why MAPE does not change when all actuals and forecasts are multiplied by the same number.

Properties and asymmetry

• Dimensionless: $\mathrm{MAPE}(c\,y,\,c\,\hat y)=\mathrm{MAPE}(y,\hat y)$ for any $c\ne0$. Dollars or thousands of dollars — it does not matter.
• Not shift-invariant: add 1000 to the actuals and the forecasts — the absolute errors are the same, but MAPE drops many times over. So MAPE is meaningless for temperature in °C and other scales with an arbitrary zero.
• Undefined at $y_i=0$ and unbounded as $y_i\to0$: a forecast of 1 for an actual of $0.01$ is an error of $9900\%$.
• A zero forecast costs exactly $100\%$: with $\hat y_i=0$ every $\mathrm{APE}_i=1$. The “always zero” model has $\mathrm{MAPE}=100\%$ on any non-zero data.
• Asymmetry: for a positive actual and a non-negative forecast an under-forecast gives $\mathrm{APE}\le100\%$, while an over-forecast is unbounded.
• Convexity: MAPE is a sum of absolute values of linear functions of the forecasts, so it is convex in the forecasts (and in the parameters of a linear model); it is zero only for a perfect forecast.

Symmetric in units, not in ratios

If you count a miss in units, MAPE is symmetric: forecasts $y-d$ and $y+d$ cost the same, $d/y$. But in business misses are often measured in ratios: “over by half again”, “half of what we sold”. Here MAPE is asymmetric: forecasting half ($\hat y=y/2$) costs $50\%$, forecasting double ($\hat y=2y$) costs $100\%$. Of two forecasts that are off by the same factor, MAPE always prefers the lower one.

The derivative and the best constant

For a single object APE is the absolute value of a linear function of the forecast, so its derivative is a step:

The size of the derivative does not depend on the size of the error — as in MAE — but it is multiplied by $1/|y_i|$: an object with actual 1 pulls the model 100 times harder than an object with actual 100. In the running example $\partial\,\mathrm{MAPE}/\partial\hat y=\tfrac{100\%}{3}\,\bigl(-\tfrac13,\;0,\;\tfrac18\bigr)\approx(-11.1\%,\;0,\;4.2\%)$ per unit of forecast: the first object, with a four times smaller error, pulls $2.7$ times harder than the third.

The best constant is a weighted median shifted downwards

Which single forecast $c$ for all objects minimises MAPE? The function $\mathrm{MAPE}(c)=\frac{100\%}{n}\sum_i|y_i-c|/y_i$ is piecewise linear and convex; between neighbouring data points its slope is $\frac{100\%}{n}\bigl(\sum_{y_i<c}\tfrac1{y_i}-\sum_{y_i>c}\tfrac1{y_i}\bigr)$. The minimum is where the slope changes sign — where the total weight $1/y$ on the left balances the weight on the right. This is the weighted median with weights $1/y_i$:

The weights decrease as $y$ grows, so the weighted median is not larger than the ordinary one: MAPE systematically pulls the forecast down. For lognormal demand this can be computed exactly: if $\ln y\sim N(\mu,\sigma^2)$, then $c^*=e^{\mu-\sigma^2}$ — the median $e^{\mu}$ multiplied by $e^{-\sigma^2}$.

A map of the error: APE over the (y, ŷ) plane

Look at the error of one object as a function of two variables — the actual $y$ and the forecast $\hat y$. The level lines of APE are rays from the origin: the condition $|y-\hat y|/y=a$ is equivalent to $\hat y=(1\pm a)\,y$. The level lines of the absolute error, in contrast, are parallel lines $\hat y=y\pm e$. The rays converge at zero, so near $y=0$ any fixed absolute error crosses all levels of APE — hence the “explosion”.

For $n$ objects MAPE is the mean of such functions. In the forecast space $(\hat y_1,\dots,\hat y_n)$ its level sets are polytopes (diamonds in 2D) centred at $y$ with semi-axes proportional to $y_i$: along axes with small $y_i$ you cannot move far without a big penalty.

Training on MAPE: 1/y weights and under-forecasting

Since MAPE is MAE with weights $1/|y_i|$, a model can be trained on MAPE by any algorithm that minimises a weighted MAE: pass sample_weight=1/|y|. LightGBM has a ready-made objective="mape", and in PyTorch the loss is one line (see the code section).

Let us train the simplest model $\hat y=w\,x$ on 40 objects with $y=10\,x\cdot e^{0.35\,\varepsilon}$, $\varepsilon\sim N(0,1)$: the true slope is 10, and the noise is multiplicative — its size grows with $y$, as with real sales.

Why: points with small $y$ get a large weight and pull the line down. In terms of the ratios $q_i=y_i/x_i$ we have $\mathrm{MAPE}(w)=\frac{100\%}{n}\sum_i|q_i-w|/q_i$, so the optimum is the weighted median of $q_i$ with weights $1/q_i$. With lognormal noise this is the median times $e^{-\sigma^2}=e^{-0.1225}\approx0.885$, i.e. on a large sample about $8.85$ instead of 10 (the 400 points of the scikit-learn listing below give $8.85$). On the 40 points of the figure sampling noise is larger, and the optimum is $9.10$.

MAPE, sMAPE, WAPE and MASE

Because of these problems MAPE has several popular relatives ($e_i=y_i-\hat y_i$):

In the running example $\mathrm{MAPE}\approx13.9\%$, $\mathrm{sMAPE}\approx13.5\%$, $\mathrm{WAPE}=2.5/16\approx15.6\%$, and MASE with the scale $\tfrac12\bigl(|5-3|+|8-5|\bigr)=2.5$ equals $0.833/2.5\approx0.33$ (for simplicity the scale is computed on the same three points).

WAPE (also called the MAD/Mean ratio) is MAE divided by the mean actual: objects with large sales weigh more, and the total is not ruined by a single product with near-zero demand. MASE (Hyndman and Koehler, 2006) compares the model’s MAE with the MAE of the naive “same as last period” forecast computed on the training part of the series (for seasonal series, of the seasonal naive forecast), and works even for series with zeros. sMAPE comes in several variants: here we use $2|e_i|/(|y_i|+|\hat y_i|)$, bounded by $200\%$, as in M4; M3 used the denominator without absolute values, $y_i+\hat y_i$, and there is also a version without the factor 2, bounded by $100\%$ — check the formula before comparing numbers. On the same data WAPE and MASE differ only by a constant factor, so they share the same optimum.

A business example: a store’s assortment

A chain forecasts weekly sales of six products. Five of them sell in tens and hundreds of units and are forecast with errors from $7.5\%$ to $16.7\%$. The sixth, F, is a slow mover: the forecast was 10 units and 4 were sold.

What to do in practice:

• report WAPE (or MAPE weighted by revenue) when the overall accuracy in units or money matters;
• compute MAPE by groups — fast and slow movers separately — or exclude items with sales below a threshold, stating the threshold explicitly;
• for slow movers use MASE or dedicated metrics for intermittent demand;
• track the sign of the error separately: the bias of the total forecast is a KPI of its own, and MAPE does not show it.

In code

In NumPy MAPE is one line; below we also compute sMAPE and WAPE and check the formulas on the running example.

pythonimport numpy as np

y     = np.array([3.0, 5.0, 8.0])
y_hat = np.array([2.5, 5.0, 10.0])

ape  = np.abs((y - y_hat) / y)      # [0.16666667 0.         0.25      ]
mape = 100 * ape.mean()             # 13.888888888888888

# The same as a weighted MAE with weights 1/|y|
w = 1 / np.abs(y)
print(100 * np.mean(w * np.abs(y - y_hat)))   # 13.888888888888888

def smape(y, f):
    return 100 * np.mean(2 * np.abs(f - y) / (np.abs(y) + np.abs(f)))

def wape(y, f):
    return 100 * np.sum(np.abs(y - f)) / np.sum(np.abs(y))

print(smape(y, y_hat), wape(y, y_hat))        # 13.468013468013467 15.625

scikit-learn returns a fraction and replaces denominators below the machine epsilon ($\approx2.2\cdot10^{-16}$) with the epsilon — so instead of a division-by-zero error you get a huge number. Median regression with weights $1/y$ is a way to train “on MAPE”:

pythonimport numpy as np
from sklearn.metrics import mean_absolute_percentage_error
from sklearn.linear_model import LinearRegression, QuantileRegressor

# y, y_hat come from the listing above
print(mean_absolute_percentage_error(y, y_hat))            # 0.13888888888888887 — a fraction, not a percentage!
print(mean_absolute_percentage_error([0.0, 5.0], [1.0, 5.0]))  # 2251799813685248.0: division by eps instead of zero

# Training “on MAPE”: median (MAE) regression with weights 1/y
rng = np.random.default_rng(0)
x = rng.uniform(1, 10, 400)
t = 10 * x * np.exp(0.35 * rng.standard_normal(400))
X = x[:, None]
ols = LinearRegression(fit_intercept=False).fit(X, t)
fit = QuantileRegressor(quantile=0.5, alpha=0, fit_intercept=False).fit(X, t, sample_weight=1 / t)
print(ols.coef_, fit.coef_)   # [10.57549043] [8.85148171] — the MAPE model under-forecasts the total by 16%

MAPE as a loss in PyTorch:

pythonimport torch

def mape_loss(pred, target, eps=1e-8):
    """MAPE as a loss (a fraction). eps prevents division by 0 but not huge weights near zero."""
    return torch.mean(torch.abs(target - pred) / target.abs().clamp_min(eps))

y     = torch.tensor([3.0, 5.0, 8.0])
y_hat = torch.tensor([2.5, 5.0, 10.0], requires_grad=True)
loss = mape_loss(y_hat, y)
loss.backward()
print(loss.item(), y_hat.grad)   # 0.1388888955116272 tensor([-0.1111, -0.0000,  0.0417])

Common mistakes

1. MAPE on data with zeros. Division by zero gives infinity or NaN, and “protecting” it with $\varepsilon$ gives astronomical numbers: scikit-learn returns $2.25\cdot10^{15}$. Filter zeros explicitly and say so, or use WAPE and MASE.
2. MAPE for scales with an arbitrary zero — temperatures in °C, indices, dates. An error of 1° at 2° and at 30° is the same, but MAPE differs 15 times.
3. Training on MAPE without accounting for the bias. The model systematically under-forecasts, which in retail turns into stock-outs.
4. Comparing MAPE across data sets with different shares of small values. The same model will show 12% on fast movers and 60% on slow movers — the data are to blame, not the quality.
5. Mixing up fractions and per cent: scikit-learn returns $0.139$, not $13.9$.
6. “Lower MAPE means a more accurate plan”. MAPE does not see the sign of the errors: a forecast with lower MAPE can under-forecast total demand by 16%, as in Figure 5.

Links to other formulas

• MAE: MAPE is MAE with weights $1/|y_i|$; the size of the gradient also does not depend on the size of the error, but it is scaled by $1/|y_i|$.
• MSE and RMSE: absolute squared errors; their relative cousin is RMSLE, the RMSE on a log scale.
• Huber: smooths the kink at zero; the same idea can be applied to the relative error.
• SGD: training on MAPE is subgradient descent whose step must shrink (in Figure 5, as $1/\sqrt t$).

Summary

Check yourself