RMSE: the root mean squared error

Why it is needed

A model predicts the price of a flat, the demand for a product, tomorrow’s temperature. How do you say with a single number how wrong it is? A plain mean of the errors will not do: misses up and down cancel out. MSE squares the errors — but then the answer is in “dollars squared” or “degrees squared”, and “the MSE is 1,417,000” tells a client nothing. RMSE takes the square root and brings the error back to the units of the target, keeping everything people like about MSE.

Where it shows up:

• evaluating regression models — one of the most common metrics in reports, papers and Kaggle competitions;
• time-series forecasting — weather, demand, energy use; it usually sits next to MAE and MAPE;
• recommender systems — the famous Netflix Prize (2006–2009) scored rating predictions by RMSE;
• physics and metrology — the root-mean-square error of measurements, GPS accuracy, the “RMS” value of a signal;
• training models — minimising RMSE and minimising MSE give the same model, but their gradients differ (see the section on the derivative).

Intuition: the side of the mean square

Draw every error $r_i=y_i-\hat y_i$ as a square with side $|r_i|$. Its area $r_i^2$ is the object’s contribution to MSE. Add up the areas and divide by $n$ — that is the mean area, i.e. MSE. And RMSE is the side of the square whose area equals the mean area.

Drag the blue points — the forecasts. On the right all the error squares are nested in one corner: the blue square has side RMSE, the dashed teal one has side MAE.

The formula bone by bone

Read the name from right to left and you get the recipe: the root of the mean of the squared error.

$y_i$

the true value of the $i$-th object (the actual);

$\hat y_i$

the model’s forecast for that object;

$r_i=y_i-\hat y_i$

the error, or residual; its sign shows whether we under-estimated ($r_i>0$) or over-estimated ($r_i<0$);

$r_i^2$

squaring removes the sign and makes large errors disproportionately heavy: an error twice as large weighs four times as much;

$\frac1n\sum_{i=1}^n$

the mean over all $n$ objects is MSE, in squared units;

$\sqrt{\cdot}$

the root brings back the original units: if $y$ is in dollars, so is RMSE.

The running example step by step:

It helps to recognise the same quantity in other forms:

The second form is the Euclidean distance between the vector of actuals and the vector of forecasts divided by $\sqrt n$ (we will come back to it in 3D). The third is the quadratic mean (the power mean of order 2) of the absolute errors. MAE is their arithmetic mean $M_1$, and the largest error is the limiting case $M_\infty$. The power mean inequality immediately gives $\mathrm{MAE}\le\mathrm{RMSE}\le\max_i|r_i|$.

Properties

• Non-negativity: $\mathrm{RMSE}\ge0$, and $\mathrm{RMSE}=0$ only for a perfect forecast $\hat y=y$.
• Units — the same as those of $y$. Convert prices from thousands of dollars to dollars and RMSE grows exactly 1000 times: $\mathrm{RMSE}(c\,y,\,c\,\hat y)=|c|\,\mathrm{RMSE}(y,\hat y)$.
• Shifts do not matter: add the same number to the actuals and the forecasts and RMSE stays the same. So RMSE does not say whether the error is large relative to the level of the series — that is what MAPE and normalised variants are for.
• The same ranking as MSE: the root is strictly increasing, so comparing models by RMSE and by MSE always gives the same order, and both have their minimum at the same point.
• Sensitivity to outliers: a single error of size $E$ makes RMSE at least $E/\sqrt n$, however many perfect forecasts surround it.
• Non-additivity: the RMSE of the whole sample is not the mean of the RMSEs of its parts. Squares add up: $\mathrm{RMSE}^2=\sum_g \tfrac{n_g}{n}\,\mathrm{RMSE}_g^2$.
• It is a norm (up to the factor $1/\sqrt n$): for error vectors $a$ and $b$ the triangle inequality holds, $\mathrm{RMSE}(a+b)\le\mathrm{RMSE}(a)+\mathrm{RMSE}(b)$.

Bias and spread: Pythagoras for errors

The residuals split into a systematic part — the mean $\bar r$ (the model over- or under-estimates on average) — and a random part, the spread around the mean $\sigma_r$. Then

In the running example $\bar r=-0.5$ (the model over-estimates by $0.5$ on average), $\sigma_r\approx1.080$, and $0.25+1.167=1.417=\mathrm{MSE}$. The practical takeaway: bias can be removed for free — add $\bar r$ to every forecast, and on the data where it was estimated RMSE drops exactly to $\sigma_r$ (here from $1.190$ to $1.080$). In practice $\bar r$ is estimated on validation data and the gain is checked on the test set.

RMSE versus MAE: what the gap says

MAE answers “how far off are we on average”, RMSE answers “how large is a typical error, counting the big ones”. Their ratio $\mathrm{RMSE}/\mathrm{MAE}$ is dimensionless and scale-free: it describes only the shape of the error distribution.

For such a mixture MSE grows $1+99\varepsilon$ times ($(1-\varepsilon)\cdot1+\varepsilon\cdot100$) while MAE grows only $1+9\varepsilon$ times, so the theoretical ratio is $\sqrt{1+99\varepsilon}\,/\bigl((1+9\varepsilon)\,\mathrm{MAE}_0\bigr)$, where $\mathrm{MAE}_0$ is the MAE of the original distribution.

The bounds: MAE ≤ RMSE ≤ √n · MAE

The left equality holds only when all $|r_i|$ are equal; the right one when the whole error sits in a single object and all other forecasts are perfect. In the running example the ratio is $1.190/0.833\approx1.43$ against the maximum $\sqrt3\approx1.73$: the errors are clearly uneven.

Geometry: RMSE as a distance

Write the actuals and the forecasts as points of $n$-dimensional space: $y=(y_1,\dots,y_n)$ and $\hat y=(\hat y_1,\dots,\hat y_n)$. The sum of squared errors is the square of the ordinary Euclidean distance between them, so

Dividing by $\sqrt n$ makes the distance comparable across sample sizes: if every error equals $c$, then $\|y-\hat y\|=c\sqrt n$ and RMSE is exactly $c$. All forecasts with the same RMSE form a sphere of radius $\sqrt n\,\mathrm{RMSE}$ centred at $y$. With $n=3$ we can draw it.

The best constant is the mean, and its RMSE is the standard deviation

Which constant forecast $\hat y_i=c$ minimises RMSE? Geometrically it is the orthogonal projection of $y$ onto the diagonal, i.e. $c=\bar y$. Its RMSE is the standard deviation of the target:

This is the main reference point for reading RMSE: a model whose RMSE is not below $\sigma_y$ is no better than “always predict the mean”. In the example $\sigma_y\approx2.055$, and the model achieves $1.190$. The coefficient of determination follows from the same picture:

Derivative and gradient

Let the forecasts depend on model parameters $\theta$. The derivative of $\sqrt u$ is $1/(2\sqrt u)$, so by the chain rule the gradient of RMSE is the gradient of MSE divided by $2\,\mathrm{RMSE}$:

What follows from this:

• Same direction, different length. The gradients of RMSE and MSE differ only by the positive factor $1/(2\,\mathrm{RMSE})$. Hence the two functions share their stationary points and their minimum.
• The gradient with respect to the forecasts has constant length: $\|\nabla_{\hat y}\mathrm{RMSE}\|=1/\sqrt n$ for any errors (the sum of the squared components is $\sum_j r_j^2/(n^2\,\mathrm{MSE})=1/n$). The RMSE gradient points only in the direction of $y$, and its length does not depend on how far away we are.
• In the running example $\partial\,\mathrm{RMSE}/\partial\hat y=(-0.140,\;0,\;0.560)$, while for MSE it is $(-0.333,\;0,\;1.333)$, exactly $2\cdot1.190=2.380$ times larger.
• There is no derivative at zero. At $\hat y=y$ the function $\|y-\hat y\|/\sqrt n$ has a kink, like $|x|$ at zero (the tip of a cone), and the formula divides by $\mathrm{RMSE}=0$. In automatic differentiation this turns into NaN (see the code section).

A one-dimensional slice: the constant forecast

Take the constant model $\hat y_i=c$ on the example data $y=(3,\,5,\,8)$. Then $\mathrm{MSE}(c)=(c-\bar y)^2+\sigma_y^2$ is a parabola and RMSE is a hyperbola:

Near the minimum the hyperbola looks like a parabola (RMSE behaves like MSE), far away like the lines $|c-\bar y|$ (like MAE). The derivative moves smoothly from $-1$ to $1$ and never leaves these bounds, whereas the derivative of MSE grows linearly without bound.

The second derivative $\mathrm{RMSE}''(c)=\sigma_y^2/\mathrm{RMSE}(c)^3>0$: the function is strictly convex as long as $\sigma_y>0$. In general RMSE is convex in the forecasts (as a norm), and therefore in the parameters of a linear model: linear regression has no local minima for either MSE or RMSE.

The RMSE(w, b) landscape and training

A linear model $\hat y=wx+b$ on 14 points. Over the parameter plane $(w,\,b)$ RMSE forms a bowl. Switch the surface to MSE: the level lines stay the same (the level $\mathrm{RMSE}=1.5$ is the level $\mathrm{MSE}=2.25$), only the height changes: MSE grows quadratically, while far from the minimum RMSE grows almost linearly, like a cone.

Why. A step on RMSE is a step on MSE with a variable learning rate:

While the errors are large the effective step is small — which is why descent on RMSE tolerates a too-large $\eta$ better. Near the minimum $\eta_{\text{eff}}\to\eta/(2\,\mathrm{RMSE}_{\min})$; here $\mathrm{RMSE}_{\min}\approx0.904$, and the step is almost half that of MSE — hence the slow finish. Descent on MSE is stable only for $\eta<2/\lambda_{\max}\approx0.133$, where $\lambda_{\max}\approx15.0$ is the largest eigenvalue of the Hessian of MSE.

RMSE in model evaluation

As a quality metric RMSE answers “by how many units of $y$ is the model typically off, with emphasis on big misses”. For the number to mean something, compare it:

• with a baseline — the “always the mean” forecast has RMSE $\sigma_y$, the “same as yesterday” forecast of a time series has its own; the model should be clearly better;
• with the scale of the task — an error of 5000 is great for flats and awful for cups of coffee. The dimensionless version is $\mathrm{NRMSE}=\mathrm{RMSE}/\sigma_y=\sqrt{1-R^2}$ (sometimes one divides by the range or the mean of $y$);
• with MAE — the ratio $\mathrm{RMSE}/\mathrm{MAE}$ shows whether there are large misses;
• only on the same data — RMSEs on different test sets are not comparable if their $y$ spreads differ.

Cross-validation: average the squares

With $K$ folds people often take the mean of the fold RMSEs. That is not the RMSE over all objects: the mean of roots is at most the root of the mean. The overall estimate is the root of the weighted mean MSE:

For example, two folds of equal size with RMSE $1$ and $3$: the mean is $2$, while the overall RMSE is $\sqrt{(1+9)/2}=\sqrt5\approx2.24$.

Variants: RMSLE, weighted and normalised RMSE

RMSLE (root mean squared logarithmic error) is RMSE on a log scale:

The logarithm turns ratios into differences: the error is determined by how many times $1+\hat y$ differs from $1+y$, not by how much. That is why RMSLE suits non-negative targets that span many scales (prices, sales, views) and punishes an under-forecast harder than an over-forecast of the same absolute size. In the running example $\mathrm{RMSLE}\approx0.139$.

Two more useful modifications:

• weighted RMSE $\sqrt{\sum_i w_i r_i^2/\sum_i w_i}$ — when objects are not equally important (expensive goods, recent observations);
• NRMSE $=\mathrm{RMSE}/\sigma_y=\sqrt{1-R^2}$ — a dimensionless version for comparing tasks of different scales.

In code

In NumPy RMSE is one line; below we also check the formulas of this page on the running example.

pythonimport numpy as np

y     = np.array([3.0, 5.0, 8.0])
y_hat = np.array([2.5, 5.0, 10.0])

r    = y - y_hat                   # residuals: [ 0.5  0.  -2. ]
mse  = np.mean(r ** 2)             # ≈ 1.4167
rmse = np.sqrt(mse)                # ≈ 1.1902
mae  = np.mean(np.abs(r))          # ≈ 0.8333

# The same via the Euclidean norm: ‖y − ŷ‖ / √n
print(round(np.linalg.norm(r) / np.sqrt(len(r)), 4))   # 1.1902

# Decomposition: RMSE² = bias² + residual variance
print(round(r.mean() ** 2 + r.var(), 4), round(mse, 4))   # 1.4167 1.4167

# Gradient with respect to the forecasts: ∂RMSE/∂ŷ = (ŷ − y) / (n · RMSE)
print(np.round((y_hat - y) / (len(y) * rmse), 4))       # [-0.14    0.      0.5601]

# The “always the mean” baseline: its RMSE is the standard deviation of y
print(round(np.std(y), 4))                               # 2.0548

scikit-learn has ready-made functions (since version 1.4; before that one wrote mean_squared_error(..., squared=False); the squared parameter was removed in 1.6). Note the difference between the mean fold RMSE and the RMSE over all objects:

pythonfrom sklearn.metrics import root_mean_squared_error, root_mean_squared_log_error
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score, cross_val_predict

# y, y_hat come from the listing above; the functions exist in scikit-learn ≥ 1.4
print(round(root_mean_squared_error(y, y_hat), 4))       # 1.1902
print(round(root_mean_squared_log_error(y, y_hat), 4))   # 0.1392 — RMSLE

X, t = make_regression(n_samples=200, n_features=5, noise=10.0, random_state=0)
cv = KFold(5, shuffle=True, random_state=0)
# scorers are “higher is better”, so RMSE comes with a minus sign
scores = cross_val_score(LinearRegression(), X, t, cv=cv,
                         scoring="neg_root_mean_squared_error")
print(round(-scores.mean(), 3))                # 10.56 — mean RMSE of the folds
pred = cross_val_predict(LinearRegression(), X, t, cv=cv)
print(round(root_mean_squared_error(t, pred), 3))   # 10.614 — RMSE over all objects

In PyTorch RMSE as a loss needs care at zero:

pythonimport torch

y     = torch.tensor([3.0, 5.0, 8.0])
y_hat = torch.tensor([2.5, 5.0, 10.0], requires_grad=True)

rmse = torch.sqrt(torch.mean((y_hat - y) ** 2))
rmse.backward()
print(round(rmse.item(), 4), y_hat.grad)   # 1.1902 tensor([-0.1400,  0.0000,  0.5601])

# A perfect forecast: the derivative of √u at zero is infinite → NaN gradient
p = y.clone().requires_grad_(True)
torch.sqrt(torch.mean((p - y) ** 2)).backward()
print(p.grad)                    # tensor([nan, nan, nan])

def rmse_loss(pred, target, eps=1e-8):
    """RMSE as a loss: eps under the root removes the NaN."""
    return torch.sqrt(torch.mean((pred - target) ** 2) + eps)

Common mistakes

1. The mean RMSE over batches or folds as the final RMSE. The root is non-linear: average MSE and take the root at the end.
2. Comparing RMSE across different data. RMSE depends on the scale and spread of $y$; compare models on the same test set or use a normalised RMSE.
3. “The model is off by RMSE on average”. The mean absolute error is MAE; RMSE is never smaller, and with heavy tails it is noticeably larger.
4. RMSE as a loss without protecting zero. The derivative of $\sqrt u$ at zero is infinite: a perfect batch gives NaN gradients. Add $\varepsilon$ under the root or train on MSE.
5. Ignoring outliers. One error of 100 units among 99 perfect forecasts gives $\mathrm{RMSE}=10$ with $\mathrm{MAE}=1$. If outliers are noise rather than important events, look at MAE, Huber or error quantiles.
6. RMSE on logarithms instead of the original scale. If the model is trained on $\ln(1+y)$, its RMSE is the RMSLE of the original task, not the RMSE in dollars; transform the forecasts back before computing RMSE.

Links to other formulas

• MSE: $\mathrm{RMSE}=\sqrt{\mathrm{MSE}}$ — the same minimum and the same ranking of models, while the gradient is divided by $2\,\mathrm{RMSE}$.
• MAE: $\mathrm{MAE}\le\mathrm{RMSE}\le\sqrt n\,\mathrm{MAE}$; the best constant for MAE is the median, for RMSE the mean.
• Huber: quadratic near zero and linear in the tails — a compromise between MSE (and hence RMSE) and MAE.
• MAPE: a relative error instead of an absolute one; RMSLE is its quadratic relative on a log scale.
• SGD: descent on RMSE is descent on MSE with learning rate $\eta/(2\,\mathrm{RMSE})$.
• Adam: divides the step by the root of a running mean of squared gradients — the same “root mean square” (RMS) idea as in RMSE.

Summary

Check yourself