The Huber loss: a bridge between MSE and MAE

Why we need it

Regression learns from its errors $r=y-\hat y$. Two classic losses answer the question “how to penalise an error” differently. MSE penalises the square: it is smooth, its gradient is proportional to the error and fades gently near the minimum, but one outlier with error 10 weighs as much as a hundred errors of size 1. MAE penalises the absolute value: outliers hardly bother it, but it has a kink at zero and its gradient always has the same size, so the model struggles to polish small errors.

In 1964 Peter Huber proposed to take the best of both: a parabola for small errors and a straight line for large ones, switching at a threshold $\delta$. Where the idea works today:

• robust regression — HuberRegressor in scikit-learn, M-estimators in statistics;
• object detection — Smooth L1 in Fast R-CNN and its descendants for box-coordinate regression; it is Huber up to a factor;
• reinforcement learning — DQN clips the temporal-difference error to $[-1,\,1]$, which is equivalent to the Huber loss with $\delta=1$;
• gradient boosting — loss='huber' in scikit-learn, objective='huber' in LightGBM, pseudo-Huber in XGBoost;
• any data with anomalies — property prices, sensor readings, labels with typos.

Intuition: a parabola with straight tails

Picture a spring that pulls the prediction towards the answer. While the error is small, it is an ordinary spring: the more it is stretched, the harder it pulls — that is the parabola $\tfrac12r^2$. But the spring has a safety catch: after a stretch of $\delta$ the force stops growing and stays equal to $\delta$. From there on the energy of such a spring grows only linearly — those are the straight tails.

Drag the point along the plot and move $\delta$. Inside the teal band $|r|\le\delta$ the blue curve lies on the grey parabola, outside it lies on the dashed line $\delta(|r|-\delta/2)$. The orange tangent passes the seam without a kink.

The formula, bone by bone

$r=y-\hat y$

the residual: how far the answer $y$ is above the prediction $\hat y$; the loss depends only on $|r|$, the sign does not matter;

$\delta>0$

the threshold: errors below $\delta$ count as “ordinary noise”, larger ones as suspicious; it is measured in the units of $y$;

$\tfrac12 r^2$

the quadratic branch — exactly half the squared error, as in MSE; the factor $\tfrac12$ makes the derivative equal to $r$;

$\delta|r|$

the linear branch — the absolute error weighted by $\delta$, as in MAE;

$-\tfrac12\delta^2$

the downward shift that stitches the branches: without it the line $\delta|r|$ would give $\delta^2$ at $|r|=\delta$, while the parabola gives $\tfrac12\delta^2$.

Let us check the stitch. At $|r|=\delta$ the quadratic branch gives $\tfrac12\delta^2$ and the linear one $\delta(\delta-\tfrac12\delta)=\tfrac12\delta^2$: the values agree. The derivatives: $r=\delta$ for the parabola and $\delta\operatorname{sign}r=\delta$ for the line — the slopes agree too. The constant $-\tfrac12\delta^2$ was chosen precisely to make the line tangent to the parabola.

The table for $\delta=1$ shows the main point: for an error of 10 the square gives 50 while Huber gives 9.5, and the derivative (last column) never exceeds 1.

The running example

Take the example from the cheat sheet: $y=(3,\,5,\,8)$, $\hat y=(2.5,\,5,\,10)$, errors $r=(0.5,\,0,\,-2)$. With $\delta=1$ the first error falls on the parabola: $\tfrac12\cdot0.5^2=0.125$; the second is zero; the third, $|r|=2>1$, falls on the line: $1\cdot(2-0.5)=1.5$. The mean is $(0.125+0+1.5)/3\approx0.542$.

The gradients are more interesting. In half-MSE they equal the errors themselves, $(0.5,\,0,\,-2)$: the outlier pulls four times harder than the first point. For Huber $\psi=(0.5,\,0,\,-1)$ — only twice as hard: a big miss cannot shout down the others.

Scale: one curve for every δ

If the error is measured in units of the threshold, all Huber losses are one and the same curve:

That is why $\delta$ cannot be chosen “in general”: it depends on the units of $y$. An error of 1 dollar and of 1 million dollars are different things. Usually $\delta$ is set as a multiple of the typical noise spread $\sigma$ (see the section on choosing $\delta$).

Properties

• Non-negative and even: $L_\delta(r)\ge0$, $L_\delta(-r)=L_\delta(r)$, the minimum $0$ only at $r=0$.
• Convex: a parabola and its tangent are glued without a “dip”, so linear regression with the Huber loss has no spurious local minima.
• Smoothness $C^1$ but not $C^2$: the first derivative is continuous, the second jumps from 1 to 0 at $|r|=\delta$.
• Squeezed between a parabola and a line: $\delta|r|-\tfrac12\delta^2\le L_\delta(r)\le\tfrac12r^2$; the gap to each equals $\tfrac12(|r|-\delta)^2$ wherever the curve does not coincide with it.
• Bounded derivative: $|L'_\delta(r)|\le\delta$ — the key property for robustness.
• Limits: as $\delta\to\infty$, $L_\delta(r)\to\tfrac12r^2$ (MSE); as $\delta\to0$, $L_\delta(r)/\delta\to|r|$ (MAE).

The derivative: a clipped residual

In statistics the derivative with respect to the residual is denoted $\psi$ and called the ψ-function of the estimator; the influence function is proportional to it, $\mathrm{IF}(x)=\psi(x-\mu)/\mathbb E\,\psi'$. Both show how hard a single point pulls the estimate.

Compare the three losses. For MSE ($\tfrac12r^2$) the pull $\psi=r$ grows without limit. For MAE it is constant. For Huber it grows while the point is “one of ours” and freezes at $\delta$ once the point is suspicious. A second useful picture is the weight of a point, $w(r)=\psi(r)/r$: what share of a “vote” the point gets compared with ordinary least squares.

The surface L(r, δ) in 3D

Let us view the loss as a function of two variables — the error $r$ and the threshold $\delta$. This shows the whole family of curves at once and how the threshold changes the shape.

Two things are visible at once. First, inside the “wedge” $|r|\le\delta$ the surface does not depend on $\delta$: it is always $\tfrac12r^2$ there, and the level lines $L=c$ are the vertical segments $r=\pm\sqrt{2c}$ (for $\delta\ge\sqrt{2c}$). Second, outside, the level line $L=c$ is $|r|=c/\delta+\delta/2$: the smaller the threshold, the further you must go to collect the same loss.

Huber’s estimate and the choice of δ

Huber designed his function for the simplest problem in statistics — estimating the centre of a sample $x_1,\dots,x_n$. Minimising $\sum(x_i-\mu)^2$ gives the mean; minimising $\sum|x_i-\mu|$ gives the median. Huber’s estimate $\hat\mu$ minimises $\sum L_\delta(x_i-\mu)$, i.e. it solves

Points inside the zone $|x_i-\hat\mu|\le\delta$ take part as in the mean, while far ones contribute only their sign, with force $\delta$, as in the median.

Look at the right plot: the path $\hat\mu(\delta)$ need not lie between the median and the mean — at $\delta=1$ the estimate is below both. The limiting values are exact: as $\delta\to0$ we get the median, and as soon as $\delta$ exceeds the distance from the mean to the farthest point (here $13.5-6.079\approx7.421$), every point is in the quadratic zone and $\hat\mu$ equals the mean.

The price of robustness: the 1.345σ rule

If the data are actually normal, with no outliers, the best estimate is the mean. How much worse is Huber? This is measured by the asymptotic efficiency: the ratio of the mean’s variance to the estimate’s variance. For $\psi$ with threshold $k$ (in units of $\sigma$) and a standard normal $X$:

Solving “efficiency $=95\%$” gives $k\approx1.345$. Hence the classic choice $\delta=1.345\,\sigma$: only a 5% loss on clean data in exchange for protection against outliers. The median’s efficiency is $2/\pi\approx64\%$.

Huber in training

Let us fit a line by gradient descent in two ways — on half-MSE and on the Huber loss — on the same data: 30 points around $y=1+0.5x$ with noise $\sigma=0.5$ and three outliers at the bottom right. The threshold is $\delta=0.7\approx1.345\cdot0.5$.

A descent step shows the mechanism: each point pulls the parameters in proportion to $\psi(r_i)$. While the line is far from all points, every Huber pull equals $\pm\delta$ — it moves like MAE, at a constant speed. Once the line reaches the bulk of the points, their errors enter the quadratic zone and the descent becomes ordinary least squares — now without the outliers, which stay on a $\delta$ leash.

Where else Huber works inside training

• Gradient boosting. Each new tree learns the negative gradient of the loss. For Huber this is $\psi(r_i)$ — clipped residuals, so a single anomalous value does not make a tree spend a leaf on the outlier. In scikit-learn the threshold $\delta$ is recomputed at every iteration as the $\alpha$-quantile of $|r_i|$ (default $\alpha=0.9$).
• DQN. In the DeepMind paper (2015) the temporal-difference error was clipped to $[-1,\,1]$. The clipped gradient is exactly $\psi_1$, i.e. training on the Huber loss with $\delta=1$: a single huge TD error (say, right after the target network changes) cannot cause an explosive step, so training is more stable.
• Fast R-CNN. For regressing the box coordinates Girshick (2015) chose Smooth L1: it is “less sensitive to outliers than L2” and does not require careful tuning of the learning rate to avoid exploding gradients.

Smooth relatives: pseudo-Huber, log-cosh, Smooth L1

The jump in the second derivative does not hurt gradient descent, but it hurts second-order methods. XGBoost and LightGBM take a Newton step: a leaf gets the value $-\sum g_i/\sum h_i$, where $h_i$ are second derivatives. The true second derivative of Huber is zero for every outlier, and if a leaf receives only outliers, the denominator vanishes (in XGBoost only the regulariser $\lambda$ is left of it). So implementations either replace it with a constant (LightGBM does this in objective='huber') or use a smooth variant.

• Pseudo-Huber (in computer vision, up to the factor $\delta$, the Charbonnier loss $\sqrt{r^2+\delta^2}-\delta$): the derivative $r/\sqrt{1+(r/\delta)^2}$ approaches $\pm\delta$ slowly; the second derivative $(1+(r/\delta)^2)^{-3/2}$ is positive everywhere. In XGBoost it is objective='reg:pseudohubererror' with the parameter huber_slope ($=\delta$).
• log-cosh: the derivative $\delta\tanh(r/\delta)$ is a smooth “clip”; on the tails $\delta^2\ln\cosh(r/\delta)\approx\delta|r|-\delta^2\ln2$.
• Smooth L1 is Huber divided by $\beta$: $\tfrac{0.5r^2}{\beta}$ for $|r|<\beta$ and $|r|-\tfrac12\beta$ beyond. As $\beta\to0$ it tends to $|r|$ (while Huber tends to zero). In PyTorch: HuberLoss(delta=δ) = δ · SmoothL1Loss(beta=δ).

In code

The formula itself is one line with np.where. The gradient is np.clip. Below: an implementation, a check on the running example and Huber regression via IRLS.

pythonimport numpy as np

def huber(r, delta=1.0):
    """Huber loss for the residuals r = y − ŷ (element-wise)."""
    a = np.abs(r)
    return np.where(a <= delta, 0.5 * r**2, delta * (a - 0.5 * delta))

def huber_grad(r, delta=1.0):
    """ψ(r) = dL/dr: the residual clipped to [−δ, δ]. With respect to ŷ the sign flips."""
    return np.clip(r, -delta, delta)

def pseudo_huber(r, delta=1.0):
    return delta**2 * (np.sqrt(1.0 + (r / delta)**2) - 1.0)

y     = np.array([3.0, 5.0, 8.0])
y_hat = np.array([2.5, 5.0, 10.0])
r = y - y_hat               # [ 0.5  0.  -2. ]
print(huber(r))             # [0.125 0.    1.5  ]
print(huber(r).mean())      # 0.5416666666666666
print(huber_grad(r))        # [ 0.5  0.  -1. ]

pythondef huber_regression(X, y, delta=1.345, iters=50):
    """Huber regression by IRLS: least squares with weights w_i = min(1, δ·s/|r_i|)."""
    X1 = np.column_stack([np.ones(len(X)), X])
    beta = np.linalg.lstsq(X1, y, rcond=None)[0]           # start from ordinary least squares
    for _ in range(iters):
        r = y - X1 @ beta
        s = np.median(np.abs(r - np.median(r))) / 0.6745   # robust scale (MAD)
        w = np.minimum(1.0, delta * s / np.maximum(np.abs(r), 1e-12))
        sw = np.sqrt(w)
        beta = np.linalg.lstsq(X1 * sw[:, None], y * sw, rcond=None)[0]
    return beta

rng = np.random.default_rng(0)
x = np.linspace(0, 10, 30)
y = 1 + 0.5 * x + rng.normal(0, 0.5, 30)
y[-3:] -= 6                                                # three outliers
X1 = np.column_stack([np.ones(30), x])
print(np.linalg.lstsq(X1, y, rcond=None)[0].round(3))      # [1.931 0.182] — least squares
print(huber_regression(x, y).round(3))                     # [1.134 0.442] — Huber

pythonimport torch
import torch.nn as nn

y_hat = torch.tensor([2.5, 5.0, 10.0], requires_grad=True)
y = torch.tensor([3.0, 5.0, 8.0])

loss = nn.HuberLoss(delta=1.0)(y_hat, y)   # mean: 0.5417
loss.backward()
print(y_hat.grad)       # tensor([-0.1667,  0.0000,  0.3333]) = ψ(ŷ − y) / n

# Smooth L1 from Fast R-CNN is Huber divided by β (β = δ):
huber2 = nn.HuberLoss(delta=2.0)(y_hat, y)        # 0.7083
sl1_2 = nn.SmoothL1Loss(beta=2.0)(y_hat, y)       # 0.3542 = 0.7083 / 2

Common mistakes

1. δ without regard to scale. $\delta=1$ for prices in dollars and for prices in millions are completely different losses. Normalise the target or set $\delta$ in units of a robust spread.
2. Confusing Huber and Smooth L1. At the same threshold they differ by a factor: Smooth L1 $=$ Huber$/\beta$, so for $\beta>1$ its loss and gradient are smaller and for $\beta<1$ larger (9 times larger at $\beta=1/9$, as in the Faster R-CNN RPN). When swapping one for the other, re-tune the learning rate.
3. Estimating σ with the ordinary std for the $1.345\sigma$ rule — outliers inflate the std, the threshold becomes too large, and the protection disappears. Use the MAD.
4. Expecting Huber to protect against outliers in the features. It limits the influence of a large residual $r$, but a point with an extreme $x$ (high leverage) can still swing the line. That needs other methods (for example, weights based on $x$).
5. Huber in Newton-step boosting without a check: zero second derivatives of the outliers make leaf steps unstable. Use pseudo-Huber or an implementation that accounts for this.
6. Comparing loss values at different δ. $L_\delta$ grows with $\delta$, other things being equal; compare models by an external metric (MAE, RMSE on validation), not by the loss itself.

Links to other formulas

• MSE: for $\delta\ge\max|r_i|$ the Huber loss equals half the squared error; if no least-squares residual exceeds $\delta$, Huber’s estimate coincides with least squares.
• MAE: as $\delta\to0$, $L_\delta/\delta\to|r|$; the location estimate tends to the median.
• RMSE: the metric most easily spoiled by outliers; Huber lets you train robustly and still report RMSE on clean data.
• ReLU and clipping: $\psi_\delta(r)=\operatorname{clip}(r,-\delta,\delta)$ is the same “linear up to a threshold” idea as in ReLU6 and gradient clipping.
• Focal Loss is the opposite philosophy: Huber weakens the largest errors (treating them as outliers), while focal loss increases the weight of hard examples relative to easy ones.

Summary

Check yourself