Triplet Loss: the positive must be closer than the negative

Why we need it

An ordinary classifier knows a fixed list of classes. But a face recognition system must recognize people who were not in the training set, and a re-identification (Re-ID) system must find the same person in another camera’s footage, from another angle and under different light. Retraining the network for every new person is impossible.

The way out is to learn not classes but a metric (metric learning). The network $f$ turns an image into an embedding vector $f(x)\in\mathbb R^d$ so that pictures of the same person end up close together and pictures of different people far apart. Then “who is this?” becomes a nearest-neighbour search by distance, and a new person is added as a single embedding in the database.

Triplet Loss states this requirement directly. Take a triplet: an anchor $a$, a positive $p$ (the same person) and a negative $n$ (a different person). Require the positive to be closer than the negative by at least a margin $m$.

• face recognition — FaceNet (Google, 2015): 128-dimensional embeddings, margin $0.2$, accuracy $99.63\%$ on the LFW benchmark;
• re-identifying people across cameras — the batch-hard triplet loss (Hermans et al., 2017); in the torchreid library it can be trained together with cross-entropy over identities (the ImageTripletEngine);
• searching for similar images, products, voices and any search by meaning where the question is “what is closest to this sample?”.

Intuition: the positive closer than the negative

Draw the embeddings on a plane. Around the anchor $a$ draw a circle through the positive $p$ — its radius is $d(a,p)$. Add a second one, wider by the margin: radius $d(a,p)+m$. The formula’s requirement is simple: the negative must lie outside the outer circle. If it falls into the orange ring or inside it, the loss is positive.

The formula, bone by bone

Let us write the loss in full — with the network, the distances and the average over triplets:

$f(x)$

the embedding: the network output for an image $x$, a vector in $\mathbb R^d$ ($d=128$ in FaceNet), often normalized to unit length;

$d(a,p)$

the distance to the positive, $\|f(x_a)-f(x_p)\|$; we want to decrease it;

$d(a,n)$

the distance to the negative, $\|f(x_a)-f(x_n)\|$; we want to increase it;

$m>0$

the margin: the negative must be farther than the positive not just barely, but with a reserve $m$;

$\max(0,\cdot)$

the hinge: a satisfied requirement is not penalized and gives no gradient;

$T$

the set of triplets we average over — usually triplets inside a batch (see the section on mining).

It is convenient to introduce a single variable — the difference of distances $\Delta=d(a,p)-d(a,n)$. Then $L=\max(0,\Delta+m)$: a ReLU shifted by $m$. A negative $\Delta$ means the positive is already closer; but as long as $\Delta>-m$, the reserve is not enough.

Examples with $m=0.2$: $d(a,p)=0.4$, $d(a,n)=1.1$ give $\Delta=-0.7$ and $L=\max(0,\,-0.5)=0$ — the triplet is already separated. $d(a,p)=0.9$, $d(a,n)=1.0$ give $\Delta=-0.1$ and $L=0.1$: the positive is closer, but the margin is too small.

Three regimes: easy, semi-hard, hard

Every triplet falls into one of three regimes depending on $d(a,p)$ and $d(a,n)$:

In the $(d(a,p),\,d(a,n))$ plane the regime boundaries are two parallel lines: $d(a,n)=d(a,p)$ and $d(a,n)=d(a,p)+m$. The loss $d(a,p)-d(a,n)+m$ is a tilted plane whose level lines are parallel to these lines; above the upper line it is cut off at zero.

Gradients: where each point is pulled

For an active triplet ($L>0$) the loss equals $d(a,p)-d(a,n)+m$, and everything reduces to the gradient of a distance. For the Euclidean norm $\nabla_u\|u-v\|=\frac{u-v}{\|u-v\|}$ for $u\ne v$ — the unit vector from $v$ to $u$ (at $u=v$ the norm is not differentiable; its subgradients are all vectors of length at most 1, and code usually takes 0).

A gradient-descent step goes against the gradient: the positive $p$ moves towards the anchor, the negative $n$ away from the anchor, and the anchor moves towards the positive and away from the negative. In the ordinary metric the moves of the positive and the negative have length 1 at any distance (the anchor’s move is between 0 and 2); in the squared one they are $2\,d(a,p)$ and $2\,d(a,n)$, that is, proportional to the distances.

This is visible on a 3D surface. Fix the anchor at the origin and the positive at distance $0.6$; how does the loss depend on the position of the negative $n=(x,y)$? We get an upside-down cone: $L=0.6+m$ in the centre, falling to zero at the edge of a disc of radius $0.6+m$. The slope of the cone is the same everywhere, $|\nabla L|=1$.

Mining: which triplets to take

A set of $N$ images contains on the order of $N^3$ triplets — they cannot all be enumerated, and almost all of them are easy anyway. Triplets are chosen online, inside a batch. The batch is built as $P\times K$: $P$ people with $K$ pictures each. Inside such a batch

With $P=4$, $K=4$ this is 576 triplets; with $P=18$, $K=4$ (the batch used by Hermans et al.) it is already 14,688. The main strategies:

• batch-all — take all triplets of the batch and average the loss over the active ones ($L>0$) only, so that the easy ones do not dilute the signal (the BA$_{\ne0}$ variant of Hermans et al.);
• batch-hard — one triplet per anchor: the farthest positive and the closest negative; $PK$ triplets, the most informative ones;
• semi-hard (FaceNet) — for a pair $(a,p)$ take a negative that is farther than the positive but inside the margin; this makes training gentler at the start.

How it works in training

Let us put everything together. Suppose every picture has two features: $x_1$ is a nuisance (pose, lighting) that varies a lot for the same person, and $x_2$ is an identity feature slightly bent by the nuisance. In the input space two pictures of the same person can be farther apart than pictures of different people: only $62.7\%$ of the triplets are correctly ordered.

A small network $f:\mathbb R^2\to\mathbb R^2$ (12 tanh neurons) is trained with the batch-hard triplet loss at $m=0.5$ and the Adam optimizer. It is not told where each person should end up — only that positives must be closer than negatives.

Re-ID works the same way: the query is a picture of a person from one camera, the gallery holds pictures from all the others. The gallery embeddings are sorted by distance to the query, and one looks at the position of the first correct match. The metrics are Rank-1 (the correct match comes first) and mAP (mean average precision over all correct matches).

Normalization on the sphere and choosing the margin

Without constraints the network can increase all distances by simply stretching the space: in the ordinary metric the margin $m$ loses its meaning if the scale is not fixed. That is why embeddings are often normalized: $\hat f=f/\|f\|$, and all points lie on the unit sphere. For unit vectors

where $\theta$ is the angle between the vectors. The squared distance ranges from 0 (identical) to 4 (opposite), and FaceNet’s margin $m=0.2$ is $5\%$ of the whole range. The requirement $d^2(a,n)\ge d^2(a,p)+m$ becomes a requirement on angles: $\cos\theta_{an}\le\cos\theta_{ap}-m/2$.

In code: batch mining and stability

The key to efficiency is the matrix of all pairwise distances in the batch, from which mining is done with masks and no loops. A numerical subtlety: $\|u-v\|$ is not differentiable at $u=v$ (the derivative of the square root at zero is infinite), and the formula $\|u\|^2-2u^\top v+\|v\|^2$ may produce a tiny negative number due to rounding. So $\max(\cdot,\varepsilon)$ is put under the root, and nn.TripletMarginLoss adds $\varepsilon=10^{-6}$ to the difference of vectors. It also has swap=True, the distance swap (Balntas et al., 2016): $\min\bigl(d(a,n),\,d(p,n)\bigr)$ is used instead of $d(a,n)$, so the negative is compared with the closer of the two same-identity points.

pythonimport numpy as np

def pairwise_dist(E, eps=1e-12):
    """Matrix of Euclidean distances between the rows of E (n × d)."""
    sq = (E * E).sum(1)
    d2 = sq[:, None] - 2.0 * E @ E.T + sq[None, :]
    return np.sqrt(np.maximum(d2, eps))     # max: guards against −1e-16 and the root of zero

def batch_hard_triplet(E, labels, margin=0.3):
    D = pairwise_dist(E)
    same = labels[:, None] == labels[None, :]
    hardest_pos = np.where(same, D, 0.0).max(1)       # the farthest positive
    hardest_neg = np.where(same, np.inf, D).min(1)    # the closest negative
    has_pos = same.sum(1) > 1                         # skip anchors that have no positive in the batch
    loss = np.maximum(hardest_pos - hardest_neg + margin, 0.0)
    return loss[has_pos].mean()

E = np.array([[0.0, 0.0], [0.9, 0.0], [0.6, 0.8]])   # anchor, positive, negative
labels = np.array([0, 0, 1])
print(batch_hard_triplet(E, labels, margin=0.2))      # ≈ 0.1728 = (0.1 + 0.2456) / 2: the negative has no positive

pythonimport torch
import torch.nn as nn
import torch.nn.functional as F

# 1) Triplets prepared in advance: the built-in loss (defaults: margin=1.0, p=2, swap=False)
loss_fn = nn.TripletMarginLoss(margin=0.2, p=2)
a = torch.tensor([[0.0, 0.0]], requires_grad=True)
p = torch.tensor([[0.9, 0.0]], requires_grad=True)
n = torch.tensor([[0.6, 0.8]], requires_grad=True)
loss = loss_fn(a, p, n)            # max(0, 0.9 − 1.0 + 0.2) ≈ 0.1
loss.backward()
print(p.grad, n.grad)              # ≈ [1, 0] and [−0.6, −0.8]: unit vectors

# 2) Mining inside a P × K batch (batch-hard)
def batch_hard(emb, labels, margin=0.3, normalize=True):
    if normalize:
        emb = F.normalize(emb, dim=1)          # points on the unit sphere
    D = torch.cdist(emb, emb)                  # n × n Euclidean distances
    same = labels[:, None] == labels[None, :]
    pos = (D * same).max(1).values             # the farthest positive
    neg = D.masked_fill(same, float("inf")).min(1).values   # the closest negative
    has_pos = same.sum(1) > 1                  # skip anchors that have no positive in the batch
    return F.relu(pos - neg + margin)[has_pos].mean()

Common mistakes

1. Random batches instead of $P\times K$. In a random batch many anchors have no positive at all, and no triplets can be formed.
2. Training on random triplets only. Almost all of them are easy, the gradient is zero, and the loss “freezes” at a small value although the model is poor.
3. The hardest triplets from the very first step without normalization or warm-up — a risk of all embeddings collapsing to a point.
4. A margin that ignores the scale. Without normalization the network stretches the space; on the sphere $m$ has to be related to the maximum distance (2 for $d$, 4 for $d^2$).
5. The square root of zero. The distance of a point to itself on the diagonal of the matrix gives a NaN gradient unless $\varepsilon$ is added or the diagonal is masked.

Links to other formulas

The closest relative is the contrastive loss: it works with pairs and absolute thresholds rather than a relative comparison. For a pair labelled $y=1$ (same) or $y=0$ (different):

• ReLU: the hinge $\max(0,\Delta+m)$ is a ReLU of the shifted difference of distances; its derivative is a step.
• Sigmoid: the derivative of the soft margin $\ln(1+e^\Delta)$ is $\sigma(\Delta)$.
• Softmax and cross-entropy: the N-pair loss and InfoNCE generalize the triplet to one positive against many negatives via a softmax over similarities; ArcFace moves the margin into an angle inside the softmax.
• MSE: the squared Euclidean distance is the same sum of squared differences.
• Focal Loss: the same idea of “learning from the hard ones”, through a per-example weight rather than triplet selection.
• Adam: the optimizer of the simulation in Figure 6.

Summary

Check yourself