Learning the Metric Itself

Every picture on this page is sixteen pixels on a side with one brightness per pixel, and holds one of four shapes, a cross, the outline of a square, a filled disc or a diagonal bar, drawn at a random position and size against a random background with a little noise on every pixel. A fifth kind, a ring, is drawn only when asked for, as a kind no network here is ever shown.

We want a distance between two pictures that is small when they are the same kind and large when they are not, and the method on this page trains that distance directly from pairs, where the usual route borrows it from a network trained to name the kinds.

The page asks six questions, in order. What goes wrong with the distance a classifier leaves behind, and why train a distance from pairs at all? What does the contrastive loss ask of one pair, worked by hand? How does one network serve both pictures of a pair, and is the slope of a weight used twice really the sum of its two uses?

Trained on pairs of three kinds, does the distance carry to the fourth kind and to rings, against the vector borrowed from a classifier? What does the margin do, and what happens with no pairs of different kinds at all? And what does the method cost in pairs, before the last Part asks where it stops being defined.

The page on a vector for a picture trained a network to name the four kinds and kept the sixteen numbers of the layer before its answer as a picture’s position. It found that a held-out picture’s nearest neighbour there is its own kind for 230 to 235 of 240 across three weight seeds, which is a very good ruler for the kinds the network was taught. It also found that the ruler is worse at the one thing it was never asked about, since 43 to 45 of 60 rings find another ring by it where the raw pixels manage 46.

Searched among all 300 pictures by straight-line distance, the same network’s layer gives a nearest neighbour of the right kind to 56 crosses, 58 discs and 59 bars of 60, and to only 40 squares, because the rings it was never shown land among the squares and take their places as nearest neighbours. The pixels give 45, 38, 50 and 54, and 44 rings of 60.

A classifier needs every example named, and it can only ever answer with one of the names it was given. A great many real problems do not come like that. A bank verifying signatures has pairs known to be by one person and pairs known not to be, and a photograph collection has pictures known to show one face, and in both the people who matter tomorrow are not among the names today. What they have is sameness, and sameness is exactly what a distance is supposed to encode.

So the method asks for it directly. Show the network two pictures and a single bit, one kind or two, and adjust the weights so that the distance between the two answers agrees with the bit.

d(a, b) as small as it can be, when a and b are one kind
d(a, b) ≥ m, when they are two kinds

The hope, and the claim the method is usually sold on, is that a distance trained this way describes what makes pictures alike rather than which name to give them, and so should serve kinds the training never contained. Part 4 tests exactly that.

The network is the shared picture network with its last layer replaced. Where the classifier ended in four scores, one per kind, this one ends in eight numbers with no nonlinear activation applied, and those eight numbers are the picture’s position. Every starting weight below that last layer is the reference network’s own, so the two networks this page compares begin from the same place.

p(x) = W₂ tanh(W₁ f(x) + b₁) + b₂      f(x) the 128 numbers the convolutions hand on
                                         W₁ is 16 × 128, W₂ is 8 × 16
d(a, b) = ‖p(a) − p(b)‖ = √( Σᵢ (p(a)ᵢ − p(b)ᵢ)² )

Before any training the mean distance between two of the 300 evaluation pictures’ positions is 0.7325. The margin used throughout is 1, chosen before any margin was compared because it is a distance the untrained network nearly reaches already, so training does not have to drag the positions to a new scale before it can start arranging them.

For two pictures of one kind the loss is half their squared distance. It is zero only when the two positions coincide, and its slope at either position points along the line to the other, growing with the distance, so a far-apart pair is pulled hard and a close pair gently, the way a spring behaves.

L_same(a, b) = ½ d(a, b)²
∂L_same / ∂p(a) = p(a) − p(b)

A step of descent moves p(a) against that slope, which is towards p(b), and moves p(b) by the opposite amount, towards p(a). The half is there only so the slope comes out as the gap itself.

For two pictures of different kinds the loss is half the squared amount by which they fall short of the margin. Closer than the margin they are pushed apart, hardest when they are closest; once they are a margin apart the loss is zero and so is its slope, and they are left alone.

L_different(a, b) = ½ max(0, m − d)²
∂L_different / ∂p(a) = −((m − d) / d) (p(a) − p(b))   when d < m,   and 0 when d ≥ m

Together, with y equal to 1 for a pair of one kind and 0 for a pair of two, the contrastive loss of Hadsell, Chopra and LeCun is the following, averaged over the pairs.

L = y · ½ d² + (1 − y) · ½ max(0, m − d)²

Two positions in a plane are enough to check every piece of the loss. Put the first at the origin and the second at 0.36 and 0.48, which is 0.6 away, and take the margin as 1.

Drag either point and switch the label. The curve on the right is the loss over distance for each label, with the margin dashed; the dot is the answer for the pair on the left.

The same arithmetic on real pictures, placed by the network once it has been trained on pairs. Two held-out crosses land close together and a cross and a square land far apart, and the loss treats each pair the way its label says.

…

Both pairs share the same first cross. Each pair’s eight numbers are drawn side by side, and the table gives the gap the loss reads.

same-kind loss ≈ ½ × 0.1536² ≈ 0.0118
different-kind shortfall ≈ 1 − 0.1536 = 0.8464
different-kind loss ≈ ½ × 0.8464² ≈ 0.3582

The loss needs a position for each picture of a pair, so each picture is run through the network, and it is the same network both times. Drawn out, that looks like two identical towers joined at the top by the distance, which is why the arrangement is called a Siamese network, though there is only one set of weights and nothing about it is doubled except the drawing.

Using one set of weights is forced by what a distance has to be. If the two pictures went through different networks, the distance from a to b would differ from the distance from b to a, and a picture’s position would depend on which side of a pair it happened to be shown on, so there would be no single place to put it when it is later searched for.

The loss of one pair is a function of two positions, and each position is a function of the same weights. Training needs the slope of the loss with respect to each weight, which the chain rule gives as one term through each picture.

∂L/∂W = (∂L/∂p(a)) · (∂p(a)/∂W) + (∂L/∂p(b)) · (∂p(b)/∂W)

Each term is an ordinary backward pass. The slope of the loss at one picture’s position arrives at the top layer, each layer turns the slope at its answer into slopes for its own weights and a slope at its input, and hands the second down to the layer below, exactly as backpropagation does for one picture. The layers themselves only know losses that score one answer against a target, so the pair loss and its slope at the two positions are computed outside the layers, and the walk down each picture’s copy is done by hand from there.

That the two slopes should be added is the whole point of the arrangement, and it is easy to get wrong, by stepping each copy on its own slope and ending up with two copies that no longer agree. The check is to nudge a weight up and down by a tiny amount, run both pictures through the nudged network again, and measure how much the pair loss changed. That measurement knows nothing about copies.

…

Four pairs through the untrained network. For each of five weights, the slope through each picture’s copy, their sum, and the slope measured by nudging.

On four pairs through the untrained network, with a mean loss of 0.2056, the summed slope and the measured one agree on every weight checked, in both convolutions, the hidden layer and the last layer, to within 1.1 × 10⁻¹¹, with the weight nudged by 10⁻⁵ each way. Either copy’s slope alone is wrong by between 0.0136 and 0.1462. Here one network is run twice and stepped once, with the sum, so there is only ever one copy of every weight to step.

The last row of the table above is a bias of the layer that produces the position, and its summed slope is exactly zero, not merely small. Through the first pictures’ copy its slope is 0.1462 and through the second it is −0.1462. The reason is that a bias adds the same amount to every position, and the loss only ever reads the difference between two positions.

(p(a) + c) − (p(b) + c) = p(a) − p(b)

So a shift applied to every position changes no distance and no loss, and nothing in training can decide where the collection as a whole sits. The same is true of turning or reflecting every position together. This is why the drawing at the top of the page moves the middle of the collection to the origin before it measures any direction, and it comes back in the last Part.

Walking two copies for every pair would cost two forward and two backward passes per pair. Training here takes eight pictures of each of three kinds, 24 pictures in a batch, which hold 276 pairs, 84 of them of one kind and 192 of two, and runs the 24 pictures through once.

∂(Σ over pairs Lᵢⱼ) / ∂p(k) = Σ over the pairs that contain k of ∂Lᵢⱼ / ∂p(k)

Because the slope of a sum is the sum of the slopes, a picture’s slope from every pair it belongs to can be added at its position first, and then one backward pass over the 24 pictures gives the same gradient that 276 pairs of separate walks would have summed to. Each picture appears in 23 pairs and is walked down once. That is the two copies’ sum from step 11 again, computed once per picture instead of once per pair.

To ask whether a distance learned from sameness reaches new kinds, the network is trained on pairs of crosses, squares and discs only, 180 training pictures, and the bar is held back along with the ring. For a fair comparison a classifier of the reference design is trained on the same 180 pictures to name the same three kinds, from the same starting weights, and its layer of sixteen is read the way the neighbouring page reads it.

Both train properly. The pair loss falls from 0.1351 in the first epoch to 0.0122 in the fortieth, and the three-kind classifier names 0.9778 of the held-out pictures of its three kinds correctly. So four spaces are compared, the pixels, the reference classifier’s layer (which was shown bars), the three-kind classifier’s layer and the positions trained on pairs, and every number depending on training is given for weight seeds 0, 1 and 2. The widgets draw seed 0 unless a button says otherwise.

First the easy question. Searching only among the 180 held-out pictures of crosses, squares and discs, so that no picture of an unseen kind can be anyone’s nearest, the positions trained on pairs do about as well as the classifier’s layer.

Nearest neighbours of the right kind among the held-out pictures of the three trained kinds only, for each seed.

On seed 0 the positions give 171 of 180 pictures a nearest neighbour of their own kind against the three-kind classifier’s 179, and on seed 2 they are ahead, 178 against 176. Seed 1 is the weak one, 157 against 174. The pixels manage 144 and the reference classifier 174.

Now the bars and rings join the search, 300 pictures in all. This is where the claim for the method is made, and on these pictures it did not hold. A held-back bar finds another bar as its nearest picture far less often in the positions trained on pairs than in the three-kind classifier’s layer, and the bars also spoil the crosses.

Every kind in every space, searched among all 300. Amber cells are kinds that space’s network never saw. The reading buttons change what each cell measures.

In the positions trained on pairs 33, 27 and 34 bars of 60 find a bar across the three seeds, where the three-kind classifier’s layer gives 52, 51 and 53 and the raw pixels 54. The crosses suffer too. Searched among their own three kinds 56 crosses found a cross, and with the bars and rings added only 39 do, because the network, having never been asked where a bar should go, puts many of them where the crosses are.

The playground at the top shows it; take a bar and look at its five nearest.

The ring is the stricter test, since no network on this page saw one, the classifiers included. It is also the place where the method is plainly worse than doing nothing, which is worth stating as found rather than explaining away.

In the positions trained on pairs 33, 33 and 39 rings of 60 find another ring across the three seeds. The three-kind classifier’s layer gives 46, 51 and 43, the reference classifier 43, and the raw pixels 44. So on every seed the distance trained directly for sameness gathers the unseen kind less well than comparing the pictures pixel by pixel.

The bar might simply be a hard kind to hold back. It is not the reason. Holding back the cross, the disc or the square instead, each on seed 0, gives the same picture, and a wider answer does not rescue it either.

The positions have eight numbers where the classifier’s layer has sixteen, so I retrained the pair network on seed 0 answering with sixteen. Its bars find a bar 31 times of 60 where the eight-number version managed 33, and its crosses 37 times where it managed 39. Its rings do better, 43 of 60 against 33, which matches the reference classifier and is still below the pixels. So the narrower answer costs the rings something and is not what goes wrong with the held-back kind.

There is a way of reading the same positions that makes the method look as advertised. Take a kind’s mean distance between its own pictures and divide it by its mean distance to every other picture. Lower is tighter, and on that reading the rings are tighter in the positions trained on pairs than in the three-kind classifier’s layer in all six studies.

ratio(kind) = mean d inside the kind / mean d from the kind to everything else

On seed 0 the ratio for rings is 0.5230 against 0.5984, on seed 1 0.4431 against 0.6945, and on seed 2 0.3532 against 0.7923, the opposite verdict to the nearest-neighbour count. The ratio has a confound. The pair loss pushes the three trained kinds a margin apart from each other, which makes the whole collection larger, and that enlarges every kind’s distance to everything else, so the ratio falls for a kind that did not gather at all.

The nearest-neighbour count cannot be moved that way, and it is the question a search actually asks, which is why this page reads it first. The reading buttons on the table in step 16 show all three.

The margin looks like a setting that says how far apart kinds should be, and it is less than that. Nothing about a position has a natural size, so a network trained at a margin of 2 can reach exactly the arrangement it would reach at a margin of 1, with every position doubled.

The loss surface at a margin of 2 is the one at a margin of 1, stretched. What does not stretch is everything else. The starting positions are the same size whatever the margin, and the slopes reaching the lower layers grow with the margin, so at one step size a larger margin takes larger steps. Dividing the step size by the square of the margin undoes the c² and is the fair way to compare margins.

Ignoring that has a striking consequence. Trained at the step size that suits a margin of 1, a margin of 2 or 4 does not give a wider arrangement. It gives none at all, because every picture’s position lands on the same point.

The widget opens on a margin of 2 at the unchanged step size. The loss stops falling and the mean distance between positions drops to almost nothing; try a margin of 4, then switch the step size to scaled.

With margin two, all embedded positions collapse to nearly one point by epoch fifteen. With margin four, they do so after the first epoch. Once every distance is zero, same-kind pairs cost nothing and different-kind pairs cost half the squared margin. There are 192 different-kind pairs among 276 pairs.

mean loss at margin 2 = (192 / 276) × ½ × 2² ≈ 1.3913
mean loss at margin 4 = (192 / 276) × ½ × 4² ≈ 5.5652

Those are the losses where the runs stall. At exactly zero distance, this implementation returns a zero positional gradient because the distance has no unique direction there. The same failure occurs with margin one half on seed one when the step size is raised to four.

collapsed loss at margin 0.5 = (192 / 276) × ½ × 0.5² ≈ 0.0870

With the step size divided by the margin squared, every margin trains. The arrangement then really does scale with the margin, and whatever else changes is small and changes its direction from one seed to the next.

On seed 0 the rings improve steadily with the margin, from 32 of 60 to 49, and at a margin of 4 they beat the pixels’ 44. It is tempting to conclude that a wide margin keeps room for the unseen. The seed buttons on the widget above say otherwise, since on seeds 1 and 2 the rings at a margin of 4 do no better than at 2, and on seed 1 a margin of 0.5 freezes. I would not trust any trend in the margin from three seeds of this size.

Pairs of one kind are the ones that say what sameness is, so it is natural to wonder whether the pairs of two kinds are needed at all. They are, and the measurement is immediate. Trained on the same batches with every two-kind pair left out, from the same starting weights, the network does the one thing that satisfies every pull at once, which is to give every picture the same position.

…

The two runs share their starting weights and their batches. Only the pairs of two kinds differ.

The mean distance between positions starts at 0.7325 in both runs. With pulls only it is 0.0393 after one epoch and 0.0018 after forty, while the ordinary run ends at 0.8284. The collapsed positions still differ in their last digits, so a nearest neighbour can still be found, and it means almost nothing. The share of each picture’s ten nearest that are its own kind falls to 0.2917 for crosses, 0.3000 for squares and 0.2633 for discs, close to the one in five that a picture chosen at random from the other 299 would give.

The alarming part is what the loss says while this happens. The run with pulls only reaches a mean loss of 0.0000078 by the fortieth epoch, where the ordinary run ends at 0.0122, so by its own measure the collapsed network is more than a thousand times better trained.

A classifier learns from each picture once per epoch. A loss over pairs has one term for every two pictures, and the number of pairs grows with the square of the number of pictures, which is harmless at the scale of this page and decisive at the scale of a real collection.

pairs among n pictures = n (n − 1) / 2

…

Left, the pairs among this page’s training pictures and what one training actually computed. Right, how the count grows.

The 180 training pictures make 16,110 pairs, of which 5,310 are of one kind and 10,800 of two. A thousand pictures make 499,500 and a million make 499,999,500,000, so at any real size most pairs are never looked at, and which pairs are shown becomes a decision rather than a detail.

At this size the training can afford to visit almost every pair. Forty epochs of seven batches of 276 pairs compute 77,280 pair losses, and those cover 16,015 of the 16,110 different pairs, 0.9941 of them. In the run I measured the forty epochs took about 1.4 seconds against about 2.3 for the three-kind classifier, since batching every pair makes each step one pass over 24 pictures; the widget above reports the times on whatever machine answers, and they will differ.

Of the pairs that are computed, most soon contribute nothing. A pair of two kinds past the margin has a loss of zero and a slope of zero, and once training has done its work that describes most of them. At the end, 0.7909 of all two-kind pairs among the training pictures are past the margin, and 0.6620 among the held-out ones, so roughly four in five of the pushes each batch computes are multiplications by zero.

That is the problem FaceNet met at scale and answered with a change of loss and a search. Its loss takes a triplet, a picture, another of the same kind and one of a different kind, and asks that the first two be nearer than the first and third by a margin α.

L = max(0, ‖p(a) − p(same)‖² − ‖p(a) − p(different)‖² + α)

Its authors then chose, within each batch, the different-kind pictures that were close enough to still violate the margin but not closer than the same-kind one, because random triplets were almost all satisfied already. None of that is built on this page; with 180 pictures every pair can simply be visited.

A loss that reads only distances cannot tell apart two arrangements that have the same distances. Shifting every position by one amount, turning the whole arrangement, or reflecting it leaves every distance unchanged and so every loss, and step 12 measured the first of these as a slope of exactly zero on the last layer’s biases. A trained position is therefore defined only up to a rigid motion, and a single coordinate of it means nothing.

Two consequences follow. Positions from two separate trainings cannot be compared with each other at all, since each training settles on its own orientation, and a direction measured from the origin is an accident of the starting weights, which is why the drawing at the top measures directions from the middle of the collection instead.

The method asks for two things of its data, pairs of both labels, and labels that some arrangement of points could satisfy. Where either fails, the loss does not become an approximation. It becomes a quantity with no single minimiser, or a slope that is not defined, and the table says which and what has to be decided.