Loading
Choosing the most confident head picked a positional one every time. It reported “it → was, 0.962” — the answer that head gives for every word in every sentence.
Open the instrumentThe first version of the interface picked the head with the sharpest row and showed you that. It seemed obviously right: of sixteen possible answers, show the most confident one.
It reported it → was, 0.962.
That number is enormous. It is also completely worthless, because that head answers "the word immediately after" for every word in every sentence you could type. It was not confident about it. It is confident about everything, in the same way, always.
The decisive idea: confidence and informativeness are different axes. The most certain head in a transformer is usually the one that has stopped paying attention to the question.
Sixteen heads — four layers, four each — trained with no instruction to divide the work and no penalty for duplicating it. They divide it anyway.
Some become positional: their weight lands on the neighbouring word regardless of which word you asked about. Others stay content-driven: their answer moves when the question moves.
You can separate the two with one statistic. For every content word, add the weight the head places on the word immediately before and the word immediately after, then average over the sentence. Call it adjacency bias. A head that always answers "next" scores near 1. A head whose answer depends on the query scores low.
Adjacency bias is the mean weight a head puts on the immediately neighbouring words, over every content row. A head above 0.35 answers “the next word” no matter what you asked, so the instrument skips it when choosing what to show.
Measured across five sentences, the sixteen heads separate cleanly, and the gap between the two populations is wide enough to put a threshold in. Above 0.35, a head is treated as positional and skipped when the interface picks what to show — leaving eleven in play on the sentences the threshold was set from.
The count is not fixed, and the figure above shows why: a head's adjacency bias is measured on the sentence in front of it, so a borderline head can cross the line on one sentence and not another. On "The keys to the cabinet are on the table", six heads clear 0.35 rather than five, because L2H3 sits at 0.434 here. The threshold is a property of the model; which heads exceed it is a property of the model and the sentence.
Choosing that number was not taste. The measured spread has an empty band between the lowest positional head and the highest content head, and 0.35 sits inside it. A statistic that jitters would make the chosen head flicker between sentences for no reason a reader could see — which is why adjacency bias won over the more obvious alternative of counting how often the argmax offset repeats. On an eight-word sentence that count swings fifteen points; adjacency bias moves two to six.
Click through the words. A positional head gives the same structural answer every time — that is exactly why confidence is a bad way to choose one. A distant head's answer depends on which word you asked about.
Click through the words. The positional head's answer marches along with you at a fixed offset — it is describing the sentence's geometry, not its meaning. The distant head's answer jumps around, because it is actually responding to what you asked.
Both rows are real attention. Both sum to 1. One of them is informative about the word in focus and the other is not, and no amount of staring at the weights alone will tell you which is which. You need the statistic.
With positional heads out of the running, the same sentences started producing answers a reader can check:
| Query | Answer | Weight |
|---|---|---|
are | keys | 0.499 |
who | Students | 0.700 |
that | Birds | 0.860 |
suitcase | trophy | 0.612 |
are → keys is subject–verb agreement across four words. that → Birds is a relative pronoun finding its antecedent. These are weaker numbers than 0.962, and they are worth vastly more, because they vary with the sentence in ways you can verify by reading it.
The trade is uncomfortable but correct: a smaller number that means something beats a larger number that means nothing. An interface optimising for impressive weights would have kept the positional heads and looked more confident while saying less.
Skipping five heads is a choice. It is disclosed in the interface, and it is defensible, but a visitor is being shown one of eleven remaining answers — selected by a rule I wrote, using a threshold I set from measurements on five sentences.
That is a long way from "here is what the model thinks." It is closer to: here is the head that says the most specific thing about the word you clicked, among the heads that are not merely reciting word order.
Naming the heads helped more than I expected. "Layer 3 · Head 2" tells a visitor nothing. "Layer 3 · Head 2 · distant" makes a claim they can check by clicking through words and watching whether the answer stays put.
Which raises the obvious next question: if there are sixteen answers, why not just average them? There is a standard technique for exactly that, it is mathematically correct, and on a four-layer model it is nearly useless. Part 7 is about why.
Next: running BERT in a browser with no inference engine at all.
More to read
Three matrices, one dot product, one softmax. The row always sums to 1 — which is a constraint the model must satisfy, not a finding it reports.
Naive int8 held attention parity at 2.97e-2 — thirty times worse than the bar. Where you spend precision is the whole engineering story.
Nine entries in the vocabulary could not survive being typed. The bug was invisible for weeks because the output looked completely reasonable.
