Loading
It halves the problem and does not dissolve it. [CLS] carries 0.63× the value norm of a content token — not the near-zero the paper reports. Four layers is not twelve.
Open the instrumentPart 6 described a problem: a median 24% of every displayed attention row disappears into [CLS] and [SEP] before you see it, worst case 95%. The interface discloses this. Disclosure is honest, but it is an apology, not a repair.
There is a published repair. This post is about testing it, and about the answer being no.
The decisive idea: what flows from one token to another is not the attention weight. It is the weight times the vector it carries — and a token that attracts enormous weight while carrying almost nothing has effectively switched itself off.
Kobayashi et al., EMNLP 2020, Attention is Not Only a Weight. The argument: a head's output is the sum over j of α·v — attention weight times value vector. Looking at α alone ignores the magnitude of what is being carried.
Measure ‖α·v‖ instead, and their headline finding is that BERT pays poor attention to special tokens. Separators attract large weight but carry small value norms, so their real contribution is far below what α suggests.
If that held here, the sink would evaporate, the disclosure chip could go, and the interface would stop apologising.
The project's own rule made the next step obvious: measure before building. And set the kill condition in advance, so the result cannot be rationalised after the fact — if the sink share does not collapse, drop the idea and say so.
The >90% band empties and the median halves. The worst row goes 97.7% to 83.4% — better, and still not a picture you would show without a warning.
Sink share across the 110 rows the instrument actually shows, under attention weight and under the norm-weighted measure. Full method in evidence/ex3-norm-weighted-attention.md.
Across the 110 rows the instrument actually shows — one per word, on the head it would open, aggregated exactly as it aggregates:
| Attention α | Norm-weighted ‖α·v‖ | |
|---|---|---|
| Median sink share | 19.9% | 10.0% |
| Worst row | 97.7% | 83.4% |
| Rows above 50% | 25 | 12 |
| Rows above 90% | 8 | 0 |
The median halves. The above-90% band empties completely.
And the worst row goes from 97.7% to 83.4%.
That is the number that decided it. The pitch was "the sink visibly evaporates when you switch lenses." It does not. It shrinks from catastrophic to bad. Twelve rows still lose more than half their mass, so the disclosure stays, the warning stays, the sink tagging stays — and now there is also a toggle to explain.
Also worth stating because it cuts against the idea: 31 of 110 rows get worse, by up to 26% relative.
The premise holds directionally and fails quantitatively, and one measurement explains everything.
Over 14 sentences × 16 heads, the value norm of a special token relative to the mean content token:
| Token | Median | Below 1.0 |
|---|---|---|
[CLS] | 0.63× | 196 of 224 |
[SEP] | 0.70× | 147 of 224 |
Special tokens do carry smaller values. But by a factor of about 0.6 — not the near-zero Kobayashi reports for BERT-base.
A 0.6× downweight cannot rescue a 95% row. It lands it at 73%. That single ratio is the entire result.
Their measurement was on twelve layers and 768 hidden dimensions. This is four and 256. The effect is a function of depth, and it does not transfer at this scale. The paper is not wrong; it is describing a bigger model.
Two side findings. Including the output projection — Kobayashi's actual f(x), not just ‖v‖ — buys nothing: median 10.3% versus 10.0%. And head selection is stable, the same head chosen on 98 of 110 rows, so the improvement comes from the measure itself rather than from dodging sink-heavy heads.
A negative result is exactly the kind of finding that deserves suspicion, because it justifies not doing work.
The oracle was re-validated against the PyTorch reference first: 1.324e-6. The runtime was temporarily instrumented to return ‖v‖ per head, and the two implementations compared over all 14 sentences — attention agreeing to 1.204e-4, ‖v‖ to 7.272e-5 relative. The instrumentation was then reverted; nothing from it shipped.
The port was checked behaviourally before it was used for anything new, by reproducing three numbers already recorded in the repository: the → scissors 0.431 at 95.2% sink, for → scissors 0.617 at 0.0426 of the row, doctor → nurse 0.607.
And it was mutation-tested. Forcing every value norm to 1.0 reproduces the α numbers to the last digit; zeroing [CLS]'s norm moves them. The measure is responding to v and not quietly re-reporting α.
A second lens would still need the disclosure chip, still need the above-50% warning, still need the sink tagging — and would add a toggle and a protocol change to a panel already carrying four caveats. The median improvement is real and invisible to someone reading a single row.
So it was not built, and the finding is the deliverable. "We tried the standard published fix against our own model and measured what it was worth" is a rarer and more useful thing to be able to say than a toggle would have been.
Reopen it if the model ever grows. The effect is a function of depth, and the measurement is now cheap to repeat.
Next, and last: what attention can and cannot tell you.
More to read
I wrote the code and the tests from the same wrong idea, and they agreed with each other perfectly. Everything was green.
Swap the one adjective that decides what “it” refers to. A reader's answer flips. Not one of the sixteen heads changes its winner.
Every rollout row in the sentence sits between 0.935 and 0.971 normalized entropy. That is a uniform distribution wearing a ranking, and it shipped as the default view.
