Loading
I wrote the code and the tests from the same wrong idea, and they agreed with each other perfectly. Everything was green.
Open the instrumentWhen a sort function is wrong, you can see it. The list is not sorted.
When an attention matrix is wrong, it looks like an attention matrix. Every value is between 0 and 1, every row sums to 1, the numbers vary plausibly across words. A subtly broken transformer produces output indistinguishable, to a human, from a correct one.
So the usual test-writing instinct — run it, look at the output, assert what you see — is not available. Worse, it is actively dangerous, because the output looks fine.
The decisive idea: you cannot test a numerical system against your intuition. You test it against a second implementation that shares nothing with the first, including the assumptions of whoever wrote it.
I wrote the WordPiece tokenizer. Then I wrote tests for it. Both came out of the same mental model of what tokenization does.
That model was wrong in one specific place, and so I produced a wrong implementation and a wrong test that agreed with each other perfectly. Green suite, confident deploy, and £5 quietly reaching the model as £ + 5 for weeks.
No amount of additional tests written by me, that day, would have caught it. They would all have been written from the same misunderstanding.
What broke the deadlock was building a second bert-mini that shares no code with the first: written in Python instead of TypeScript, from the published description rather than from the existing implementation, running against the original fp32 HuggingFace weights in fp64 NumPy. Different language, different arithmetic, different day, different assumptions.
When two independent implementations disagree, exactly one of two comfortable beliefs survives, and you have to find out which.
It disagreed on £5. The oracle was right.
And once, the oracle was wrong. Its first version lacked the step where HuggingFace pads every CJK character with spaces so each becomes its own word. It disagreed with the shipped runtime on 中文 — and that time the runtime was correct. A committed fixture settled it.
That episode changed how the oracle is used. An unvalidated oracle is just a second opinion, and a confidently wrong one is worse than none, because it sends you to fix code that was already right. So the oracle now reproduces a committed PyTorch reference to 1.324e-6 before it is permitted to judge anything, and that check runs at the start of every comparison rather than living in someone's memory.
With a trustworthy oracle, the question becomes what to compare on. Nine hand-picked sentences is what the first audit used, and it is not enough — hand-picked sentences share the biases of the hand.
The corpus is now 64 sentences chosen to break things: symbols, contractions, possessives, CJK, emoji, accented Latin, German and French, file paths, email addresses, single words, repeated words, tabs, a word that shatters into thirteen pieces, sentences at the token limit, and the recorded reference pair. Every stage of the pipeline is compared, not just the final numbers:
oracle vs PyTorch reference: 1.324e-06
sentences 64 compared 64 over token limit 0
worst abs error attention 2.251e-04 (bar 1e-03)
worst abs error rollout word mx 5.394e-06 (bar 5e-05)
worst abs error rollout sink 1.442e-05 (bar 1e-04)
worst abs error per-head word mx 1.700e-04 (bar 6e-04)
measured pairs quoted in prompts.ts: 29 hold
PASSThe strongest single test is also the least clever. Take all 30,522 vocabulary entries, feed each one back through the tokenizer alone, and require it to come back as itself. That found the class of bug rather than an instance, and it is why I know there is no tenth broken entry hiding behind the nine.
The first version of that comparison script printed the rollout and per-head errors and asserted nothing about them. Only attention had a threshold. So I perturbed the dump by 1e-4 to see what happened.
It printed PASS.
Two entire stages were being reported and never checked. The numbers scrolled by every run, looked reassuring, and would have gone on looking reassuring through any regression that did not touch attention.
Every stage now carries a bar, and every bar was verified by deliberately breaking something:
Each button corrupts one value in the dump the checker reads. A check that cannot fail is not a check, so every stage was verified this way before the numbers were trusted.
Six mutations, six failures. Plus a seventh: an oracle that has drifted from the PyTorch reference aborts the run before any comparison happens.
Two implementations is double the work, and it is only worth it when your intuition genuinely cannot referee the output. For a sort function it would be absurd. For a quantized transformer it is the difference between believing your numbers and knowing them.
The tests here that caught real bugs were not the ones asserting expected values. A previous session typed three "expected" token ids from memory and two of them were wrong — a test that would have locked in the error had anyone trusted it. The rule that came out of that is blunt: never hand-write an expected value. Derive it from the oracle, or from a committed reference, or do not assert it.
Your intuition is a hypothesis generator. It is not a measuring instrument, and on anything numerical it should never be the last thing standing between a bug and production.
Next: a published fix for the sink problem, measured — and declined.
More to read
Every rollout row in the sentence sits between 0.935 and 0.971 normalized entropy. That is a uniform distribution wearing a ranking, and it shipped as the default view.
It halves the problem and does not dissolve it. [CLS] carries 0.63× the value norm of a content token — not the near-zero the paper reports. Four layers is not twelve.
A median 24% of every displayed row had already been deleted before you saw it. In the worst case, 95%. The arithmetic was right the whole time.