Loading
Naive int8 held attention parity at 2.97e-2 — thirty times worse than the bar. Where you spend precision is the whole engineering story.
Open the instrumentThe obvious way to run a transformer in a browser is ONNX Runtime Web. That is what this project used first, and it worked.
It also cost a WebAssembly runtime in the bundle, a Content-Security-Policy exception to allow WASM compilation, and a dependency whose release cycle I did not control — all to execute four transformer blocks on a sentence of at most twenty-four tokens.
So I deleted it and wrote the forward pass by hand.
The decisive idea: a four-layer model does not need a general-purpose inference engine. It needs about seven hundred lines of arithmetic over
Float32Array, and then every constraint the engine imposed disappears.
The whole runtime is now plain TypeScript: embeddings, layer norm, three linear projections per layer, scaled dot products, a softmax per row, the weighted value sum, a two-layer feed-forward with exact GELU, and residual connections. Roughly 770 lines including the WordPiece tokenizer.
For a twelve-word sentence, that is tens of milliseconds. It runs in a module Web Worker, so the main thread never blocks and the animation stays at frame rate while the model runs.
What it bought:
'wasm-unsafe-eval'. The CSP tightened to script-src 'self', and the browser test now asserts the header does not contain the exception it used to require.Writing the forward pass took an afternoon. Making it agree with PyTorch took considerably longer, and it is the only part worth writing about.
The weights ship quantized — int8, with one scale per output channel rather than one per tensor, because a single scale loses too much on the feed-forward layers. Channel scales cost four bytes per row and are worth every one.
The bar: worst-case attention error under 1e-3 against an fp32 PyTorch reference, which keeps the third displayed decimal trustworthy. If the interface prints 0.612, the model's own answer must round to 0.612.
Naive all-int8 measured 2.97e-2. Thirty times over the bar. Attention was visibly wrong.
Worst-case attention error against fp32 PyTorch, measured at each step of the quantization design. The bar is 1e-3. Numbers from the parity gate in engine/bert.parity.test.ts.
What fixed it was not uniform precision. It was finding out where error becomes visible and spending bytes only there.
Position and token-type embeddings in fp32. These are tiny tables, and their error enters layer 0's logits directly, before any block has run. At int8 they held parity at 9.4e-3 all by themselves — the network hadn't done anything yet and was already coarse.
Q and K in int16. Their error is amplified by the softmax's exponential. A small perturbation in a logit becomes a large perturbation in a probability.
V and the attention output in int16. These write into the residual stream, so their error propagates to every subsequent layer.
Feed-forward int16 for every layer except the last. The final layer's feed-forward feeds no attention at all — nothing downstream reads it — so it stays int8 for free.
Final measurement: 1.218e-4, comfortably inside the bar, at a 6.32 MiB cold load.
Each rung above is a hypothesis that was measured, not a rule of thumb. "Quantize everything to int8" and "keep everything in fp32" are both easy and both wrong — the first breaks the output, the second wastes megabytes on tables that could not care less.
The useful question is never "what precision?" It is "which tensors does the error escape from, and where does it get amplified?" Softmax amplifies. Residual connections propagate. A dead-end feed-forward does neither.
That is knowable only by measuring per layer. The parity test reports error at each layer separately for exactly this reason: if layer 0 is already coarse the embedding path is at fault, and if error grows with depth the block arithmetic is drifting. Two different bugs, two different fixes, and one aggregate number would hide both.
Parity to 1.2e-4 says this implementation agrees with PyTorch on the sentences tested. It does not say the arithmetic is correct in general — that is a claim about all inputs, and no finite test makes it.
Which is why the test suite does not stop at the reference sentence, and why a second implementation exists that shares no code with this one. Part 8 is about that, and about the bug it found.
Next: downloading twenty rows of a thirty-one megabyte table.
More to read
Choosing the most confident head picked a positional one every time. It reported “it → was, 0.962” — the answer that head gives for every word in every sentence.
The word-embedding table is 31,254,528 bytes. A twelve-word sentence needs about 20 KB of it. HTTP Range makes that the only part that travels.
Three matrices, one dot product, one softmax. The row always sums to 1 — which is a constraint the model must satisfy, not a finding it reports.