· Measurement · 5 min
Why "faster" needs a noise floor: Bayesian verdicts for GPU benchmarks
A 17.6% speedup Kernalin called "equivalent": how bootstrap resampling and a ±2% equivalence band stop benchmark noise becoming false claims.
By Kernalin Engineering
Here is a real line from a Kernalin benchmark on 9 October 2026 (CPU sandbox, attention kernel, after the merge in our end-to-end demo):
| Runtime | Batch | Verdict vs PyTorch eager | Speedup | P(faster) |
|---|---|---|---|---|
| torch_compile | 1 | Faster | 1.222× | 100.0% |
| torch_compile | 8 | Equivalent | 1.176× | 94% |
| torch_compile | 32 | Slower | 0.371× | 0.0% |
At batch 8, torch.compile’s median was 17.6% better, and Kernalin still called it equivalent. That is deliberate.
A median is not a verdict
Timings are noisy: clocks drift, caches warm, neighbours share the machine. A single median ratio can’t tell a real 17% from a lucky streak. Gate a CI pipeline on raw ratios and it will flap; publish them and you will overclaim.
What Kernalin does instead
- Bootstrap. Each side’s samples are resampled 2,000 times to get a distribution of the speedup, with a 95% interval.
- A region of practical equivalence (ROPE). Differences within ±2% are treated as “the same”, because nobody should ship or block on less.
- A probability threshold. “Faster” or “slower” needs at least 95% of the posterior outside that band, on the right side.
- Enough data, or no answer. With fewer than 8 samples per side, Kernalin abstains rather than guessing.
At batch 8 the posterior put 94% on “faster”: close, but under the bar. More samples might well get it over. Until then, it isn’t a claim.
Why this matters more for AI-written kernels
An optimizer that is rewarded for speed will find noise and call it progress if you let it. The same rule decides whether an AI rewrite is kept: in the demo, rewrites #2 and #3 were correct but measured 0.37× and 0.40×, and were reverted; #1 cleared the bar at 7.64×.
And for CI
The same statistics run in the rnx-benchmark GitHub Action. Comparisons only happen within one environment (GPU runs are keyed to GPU, driver and CUDA build), and an unreachable Kernalin produces a warning, never a failed build. On shared GitHub-hosted runners you can loosen the blocking threshold in a config file and still see smaller moves reported.