Benchmarking / Measurement
Built · hardware run still ahead
Rust · clippy -D warnings clean
refbench
A benchmark harness that refuses to report a number it cannot defend.
I am renting a data-center GPU for about a week to compare kernels against the stock libraries. A short window on paid hardware is exactly the situation where you talk yourself into a number. refbench measures nothing itself — the backends do that — and instead decides what is allowed to become a result. Correctness gates performance, a flagged reading never becomes a ratio, and the baseline runs before any subject so that if the session dies mid-run I still hold stock performance on that exact machine, measured by me.
refbench — self-test
$ refbench run --plan plan.selftest.toml
✔ provenance captured before the first measurement
✔ control ok · baseline ok · referee named on every entry
✕ n4_perfect max_abs_err = 0.0 zero_error
✕ n5_impossible gflops above ceiling above_ceiling
✕ n8_hang killed and reaped at 3s TimedOut
2 clean · 9 flagged · exit 1 · trusted: true
Representative output — not a live instance.
What it refuses to do
●Print throughput for a wrong answer — correctness gates performance. A fast incorrect kernel produces no gigabytes per second and no gigaflops, because a fast wrong answer is not a fast answer.
●Turn a flagged measurement into a ratio — the report prints not ranked instead of a speedup. There is no path by which a suspect reading gets laundered into a headline number, because the ratio is never computed in the first place.
●Rank on anything but the minimum — contention only ever adds time, so the minimum is the honest comparison. The mean is not recorded at all, which means it cannot be reported by accident.
●Offer a "probably fine" outcome — a run is Ok, Failed, TimedOut, BadOutput or NotLaunched, and nothing else. BadOutput is separate from Failed on purpose: it means the adapter is wrong, not the kernel, and those get fixed in different places. Timeouts kill and reap, so nothing is left behind to skew the next measurement.
●Keep going when the control breaks — if the known-good backend stops behaving, the run halts and every row already collected is marked untrusted. Forcing past it is allowed; the rows stay marked untrusted anyway.
●Invent its own result format — the measurement record is field-identical to the one my GPU kernel autotuner emits, so results move between the two tools with no translation and no field quietly changing meaning on the way across.
What it insists on before anything runs
●The baseline goes first — the plan validator refuses any other order. On rented hardware that ordering is the cheapest insurance there is: stock performance on that exact machine, measured by me, is a real artifact on its own even if nothing after it finishes.
●A named referee on every entry — implementation, platform and version, or the plan does not load. No control backend, or a warmup of zero, and it does not load either. A comparison with an unnamed opponent is not a comparison, and it is the failure mode I most wanted to make impossible.
●Provenance captured up front — recorded before the first measurement, and an incomplete capture is reported as incomplete rather than rendered as empty fields. Absent and unknown and measured are three different things and the output says which one it has.
●A row on disk before the next one starts — flushed to the ledger and resumable by key, so a dropped session picks up where it stopped instead of quietly re-measuring and averaging two different machine states together.
●A suspicious-perfection auditor — five fatal flags (incorrect, zero time, incoherent ordering, above the physical ceiling, non-finite) and four suspect ones (zero variance, implausible stability, zero error, no throughput). The rule behind all nine: a perfect reading is a defect until proven otherwise.
It rejected its own control on the first run
●Nine deliberate failure modes, one per rejection path — the self-test control is a plain CPU matrix multiply that can be told to be wrong, frozen, instant, perfect, impossible, silent, crashed, hung, or simply absent. Running it is how I know the gates fire rather than assuming they do: 2 clean, 9 flagged, exit 1.
●The one I did not plant — the control's fast kernel used 32-wide blocks at a problem size of 96, which is exactly three blocks, so it walked the reduction in the same ascending order as its own reference. The two outputs were bit-identical, the error was exactly 0.0, and the auditor rejected the control with zero_error. It was right to.
●Why exactly zero is a defect — a reference that accumulates in the same order as the thing it is checking is not a reference. It is the same computation twice, and it passes forever while catching nothing. The reference now walks the reduction backwards, so genuinely different rounding produces a real, tiny discrepancy and the comparison is doing work.
●The tool's own thesis landed on the tool first — which is the only reason I believe the rest of it. A harness that has never rejected anything has not been shown to be capable of rejecting anything.
Proof
0
rejection paths · each with a control proven to fire
2 / 9
clean vs flagged on its own self-test · exit 1
not ranked
what a flagged measurement becomes. Never a ratio.
0
lines of Rust · every file under 400
0
"probably fine" variants in the outcome type
0.0
the error that got its own control rejected
RustJSON over stdin / stdoutflushed-per-row ledgerclap · serde · tomlbackend-agnostic — any program
ℹStatus, stated honestly. refbench is built, clippy-clean, and proven against its own negative controls. It has not yet run on the rented GPU — that trip is still ahead, and the three thin backend adapters it needs are unwritten. The self-test is what exists today, and the self-test is a real run rather than a claim.
🔒Rules and verified self-test results shown. Source and the plans are private — this is a capability showcase, not a distribution.
Getting numbers
you want to believe?
Rented hardware, a deadline, and a result that looks good is how bad benchmarks get published. I build the harness that decides what counts as a measurement before any measuring starts — and I show you what it rejects. Tell me what you need compared.
Start a project →