← MRM/Projects/Zeus
Machine Learning / Systems Research build Rust · custom GPU kernels

Zeus

A language model built from scratch — and the measurement that caught it training on half a gradient.

Zeus is a four-reservoir RWKV-7 language model where everything is mine: the architecture, the training loop, the GPU kernels, the tooling. It runs in Rust on one consumer AMD card that also drives my display, and at byte level — no word tokenizer, one byte at a time — it writes proper sentences, named characters and dialogue. The part worth showing you is not the model. It is that a training run can be silently wrong for months while every dashboard still looks fine.

zn-train — head gradient probe
$ cargo run -p zn-train --example head_lds_probe ✔ control n=4096 rel-L2 1e-6 vs f64 host oracle [in bounds — harness valid] ✔ control n=8192 rel-L2 1e-6 vs f64 host oracle [in bounds — harness valid] ✘ live n=16384 rel-L2 1.09e0 vs FULL · 1.5e-6 vs TRUNCATED-at-8192 the head gradient is not approximate. it is a different vector.

Representative output — not a live instance.

What it is
●Written from scratch, in Rust — a four-reservoir RWKV-7 stack with hand-written HIP kernels for the forward and backward passes. The training loop is Rust end to end — no framework doing the part I'd otherwise have to take on faith.
●Grown, not restarted — the model went from 48 to 56 blocks with a function-preserving growth step, verified at rel-L2 0e0 against the network it grew from. The coherent base was never thrown away.
●Byte-level English — vocabulary 256, so it has to learn spelling before it can learn words. It does, and the samples read as real sentences with named characters and dialogue.
●Judged by output, not by the loss curve — change the data mix and the loss number stops being comparable to yesterday's. Every 500 steps dumps generated text, and the text is the verdict.
●It ships its own instruments — an f64 host oracle for kernels, a checkpoint differ that finds weights which never moved, a bounds census over the live corpus, and config validation that refuses a kernel that is unsafe at the configured vocabulary size rather than running it.
The 20% measurement found
●A kernel at 0.36% of peak is a defect, not a missing optimization — that number was read as "an untiled matrix multiply" for three weeks. It wasn't slow. It was wrong.
●The head gradient was truncated — a shared-memory buffer fixed at 8,192 floats, launched with 16,384. The hardware silently drops the out-of-range writes and returns zero on read, so the one gradient carrying loss into every block and the embedding table was summed over the first half of the vocabulary only — for roughly 221,000 training steps.
●Controls before the verdict — the probe ran in-bounds cases first and matched an f64 host oracle at ~1e-6, which is what makes the harness trustworthy. Only then does the live shape mean anything: 1.5e-6 against the truncated reference, 1.09e0 against the correct one.
●Sized honestly, which made it smaller — the tokenizer is frequency-ordered, so the dark half is the rare half: 8.40% of target tokens lost their target term outright. "Half the training was broken" would have been the dramatic claim. It would also have been false.
●Then a second one, found the same way — a constant in a backward kernel encoding an assumption about a forward pass that had since been upgraded. Measured result: 163,840 parameters sitting at their initial values across all 40 layers of that run, one whole output pathway that did not exist. Both bugs share a greppable shape — a magic number in a backward pass with a comment explaining why it's safe.
●And the honest ending — after the repairs, validation loss fell seven runs in a row, every one below the previous floor. The text dumps did not change. Perplexity 252 → 200 is real, and it is not coherence. Reporting the win I actually got is the whole discipline.
Proof
~221k
steps trained before the truncation was caught
1.09e0
rel-L2 vs the correct gradient — a different vector, not a rounding error
8.40%
of target tokens actually affected — measured, not assumed
0
parameters found frozen at their init values
0.36%
of roofline — the clue that read as "unoptimized" for three weeks
60.1 ms
corrected kernel, down from 751.6 ms at the live shape
Rusthand-written HIP / ROCm kernelsRWKV-7 · four reservoirsbyte-level + 16k BPEf64 host oraclesingle consumer GPU
ℹ
What it is and what it isn't: Zeus is a research build, not a product and not an assistant. At the rung it has reached it writes coherent short-story English; it does not answer questions, and I don't claim it does. I built it to understand the whole stack end to end — including what it looks like when that stack lies to you.
🔒
Architecture and measured results shown. Source, kernels and checkpoints are private — this is a capability showcase, not a distribution.

Is your training run
actually training?

Most of my work on model code is measurement: known-positive controls, host-side oracles, gates that refuse a bad configuration instead of quietly producing a number. Tell me what you're training and what you'd need to see to believe it.

Start a project →