Shipped, and the discipline behind it: production on AMD MI350X

Correctness is the gate; speed is a report.

Yantrion's original production home is a live SGLang deployment on eight AMD MI350X GPUs (gfx950, tensor-parallel across all eight). Shipping a change into that path is a different problem from proving a kernel in a notebook, because the failure mode that matters most is silent: a change that improves latency while quietly corrupting an answer. A latency win is easy to measure and easy to celebrate; a corrupted state can pass every timing check and still be wrong. So the governing rule of the production discipline is stated as a principle, not a preference: every change is gated on a downstream task metric, and never on latency alone.

The task metrics are chosen to be complementary. Long-context retrieval (needle-in-a-haystack, NIAH) tests whether a fact buried deep in a long context still surfaces. Multi-step reasoning (GSM8K) tests whether chained inference holds. Both are held at parity on every shipped configuration, across multiple production models. The capacity result on this stack is read directly from the engine's startup log: 3.31M resident tokens against a stock 1.84M, a factor of 1.80× realized at iso-accuracy after fixed overhead — a single auditable number a skeptic can reproduce with one grep.

The fidelity-versus-task trap.

There is a subtle methodological hazard here worth naming, because getting it wrong is the classic way a serving optimization ships broken. It is tempting to gate on an internal fidelity or proxy score — some cheap internal measure of how faithful a change is — because such scores are fast to compute and correlate with quality most of the time. The trap is that they correlate only most of the time. A high internal similarity score can coexist with a failed task: the needle is found, but a digit in the answer is wrong. The only measure that discriminates the good change from the bad one is the downstream task result itself. This is why the accept condition above is written on A_task and nothing else — we never substitute a proxy for the task.

Capture-replay bisection: debugging as measurement.

When a task metric does regress, the method for localizing the cause is capture-replay bisection, and it is worth describing as a general methodology because it is what makes hard, nondeterministic serving bugs tractable. Capture the real inputs flowing through the live serving path. Replay them offline against an independent reference. Diff the two in stages, narrowing the interval until the discrepancy localizes. The power of the method is that it turns a vague “the server is a little wrong sometimes” into a reproducible offline comparison with a definite answer. It cracked the hardest accuracy regression on this stack — root-caused by replaying captured production inputs against the reference and bisecting the staged diff until the responsible stage was isolated, then fixed and re-gated to parity.

Statistical rigor for a nondeterministic server.

Production LLM serving is not deterministic — mixture-of-experts routing makes run-to-run outputs vary — so an accuracy number from a single run is a noisy estimate, and a gate built on a noisy estimate flaps. The response is to treat each task metric as a statistical functional and to stabilize it. Retrieval is taken as the majority of three independent runs, which suppresses single-run routing noise; the reasoning benchmark serves as a larger, stable anchor. This is ordinary statistical hygiene applied to a serving gate: measure the functional, not a lucky sample.

Underneath the statistical gate sits the anchor that makes the whole thing cheap to trust — and it is the same property that makes adoption safe: flag-off exactness, where off reproduces stock output exactly. It doubles as a free, always-available correctness oracle, which we lean on constantly.

One flag: reversible adoption and one-flag fault isolation.

With the layer disabled, the serving path is the reference server exactly — stock pool, stock weights, stock kernels, nothing of Yantrion's compiled or launched, and it reproduces stock output exactly. This has two consequences that matter to an operator. First, adoption is reversible: flag-off is your exact current baseline, so the downside of trying the layer is bounded to turning the flag back off. Second, it makes the system bisectable at the coarsest possible granularity — a single flag cleanly partitions “is this Yantrion or the base server?” — which collapses the first and most expensive step of any production incident to one toggle.

Generalization without architecture change, and the honest cost.

A production technique earns trust partly by not being overfit. New models are brought onto this stack by swapping the model and re-fitting a calibration once — no architecture change to the serving integration. Multiple production models have been added this way, each re-passing the same NIAH-plus-GSM8K gate. On the latency side the disclosure is plain: at production concurrency (CONC 32–128) the per-step latency ratio τ is roughly 0.96–0.98 — at or slightly under stock — and the only measurable tax appears at trivial single-user batch, which is not the regime a capacity technique is for. As concurrency rises the per-step cost amortizes, so the tax is smallest exactly where the capacity is most needed.