Bring-up as science: validating on NVIDIA Blackwell

Porting is an experiment, not a copy.

Moving a serving-layer technology to a new GPU architecture is often described as “just a port.” Treated seriously it is an experiment with a hypothesis: that the ratios established on one silicon — capacity, accuracy, latency — reproduce on another. Blackwell (NVIDIA GB10, sm_121a, CUDA 13) was brought up from scratch on real hardware, and the discipline that made the result trustworthy was to gate every milestone on a strict numerical check against an fp32 reference before allowing any performance claim to be spoken. The order matters: correctness first, speed second, always.

The controlling invariant of the bring-up is a numerical identity. For every algorithm-aware kernel, at every milestone, its output was compared against an independent fp32 reference computed from the exact same operands.

This is a strong gate. It is not “close enough,” it is the full six-nines of a cosine that has saturated to one, held across every configuration in the matrix — all tested dtypes, head geometries, batch sizes, and sequence lengths out to 16k. A companion invariant checks that the parallel reductions inside a kernel are order-independent, so a result does not depend on how the work was tiled across the machine.

Milestone gates, each with a falsifiable check.

The bring-up ran as an ordered sequence of milestones, and each one had a pass/fail numerical condition that had to be met before the next began. Early milestones established that the native numeric formats ran on the machine's tensor cores and that a reference decode harness reproduced a known-good result exactly. Middle milestones extended the harness to a complete flow on captured, real per-layer tensors — still gated cos = 1.000000. Later milestones proved accuracy on a real model (retrieval and multi-step reasoning both at parity), then confirmed the capacity multiplier in the serving engine's live KV manager, then closed latency. Two risks the pre-hardware plans had carried were retired by measurement rather than assumption. Retiring a risk by measurement is the whole point of a gated bring-up.

The latency close, read off the roofline.

The interesting part is what the roofline said about where to look. Decode on GB10 is bandwidth-bound: per-step time is set by bytes moved. Yantrion moves proportionally fewer bytes per step — the same ρ that shows up in capacity shows up here as a bandwidth reduction — so the memory-bound floor for the latency ratio is well under one. Any gap to that floor is the kernel spending time on things other than moving bytes. That reframed the work from “is this inherently slow?” to “find where the kernel is not bandwidth-bound and remove it.”

The close was pure kernel engineering, described here only at the level of latency and bandwidth. The decisive decode kernel was taken from 228 µs to 80 µs — a 2.85× speedup — by cutting launch and dispatch overhead and parallelizing across the machine more fully. Every optimization step was re-gated cos = 1.000000 against the validated reference and re-checked on retrieval, so no speedup was allowed to purchase itself with a silent numerical error. The result: τ ≈ 1.07 at 3.4× capacity — within a few percent of the bandwidth-bound parity the roofline promised.

Capacity (3.45× live) and latency (τ ≈ 1.07×) are reported here as separate measurements. We do not divide one by the other into a single throughput figure — they are not co-measured at one operating point, so the quotient would not describe an attainable configuration.

Same silicon, two independent stacks.

Reproducibility is the scientific standard for believing a result, and on Blackwell it was met at a strong level: the same outcome, on the same silicon, through two independent serving stacks that share no integration code. On vLLM the layer ships as a real out-of-tree plugin — a first-class engine entry point, not a runtime patch — reproducing the capacity multiplier in the engine's own KV manager, retrieval at parity, and the near-parity latency above. On TensorRT-LLM (DeepSeek-V2-Lite) the layer was validated in our integration: its C++ KV manager accepted the reduced per-token footprint at 3.43× capacity (3.51× by serving footprint) and served long-context retrieval at stock, with the layer disabled reproducing the reference server's output exactly. Two stacks arriving at the same ratios on the same hardware is a much stronger statement than one stack doing it twice.