Past the capacity wall: the physics of KV-bound inference

The bottleneck is bytes, not FLOPs.

A useful way to reason about an inference fleet is to ask what physically fills the GPU first. For long-context, high-concurrency serving the answer is rarely the matrix units — it is the key/value cache, the running per-request state that attention must read on every decode step. That state is not optional, it does not shrink over a request's life, and its size is fixed by a simple accounting identity.

Three of these four factors are fixed by the model and the workload. L is architectural. T and R are precisely the two axes a product roadmap pushes on — longer documents and more simultaneous users or agents — and because they enter as a product, demand grows in the directions the business grows, multiplicatively. That leaves exactly one factor that is neither dictated by the model nor by the customer's ambition: b, the per-token footprint. It is the only free variable in the identity, and therefore the only lever that moves the wall without asking the workload to be smaller.

Why the regime is memory-bound.

The reason capacity is destiny here is that decode is also bandwidth-bound, so the same cache that fills memory also gates latency. Each generated token requires the model to read the entire resident state for its request. The arithmetic performed against those bytes is small, so the operational intensity — useful work per byte moved — sits far to the left of the hardware's ridge point, deep in the bandwidth-limited region of the roofline.

In that regime the per-step time is governed by how many bytes must cross the memory system, not by how fast the arithmetic units can run. To first order, step time is proportional to the bytes read per step, which is itself proportional to b. This is the pivotal fact of the whole analysis: b appears in the capacity identity and in the latency model at once. Shrinking b relieves both pressures with a single change — it is not a capacity trick that costs latency, it is the term both equations share.

Why weight quantization does not move this wall.

It is worth being precise about a common misconception, because it isolates what actually changes. Weight quantization — MXFP4, INT4 and their relatives — shrinks the model's parameters. That is real and valuable, but it acts on a different term. It does not touch b. In the identity M_kv = b · L · T · R, weight quantization changes none of the four factors, because the parameters do not live in the KV cache. After quantizing weights the per-token cache footprint is exactly what it was, so the wall is exactly where it was. The two techniques are orthogonal and compose: one shrinks the model, the other shrinks the per-token cache footprint, and they stack. Throughout, we treat the outcome b_yantrion < b_stock strictly as a measured result — a smaller value of the one free variable — and say nothing about the method that produces it.

The outcome: a capacity ratio, honestly ranged.

The consequence of a smaller b is a capacity ratio. Define it directly from the footprints.

Both endpoints of that range are measured, and the honesty is in reporting the range rather than a single number. The upper end, 3.43–3.45×, is the allocator ceiling — what you read straight from a serving engine's own KV manager when it reports resident-token capacity at fixed VRAM. On one NVIDIA GB10 configuration the live pool allocated 9.07M tokens against a stock 2.63M — a factor of 3.45× — in the same running engine that served long-context retrieval at parity; a second GB10 stack reproduces 3.43× (3.51× by the live serving footprint). The lower end, 1.8×, is the conservative realized floor: on a production high-concurrency AMD MI350X deployment, once fixed overhead shares the VRAM, the realized multiplier settles at 1.80×. Plan with 1.8× as the floor and 3.4–3.5× as the ceiling.

From capacity to cost and fleet size.

Capacity is the lever; the quantities a fleet owner cares about follow from it. Two of them derive cleanly from a measured capacity ratio ρ — and one does not. We deliberately do not divide capacity by the latency ratio into a single throughput number: capacity and latency are measured separately, at different operating points, so their quotient would not describe one attainable configuration. What does derive cleanly is unit cost and fleet size.

Cost per unit working set lands near 0.28–0.31× at the ceiling and near 0.56× at the 1.8× floor (derived, labeled). And N = P/C makes the agentic case concrete: a node holds a fixed pool P, so multiplying P by ρ multiplies the sessions it can hold resident before it must evict and re-process — which is exactly where memory-bound serving latency cliffs rather than degrades.