Why memory bandwidth predicts decode speed

Generating one token requires reading every active weight from memory exactly once. That single fact explains most of what the benchmark table shows.

A dense 8B model at 4-bit quantisation holds roughly 4 GB of weights. A card with 900 GB/s of memory bandwidth can read that about 225 times per second, so 225 tok/s is its hard ceiling and no amount of extra compute moves it. Real backends land somewhere below, and this site fits that fraction from the verified records rather than assuming it.

Two consequences worth internalising:

Compute barely matters for single-user decoding. Two cards with similar bandwidth and very different TFLOPS generate at similar speeds. Compute matters for prompt processing, which is why this site tracks prefill and decode as separate columns.

Mixture-of-experts models break the intuition. A 30B MoE with 3B active parameters reads only the active experts per token, so it decodes at roughly the speed of a 3B dense model while needing the memory of a 30B one. That is why the data model stores active_params_b separately, and why the estimate formula uses it rather than the total.